Boosting Text-to-Video Generative Model with MLLMs Feedback

Authors: xun  wu
shaohan  huang
guolong  wang
jing  xiong
furu  wei
microsoft  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 02:06:41

Abstract: Recent advancements in text-to-video generative models, such as Sora, have showcased impressive capabilities. These models have attracted significant interest for their potential applications. However, they often rely on extensive datasets of variable quality, which can result in generated videos that lack aesthetic appeal and do not accurately reflect the input text prompts. A promising approach to mitigate these issues is to leverage Reinforcement Learning from Human Feedback (RLHF), which aims to align the outputs of text-to-video generative with human preferences. However, the considerable costs associated with manual annotation have led to a scarcity of comprehensive preference datasets. In response to this challenge, our study begins by investigating the efficacy of Multimodal Large Language Models (MLLMs) generated annotations in capturing video preferences, discovering a high degree of concordance with human judgments. Building upon this finding, we utilize MLLMs to perform fine-grained video preference annotations across two dimensions, resulting in the creation of VideoPrefer, which includes 135,000 preference annotations. Utilizing this dataset, we introduce VideoRM, the first general-purpose reward model tailored for video preference in the text-to-video domain. Our comprehensive experiments confirm the effectiveness of both VideoPrefer and VideoRM, representing a significant step forward in the field.

Main paper: 
 introduction: 
 Diffusion models have significantly enhanced the quality of generation across various media formats, including images [30, 31, 55] and videos [13, 22, 42] . In the context of text-to-video generation, recent advancements in text-to-video diffusion models, exemplified by Sora [4] , have achieved remarkable success in generating high-quality videos from textual prompts. However, despite recent progress, the visual fidelity of generated videos still presents opportunities for refinement [38] . Video generation poses greater challenges than image generation, as it entails modeling a higher-dimensional spatio-temporal output space while remaining conditioned solely on a textual prompt. Consequently, existing text-to-video generative models often produce results that are visually unappealing and inadequately aligned with the provided textual prompts.To address these limitations, a pivotal solution is Reinforcement Learning from Human Feedback (RLHF), which has proven its efficacy in text-to-image diffusion models [51, 2, 26] . These methods aim to fine-tune a diffusion model to maximize a reward function corresponding to specific aspects of image quality or alignment with text prompts. Despite this, when a Reward Model trained in the image domain is transferred to the video domain, it can exhibit significant disparities with the objectives of video optimization (e.g., InstructVideo [53] ). Training a reward model directly in the video domain is becoming increasingly urgent, but the scale of preference datasets in the video field is small (see in Table 1 ). Recognizing the importance of addressing these challenges in text-to-video generative models, we first construct a large-scale fine-grained preference benchmark by utilizing MLLMs as annotators, namely VIDEOPREFER. VIDEOPREFER contains following strength: (1) VIDEOPREFER is the largest open-source video preference dataset, containing 135,000 preference choices (see in Table 1 ).(2) Different from existing video preference datasets, VIDEOPREFER contains true video captured by human, making VIDEOPREFER more generalizable. (3) VIDEOPREFER is annotated using Multimodal Large Language Models, which is cost-effective and easily scalable.Based on VIDEOPREFER, we release the first general-purpose text-to-video preference reward model, VIDEORM. Unlike the preference rewards in the image domain used by previous methods for video generation alignment [53] , our VIDEORM automatically captures the temporal information in videos, enhancing the modeling of quality assessment and alignment. Furthermore, we investigate the alignment of text-to-video generative models using VIDEORM and conduct extensive experiments that demonstrate its efficacy as a reward model and metric for video alignment. This significantly improves the generation quality of video generation models across multiple aspects. Our main contributions are:• We systematically identify the challenges for text-to-video human preference annotation. Consequently, we employ MLLMs for preference annotation and construct the largest and most comprehensive video preference dataset, VIDEOPREFER. • We systematically identify the lack of general-purpose text-to-video preference reward model and propose VIDEORM, outperforming existing reward models for text-to-video alignment. • Extensive experimental results validate the effectiveness of both the VIDEOPREFER and VIDE-ORM. Furthermore, through detailed discussions, we demonstrate that MLLMs are effective and cost-effective as annotators for video preferences, revealing the promising future prospects of RLAIF in the domain of video preference alignment. 
videoprefer: 
 We introduce VIDEOPREFER, a large-scale, fine-grained video preference dataset constructed by collecting feedback from multimodal large language model annotators. In total, VIDEOPREFER contains 135K pairs of binary preference choices for 54K videos. In § 2.1, we provide evidence demonstrating that GPT-4 V is a human-aligned preference annotator in the video domain. The construction pipeline of VIDEOPREFER is introduced in § 2.2. Detailed analysis of the statistics of VIDEOPREFER can be found in Appendix § B.1.2.1 Why GPT-4 V(ision) can act as a human-aligned preference annotator?GPT-4 V has already demonstrated annotation performance aligned with human consistency in textto-image generation [7, 46, 44] . To further validate GPT-4 V's ability to provide reliable preference annotations in the video domain, we used different MLLMs to annotate a subset of the TVGE dataset and calculated their correlation with the ground truth annotations, e.g., accuracy and kendall. The results are shown in Table 2 . We find that (1) GPT-4 V's annotation accuracy and correlation are the best among all MLLMs. (2) The accuracy of GPT-4 V is 69.65%, which is very close to the previously reported agreement rate between qualified human annotators (approximately 70% [9, 46, 25, 11] ). This demonstrates that GPT-4 V is a reliable annotator.The construction pipeline of VIDEOPREFER can be split into following three steps:Step-1: Prompts Collection. We collect prompts from VidProM [40] which contains 1.67 million unique text-to-video prompts, as well as from two video-captioning benchmarks: ActivityNet [5] and MSR-VTT [49] . For VidProM, we just randomly sample 12.9k prompts from its corpus. For video-captioning benchmarks, we directly utilize the provided video segment captions from the original dataset as prompts. For instance, in the ActivityNet [5] , we utilize the text provided in caption corpus, comprising 0.9K thousand text instances corresponding to distinct video segments. Consequently, we obtain a total of 14K prompts, constituting the prompt set for our VIDEOPREFER.Step-2: Video Collection & Generation. We generate videos with text-to-video models and collect real-world videos based on prompts collected in Step-1 for preference annotation. Through these two ways, we obtained 54K video candidates:• Model-generated videos. We selected the top-ranked text-to-video generation models on Hugging-Face1 (e.g., ModelScopeT2V [38] ), as well as the open-source state-of-the-art text-to-video models (e.g., Open-Sora [57] ), to constitute our video generation model pool. Details regarding these models, as well as the proportions and resolutions of the generated videos, can be found in Table 5 . For each prompt, we randomly sample models from the model pools, along with class-free guidance scales, to generate four different corresponding videos. This approach results in a high degree of diversity, facilitating the training of a more generalizable and comprehensive preference reward model.• Real-world videos. Given the substantial quality disparity between videos generated by existing text-to-video generation algorithms and real videos, we also incorporate real-world videos to enhance the generalizability and diversity of our VIDEOPREFER. Specifically, we incorporate video segments from the two datasets introduced in Step-1 (ActivityNet [5] and MSR-VTT [49] ), as one of the four candidate videos, with the remaining three being generated as above.Step-3: Preference Generation. Due to the prohibitively high cost of manual annotation, we utilize the state-of-the-art multimodal large language models (MLLMs), i.e., GPT-4 V to annotate video preferences, enabling extensive and fine-grained annotation at scale. The reliability of GPT-4 V as annotators is thoroughly discussed in § 2.1. Specifically, for each prompt and its corresponding four video candidates, we employ GPT-4 V to assign preference scores on a 1-to-5 Likert scale to each video candidate for two aspects: Prompt-Following and Video Quality. Lower scores indicate lower preference and vice versa. Finally, we obtain 135K preference choices. Detailed input instructions used for GPT-4 V's annotation can be found in the Appendix E.By performing the above three steps, we finally have 14K data items in VIDEOPREFER, while each data item contains a textual prompt and four corresponding generated videos. For each video, there are preference annotations for two aspects: Prompt-Following and Video-Quality provided. More details of VIDEOPREFER, e.g., visualization of example data item (see in Figure 10 ), can be found in Appendix B.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The analysis section contains a discussion of limitations in our work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
videorm: a general-purpose video preference reward model: 
 Based on VIDEOPREFER, we implement the reward model training and derive the first generalpurpose video preference reward model, VIDEORM. We detail the architecture of VIDEORM in § 3.1, its optimization objectives in § 3.2, and the methodology for fine-tuning video generative models with VIDEORM in § 3.3.Existing text-to-video alignment work [53] adopts HPS v2 [45] as the reward model, which is finetuned from CLIP [27] and optimized upon a large image preference benchmark and demonstrates state-of-the-art preference evaluation capabilities for text-to-image generation.However, we contend that the direct application of HPS v2 may lead to a one-sided assessment of video preferences, as it lacks the capability to evaluate the overall attributes of a video, such as temporal coherence and dynamics. Nevertheless, given the strong correlation between video preferences and the preferences of individual frames, the proficiency of HPS v2 [45] in evaluating single-frame image preferences can still aid in enhancing the assessment of video preferences. Thus, to achieve a better evaluation for video preferences, we modify the structure of HPS v2 by integrating several temporal modeling modules, to develop a specialized reward model VIDEORM for the video domain. Specifically, inspired by recent advancements [39, 21] in enhancing the temporal modeling capabilities of the CLIP [27] , we add two kinds of temporal modeling modules into HPS v2 to modeling videos:Temporal Shift [20] , a parameter-free module which shifts part of the feature channels along the temporal dimension and facilitates information exchange between neighboring input frames. Following [39] , we insert the module between every two ViT Layers of the image encoder of HPS v2.Temporal Transformer. The sequence of framewise features extracted by the image encoder are then fed into a temporal transformer to modeling the temporal features of the video.During the optimization process of VIDEORM, we freeze the text encoder and ViT layers contained in the image encoder to retain the single-frame preference modeling capability of HPS v2, while only optimizing the temporal transformer module. Thus, VIDEORM achieves the capability to model video preferences from both the perspectives of individual frames and temporal dynamics, allowing for a more comprehensive understanding of video content.Similar to previous works [36, 25] , to train VIDEORM on VIDEOPREFER, we first average the scores of all 16 aspects for each video candidate, obtaining the final preference score for that video candidate. Then we formulate the preference score for each pair-wise video candidates as rankings. Thus, for each data item (consisting of one prompt T and its corresponding four video candidates v 1 , v 2 , v 3 and v 4 ), we get at most C 2 4 comparison pairs if there are no ties between two video candidates. For each comparison, if v + is better and v -is worse (v + ≻ v -), the loss function can be formulated as:EQUATIONwhere R θ (T, v) is a scalar value of preference model for prompt T and generated video v.Current exploration of reward reinforcement learning algorithms for fine-tuning text-to-video generative models is quite limited. The only related work is InstructVideo [53] , which utilizes an image-domain reward model (HPS v2 [45] ) to fine-tune text-to-video models.Consequently, InstructVideo may have the following shortcomings: (1) Due to the inherent gap between images and videos, directly using image-domain reward models cannot accurately calculate the reward for generated videos, leading to visual artifacts such as structural twitching and color jittering [53] . (2) Additionally, InstructVideo [53] calculates the reward value for all frames selected from generated videos throughout the full DDIM [34] sampling procedure, making the fine-tuning process highly sample inefficient.VIDEORM is specifically designed for the video domain. It is initialized with the weights of the best image-domain reward model and trained on video preference datasets, utilizing temporal modeling mechanisms (e.g., temporal transformers) to capture temporal features of videos. This enables it to evaluate the reward score of generated videos more effectively from both individual frame and temporal perspectives. Additionally, unlike the image-domain reward model used in InstructVideo, which takes single video frames as input, VIDEORM processes the entire video as input. Therefore, our approach is more computational efficient and can directly leverage various effective image-domain reward reinforcement learning algorithms (such as DRaFT [8] ) without the need to balance rewards between frames generated at each step, as required by InstructVideo.Based on the above, we integrated VIDEORM into the image-domain DRaFT [8] algorithm and design a reward reinforcement learning algorithm suitable for text-to-video generative models, named DRaFT-V. The specific algorithm details are shown in Algorithm 1. DRaFT-V truncates the backward pass, differentiating and computing reward score from VIDEORM through only the last K steps, making the full fine-tuning process more efficient. Lpre ← ψ θ i (ti, vi) 7:wi ← wi // Update Υ θ i using Lpre 8:xT ∼ N (0, I) // Sample noise as latent 9:for j = T, ..., 1 do 10:if j > K then 11:no grad: xj- In implementation, rather than fine-tuning the full set of model parameters, we follow [53] to adopt LoRA [14] to further accelerate fine-tuning and circumvent the issue of computational intensity as well as the risk of catastrophic forgetting associated with the reward loss in diffusion models.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will public our data and code upon paper acceptance, due to the management regulations of our institution. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: All relevant human questions and relevant datasets have been checked for privacy compliance prior to experiments and submission.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 We conduct extensive experiments to validate the effectiveness of VIDEOPREFER and VIDEORM. We first train VIDEORM and evaluate it on existing human-preference benchmark ( § 4.1). Next, we finetune existing text-to-video diffusion models for aligning human preference by utilizing DRaFT-V with VIDEORM ( § 4.2). Finally, we present ablation studies ( § 4.3).Setup. Based on VIDEOPREFER, we develop VIDE-ORM, an advanced open-source general-purpose reward model that provides preferences for generated videos. Specifically, we train three versions of VIDE-ORM. (1) Firstly, to validate the effectiveness of VIDEOPREFER, we average the preference scores in each aspect of VIDEOPREFER to get a final preference score, and train VIDEORM-V with the merging version of VIDEOPREFER. (2) Then, we train VIDEORM-H on several open-source human-crafted preference datasets listed in Table 1 . (3) Finally, to build a stronger RM for text-to-video generation, we mix these open-source human-crafted preference datasets with VIDEOPREFER to train VIDE-ORM. The details for dataset processing can be found in Appendix A. All VIDEORM series models are trained in half-precision on 8 × 32GB NVIDIA V100 GPUs, with a learning rate of 1e-5 and batch size of 64 in total. We set the input frames N = 8.Compared Baselines. Due to the lack of reward models in the video domain, we compare VIDEORM with state-of-the-art reward models from the image domain, i.e., CLIP ViT-H/14 [27] , ImageReward [48] , PickScore [17] and HPS v2 [45] . For all these reward models from the image domain, we calculate the scores for all video frames and then take the average as the final reward score for the video.Preference Accuracy. The preference prediction accuracy across test benchmarks from three humancrafted preference datasets are reported in Table 3 . As we can see, the VIDEORM series outperform baseline reward models by a large margin, indicating that VIDEORM series are the best open-source reward models for text-to-video domain. We also find that VIDEORM-V which does not train on any open-source video preference datasets also surpasses all other baselines. This result validates the high quality of VIDEOPREFER enables strong out-of-distribution generalization and validate TVGE [56] VBench [15] T2VQA-DB [18] Figure 3 : Win rates of VIDEORM compared to other reward models across three test benchmarks. On average, 72% to CLIP, 67% to ImageReward, 62% to PickScore and 65% to HPS v2.the effectiveness of treating state-of-the-art MLLMs, i.e., GPT-4 V, as preference annotator for text-to-video generation domain.Best-of-n Experiments. To verify that our VIDEORM could serve as a good indicator of video generation quality, we conduct best-of-n experiments on the T2VQA-DB [18] test benchmark. For each data item, which includes a prompt and ten corresponding videos, we calculate the reward score for each of the ten videos with VIDEORM. Thereafter, we select the best-of-{1, 2, 4, 8} responses and calculate their scores. The final results are presented at Figure 2 , we find the win rate at the test benchmark increases proportionally with rewards, which validates that our VIDEORM gives rigorous rewards that reflect the overall preference score.Human Evaluation. To evaluate the ability of VIDEORM to select the more preferred videos among large amounts of generated videos, we produce human study. We randomly select 200 textual prompts from these three test benchmarks and generate 32 different videos for each prompt by utilizing ModelScopeT2V [38] . Then, we perform different reward to select from those videos to get top3 results. After that, five annotators are asked to identify which video was superior or if both were of equal quality (denoted as 'Tie') and we show the corresponding win rates against other reward models [45] at Figure 3 . Qualitative results can be seen at Figure 11 in Appendix. All of these results show that VIDEORM can select videos that are more aligned to text and with higher fidelity and avoid toxic contents.Setup. Based on VIDEORM, we validate the effectiveness of DRaFT-V. We adopt the publicly available text-to-video diffusion model ModelScopeT2V [38] as base model, which is trained on WebVid10M with T = 1000 and is able to generate videos of 16 × 256 × 256 resolution. We random sample 2K prompt-video pair from the mixture of existing video preference datasets (TVGE [56] , VBench [15] and T2VQA-DB [18] ) and VIDEOPREFER as the training data. Each group of experiment is trained in half-precision on 4 × 32GB NVIDIA V100 GPUs, with a learning rate of 1e-5, batch size of 8, fine-tuning step of 400 and K adopted in DRaFT-V as 10 in total.Compared Baselines. Since the only existing reward-based fine-tuning algorithm for text-to-video models is InstructVideo [53] . We compare our DRaFT-V with the the baseline text-to-video model (i.e., ModelScopeT2V [38] ) and InstructVideo implemented by ourselves. Note that in the aforementioned algorithms, InstructVideo uses HPS v2 [45] as the reward model. Additionally, to verify the effectiveness of using VIDEORM as a reward model for fine-tuning video generation models, we TVGE [56] VBench [15] T2VQA-DB [18] Figure 5 : Win rates of text-to-video models fine-tuned with DRaFT-V compared to other baselines.Here baseline denotes the base text-to-video model without any fine-tuning. introduced two comparison groups: (1) DRaFT-H, which uses the same algorithm as our DRaFT-V (i.e., DRaFT [8] ), but with HPS v2 as the reward model. (2) InstructVideo-V, which uses the same algorithm as InstructVideo [53] , but with VIDEORM as the reward model. Human Evaluation. To evaluate the generative ability of text-tovideo models fine-tuned with DRaFT-V and other compared baselines, we produce human study. We randomly select 100 prompts from these three test benchmarks (TVGE [56] , VBench [15] and T2VQA-DB [18] ), and for each prompt, we generate videos using different fine-tuned models. Five annotators are asked to identify which video was superior or if both were of equal quality (denoted as 'Tie') and we show the corresponding win rates at Figure 5 . We find that: (1) All of these results show that DRaFT-V can generate videos that are more prefered by human. ( 2) By comparing DRaFT-V with DRaFT-H, as well as InstructVideo-V with InstructVideo, we demonstrate that VIDEORM is a more effective reward model for fine-tuning text-to-video models.Besides, qualitative results are presented at Figure 12 Efficiency Evaluation. We compare the efficiency of our algorithm on two aspects: (1) Convergence speed: we visualize the changes in reward values with training steps for DRaFT-H and DRaFT-V in Figure 6 (a). We find that under the same reward-based fine-tuning algorithm, using VIDEORM leads to earlier convergence and better performance (as demonstrated by the results above) compared to HPS v2, further validating the effectiveness of VIDEORM in fine-tuning text-to-video models. (2) Inference speed. For a single video with 8 frames, HPS v2 requires approximately an average of 5 seconds for inference, while VIDEORM only needs an average of 1.3 seconds. Besides, InstructVideo requires 20 DDIM steps for a single fine-tuning step, taking approximately an average of 8.8 seconds, while DRaFT-V only needs an average of 2.3 seconds.Scalability. To investigate the effect of training dataset sizes on the performance of VIDEORM, and to verify the scalability of VIDEOPREFER, comparative experiments are conducted. Note that VIDEORM in this experiment are only trained on VIDEOPREFER. Figure 6 (b) shows that adding up the scale of the dataset significantly improves the preference accuracy of VIDEORM. It's promising that if we employ GPT-4 V to collect more annotation data in the future, VIDEORM will get better performance.Number of Video Frames N Input to VIDEORM. We investigate the impact of different numbers of input video frames on the performance of the RM model. Specifically, we set the number of input frames to N = 1, 4, 8, and 12 and test corresponding model at TVGE [56] test benchmark.The corresponding results are presented at Figure 6 (c), we find that the VIDEORM's performance significantly decreases when N is small. This is promising because too few input frames prevent the model from accurately capturing all the information and temporal features of the video, e.g., the degree of motion and temporal consistency. Backbone of VIDEORM. We explore the impact of different backbone on the final performance of VIDEORM. In our design, VIDEORM is initialized with the weights of HPS v2 [45] , which is demonstrated as the best reward model for image domain. Here we replace HPS v2 with other reward models in image domain, e.g., PickScore [17] (denoted as VIDEORM σ ) and ImageReward [48] (denoted as VIDEORM β ) while keep the count of trainable parameter equal. The corresponding results are presented at Table 4 . We find that the final performance of the corresponding VIDEORM is proportional to the performance of the initialized backbone weights. For example, ImageReward, which has the lowest performance among the three (see Table 3 ), corresponds to the worst VIDE-ORM β , while PickScore, which has the best performance among the three (see Table 3 ), corresponds to the best VIDEORM σ .Step K adopted in DRaFT-V. We investigate the impact of different K adopted in DRaFT-V on the final generative performance. The results are presented at Figure 7 , we find that: (1) Performance degrades as K increases for K > 10.(2) Even with smaller values of K, e.g., K = 1, DRaFT-V can achieve good results. Both findings are aligned with the results in DRaFT [8] . 
analysis: 
 In this section, we analyze the impact of different parameter selections-specifically, the number of frames (N ) sampled from a single video and the temperature setting τ of the GPT-4 V, on the accuracy of GPT-4 V-generated annotations. Due to GPT-4 V's limitation of processing a maximum of 10 frames, we analyze annotation accuracy for N = 4, 6, 8, and 10 frames. The temperature τ controls the diversity and randomness of GPT-4 V's outputs. We analyze annotation accuracy for τ = 0.3, 0.5, 0.7, and 0.9. The summarized results are shown in Figure 8 . For N , accuracy initially increases with more frames but decreases beyond N = 8, achieving the highest accuracy at N = 8. This is because more frames allow for a comprehensive evaluation, but too many frames can lead to overly long contexts, reducing accuracy due to GPT-4 V's limited capacity. For τ , lower values yield higher accuracy, likely because reduced randomness leads to more comprehensive predictions, while higher randomness might cause GPT-4 V to overly focus on specific frames or aspects, resulting in less comprehensive outcomes. 
related work: 
 Generative models, particularly those based on diffusion techniques, have demonstrated high-quality generation capabilities [32, 28, 31, 30, 35] by training on extensive internet-scale datasets, but the mixed quality of these datasets often leads to visually unappealing and misaligned outputs.Aligning text-to-image generative models. Aligning text-to-image generative models [37, 17, 8, 26, 37] has garnered increasing attention in recent years and has shown promising results in producing outputs that are more aligned with human preference. Imagereward [48] , HPS v2 [47, 45] , and PickScore [17] are the three most commonly used reward models for align text-to-image models. They are respectively trained on three major image domain preference datasets, namely Imagereward, ImageRewardDB [48] , HPD [47, 45] and Pick-a-Pic [17] . The effectiveness of these reward models has been validated across alignment algorithmss like AlignProp [26] , DRAFT [8] and ReFL [48] .Aligning text-to-video generative models. Compared to text-to-image, exploration related to aligning text-to-video generative models is relatively sparse. InstructVideo [53] instruct text-to-video diffusion models with by fine-tuning with existing image-wise preference reward model HPS v2. We hypothesize that this may restrict effective preference evaluations for generated videos, as imagebased reward models cannot adequately capture temporal features, impairing assessments of video coherence and dynamics. Additionally, preference datasets for text-to-video generation are also scarce. The lack of large-scale and effective open-source preference datasets severely restricts the development of research related to align text-to-video generative models.Reinforcement Learning from AI Feedback. LLMs have been extensively used for data generation [43, 24] , augmentation [12] and in self-training setups [41, 23] . Some works [1] introduced the idea of reinforcement learning from AI feedback (RLAIF), which used LLMs labeled preferences in conjunction with human labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from LLMs [29, 19, 50] . These works demonstrate that LLMs can generate useful signals for reinforcement learning fine-tuning. However, leveraging the feedback from MLLMs for aligning text-to-image generative model or for text-to-video generative model is less explored. 
conclusion: 
 In this paper, we identify the current obstacles faced in aligning text-to-video generative models, i.e., lacking of large-scale preference datasets and reward models specifically tailored for videos. Thus, we introduce VIDEOPREFER and VIDEORM to address the aforementioned issues. Experimental validation confirms that GPT-4 V can act as a human-aligned preference annotator. We utilized it to label 135K video preference annotation, forming VIDEOPREFER. Based on this, we trained a videospecific reward model, VIDEORM. Extensive analytical usage has demonstrated the effectiveness of both VIDEOPREFER and VIDEORM.A Datasets Processing T2VQA-DB [18] . The T2VQA-DB dataset only comprises a single overall preference evaluation score for each data item. We randomly selected 2.3K samples to serve as the final test benchmark.TVGE [56] . The TVGE dataset comprises preference evaluations from 2 aspects, namely: text alignment and video quality. We just simply average the scores across these two aspects to obtain a final overall evaluation score as with the aforementioned datasets. We randomly selected 15K samples to serve as the final test benchmark.VBench [15] . The VBench dataset comprises preference evaluations from 16 aspects, namely: subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, imaging quality, object class, multiple objects, human action, color, spatial relationship, scene, temporal style, appearance style, overall consistency.Due to the fact that the videos and prompts evaluated for preference in each aspect are largely non-overlapping, we cannot simply average the scores across all aspects to obtain a final overall evaluation score as with the aforementioned datasets. Instead, we mixed all samples from the 16 aspects and randomly selected 10K samples to serve as the final test benchmark. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our work supports the alignment of generative models with human values, and its societal impact is discussed in the introduction.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our contribution does not include new datasets or pre-trained models that pose a risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Code that we derive from earlier work is properly licensed and referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: Our experiment solely involves measurement and does not entail behavioral manipulation; therefore, we did not apply for IRB approval. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Table 5 , we present the sources and distribution of the videos in our VIDEOPREFER. We find that VIDEOPREFER comprises a diverse range of video sources, including videos generated by state-of-the-art text-to-video models as well as real videos. This extensive variety of video sources enhances the robustness and generalization capabilities of VIDEOPREFER.We visualize the distribution of preference annotations across two evaluation aspects (promptfollowing an video-quality) in VIDEOPREFER in Figure 9 . We find that the overall score distribution of the dataset is close to the normal distribution for the Prompt-Following evaluation perspective.From the perspective of Video-quality evaluation, the overall distribution of data sets tends to be low, which indicates that the generation effect of the existing text-to-video model is not satisfactory.Besides, we also visualize some data examples in VIDEOPREFER at Figure 10 . Preference Instruction for Prompt-Following Prompt-Following: You are an AI assistant programmed to assess videos with impartial and balanced standards.A video has been created based on a piece of text. Your task is to evaluate how well The content of the video aligns with the original text provided as ("Input") The video evaluation is based on sampled frames shown in sequence.Scoring: Rating outputs 1 to 5: Video-Quality: You are an AI assistant trained to impartially assess temporal consistency quality, dynamic quality and aesthetic quality in videos. A video has been created based on a piece of text. Your task is to analyze the quality of this video based on the following guidelines and provide a comprehensive evaluation. Scoring: Rating outputs 1 to 5:1. Bad: blurry, underexposed with significant noise, indiscernible subjects, exhibits significant inconsistencies and noticeable discrepancies in appearance of subjects. 2. Poor: Noticeable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects, suffers from noticeable issues in maintaining uniformity of subjects and backgrounds. 3. Fair: In focus with adequate lighting, dull colors, decent composition but lacks creativity. Subjects and backgrounds maintain a reasonable degree of uniformity throughout most of the video, with only minor discrepancies. 4. Good: Sharp, good exposure, vibrant colors, thoughtful composition with a clear focal point. Good video dynamics and temporal consistency. 5. Excellent: Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact. perfect temporal consistency and excellent dynamics. Please present your assessment as follows: Output Rating: [Provide the rating] Rationale: [Explain the reason for your rating in concise sentences] - ---------------- Justification: The paper includes our motivation, preference data construction details, experimental settings, quantitative experimental results, and qualitative visual examples that reflect and justify the claims in our abstract and introduction.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://huggingface.co/models?pipeline_tag=text-to-video 


Decision: The paper presents a dataset and a learned reward model for video preference to help align text2video generation models. Using GPT-4V, authors automatically label a large dataset of videos to learn this preference model, and fine-tune text-to-video models with this proposed preference model showing improvements in alignment with human feedback through quantitative/human evaluations and ablation experiments.

The paper was reviewed by four expert reviewers, all of who leaned positively towards accepting the paper. ACs concur with the reviewer consensus, and believe this is an important field of study for further improvements in text2video models. Authors must incorporate additional discussions and experimental results shared/promised during the rebuttal phase into their final paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a new dataset called VideoPrefer, a collection of (simulated) human preferences on videos conditioned on certain language prompts. VideoPrefer utilizes GPT-4v as an automatic reward assessor, which contains videos from both machine-generated and real-world curated videos. The work then utilizes the collected preference as a reward to learn a reward model named VideoRM, constructed on top of a prior model HPS v2, extending its capabilities to temporal dimension and thus can present reward modeling in videos.
strength_weakness: strength_weakness: - While I appreciate the Table 2’s analysis on TVGE dataset among the tested video assessor candidates, solely performing statistical analysis on the VideoPrefer dataset is still required. People would be relying on the VideoRM which is trained on the VideoPrefer data and it remains questionable how faithful the dataset is.
- Lack of some analysis between real-world and generated videos. Under the same prompt, are real-world videos always or more likely to be better? What heuristics can people use to expand beyond the collected video sets?
- Is it unclear how text prompts are aligned with real-world videos in Section 2.2.
- Lack of rigorous statistical analysis on the curated video prompts. How diverse are they? What is the type-token ratio? What are the top frequent predicates and affected entities? What are the genres? It is hard to understand how challenging and/or how underlying bias would have affected the video generation process from this curated resource.
- Why only consider extension of HPS v2 as the base reward model? There are plenty of good video-language models that can be potentially useful (such as [1]) to be adapted as a reward model. If the work can show more unbiased learning from the dataset solely with video-based preference models, it would strengthen the work more.

[1] Sung, Yi-Lin, Jaemin Cho, and Mohit Bansal. "Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks." CVPR 2022.
questions: questions: - There are some minor typos, for example, L84 the “K” is already a thousand there. Please be more mindful when writing.
- What or who is deciding the “win” in Figure 2? The GPT-4v assessor?
limitations: limitations: - The limitations of this work do not seem to be explicitly addressed.
paper_summary: paper_summary: The paper presents a dataset and a learned reward model to better align the generated outputs of text-to-video models with human preferences. Specifically, the authors make use of GPT-4V to provide preference scores for a large dataset of videos by learning from human feedback on a smaller set of videos. The authors then develop their proposed architecture for the reward model on top of this dataset to predict preference scores given prompts and videos. They fine-tune text-to-video models with their proposed reward model and show the improvements in alignment with human feedback through quantitative evaluations, ablation experiments, qualitative results, and user studies.
strength_weakness: strength_weakness: 1. Some key aspects of the dataset collection process are unclear.

    1.1. While GPT-4V has the best agreement with human feedback, it is still not very high (around 70%, according to the authors). Did the authors explore any prompt-tuning or other approaches to improve the agreement? Did the authors observe any potential patterns in the disagreement? Perhaps certain categories of videos have more agreement between GPT-4V and human feedback than other categories?

    1.2. When collecting data from video caption datasets, do the authors process the captions in some way to make them structurally and semantically similar to video generation prompts?

    1.3. Did the authors run any data filtering for discriminatory, harmful, or abusive content in the collected/generated prompts and videos?

    1.4. Did the authors consider various levels of complexity in the prompts, such as foreground and background details, number of objects, actions, interactions, etc.?

    1.5. How did the authors quality-check the videos generated for the proposed dataset? It is not fully clear how the proposed approach avoids a cyclic process, where improving video generation depends on a better reward model, which, in turn, requires generated videos of a certain quality.

    1.6. Can the authors please clarify the specific numbers regarding the dataset size? How did they go from 14K data items to 135K preference choices?

2. Some key aspects of the user study are unclear, making it hard to fully follow the results.

    2.1. Is five participants a sufficiently high number from which to draw conclusions? How many videos did each participant respond to? Was there a mix of non-experts and subject matter experts such as content creators or video editors? If so, were their responses segregated in any way?

    2.2. Did the authors check for participants' engagement, e.g., whether participants responded too quickly or too slowly, and whether any such responses needed to be discounted? Did the authors check for any recency bias or response drifts, where the participants may have responded differently to similar video content over time?
questions: questions: 1. In their ablation study, why do the authors consider up to 12 frames? Is this limit due to computational constraints?
limitations: limitations: Yes, but needs more details on how the authors filter for discriminatory, harmful, or abusive content in their collected dataset.
paper_summary: paper_summary: Recent advancements in text-to-video generative models, such as Sora, have shown impressive capabilities, generating significant interest for their potential applications. However, these models often rely on extensive datasets of variable quality, resulting in generated videos that may lack aesthetic appeal and fail to accurately reflect the input text prompts. To address this, leveraging Reinforcement Learning from Human Feedback (RLHF) aims to align model outputs with human preferences, but the high costs of manual annotation have limited comprehensive preference datasets. This study investigates the efficacy of Multimodal Large Language Models (MLLMs) for generating annotations, finding a high degree of concordance with human judgments. Building on this, the study uses MLLMs to perform fine-grained video preference annotations, creating VIDEOPREFER, a dataset with 135,000 annotations. Utilizing this dataset, the authors introduce VIDEORM, the first general-purpose reward model for video preference in the text-to-video domain. Comprehensive experiments validate the effectiveness of both VIDEOPREFER and VIDEORM, representing a significant advancement in aligning text-to-video models with human preferences.
strength_weakness: strength_weakness: 1. **Representativeness of Generated Video Quality**: Over 98% of the videos in VIDEOPREFER are generated using open-source models (as indicated in Table 5). This high percentage raises a crucial question about the extent to which the quality rankings of these synthetic videos can represent the quality rankings of real videos. It is important to clarify that the concern is not whether GPT-4 V aligns with human judgment, but rather how well the preference dataset and the resulting reward model genuinely capture and represent real-world video quality.

2. **Quality Demonstration through Video Demos for VIDEOPREFER**: The quality of the VIDEOPREFER dataset cannot be fully assessed through the images shown in Figure 10 alone. To better judge the dataset's quality, it is recommended that the authors provide 5-10 videos from the dataset. These demos would offer a clearer and more comprehensive understanding of how well the dataset captures human video preferences and the overall quality of the included videos.

3. **Assessment of VIDEORM through Comparison Demos**: Figure 11 showcases images selected by VIDEORM, but these images alone are insufficient to evaluate the actual quality of the selected videos. To properly assess the effectiveness of the VIDEORM reward model, the authors are recommended to provide 3 comparison sets of the selected videos by different reward models. These demos would allow for a more accurate judgment of the reward model's value and its ability to select high-quality videos that align with human preferences.

4. **Effectiveness of Fine-Tuning with VIDEORM Demonstrated through Demos**: The images in Figure 12 illustrating the results of the VIDEORM fine-tuned model do not adequately convey the actual video quality. To better demonstrate the effectiveness of fine-tuning with the proposed reward model, the authors are recommended to present 3 sets of comparison videos. These demos would offer a concrete demonstration of how fine-tuning with VIDEORM improves video generation quality, thus providing a more tangible proof of the model's effectiveness.

By addressing these points and providing the suggested video demos, the study could offer a more comprehensive and transparent evaluation of the proposed methods and datasets, significantly enhancing the understanding and validation of the results presented.
questions: questions: Please refer to the weaknesses.
limitations: limitations: This paper discusses the limitations of choosing hyperparameters, which significantly influence the annotation accuracy of GPT-4 V. In addition, it would be better to also discuss the limitations of VIDEOPREFER and VIDEORM.
paper_summary: paper_summary: Recent text-to-video models like Sora show potential but suffer from poor video quality and misalignment with text prompts due to variable dataset quality. This study addresses these issues by leveraging Reinforcement Learning from Human Feedback (RLHF) to align outputs with human preferences. Due to the high costs of manual annotation, Multimodal Large Language Models (MLLMs) were used for video preference annotations, demonstrating strong concordance with human judgments. This led to the creation of VideoPrefer, containing 135,000 annotations, and the development of VideoRM, a reward model for video preferences. Experiments confirm the effectiveness of both VideoPrefer and VideoRM.
strength_weakness: strength_weakness: Weakness:

1. Although existing research has demonstrated that GPT-4 can be used for data annotation, it is essential to perform a sample check of the annotated data to assess its quality. Relying solely on the literature to support the use of GPT-4 for annotation without conducting thorough spot checks and corrections undermines the reliability of the annotated content. Ensuring the accuracy and quality of annotations through systematic verification is crucial for maintaining the integrity of the dataset.

2. Furthermore, the evaluation of generated videos should consider perspectives from multiple roles. Incorporating role-playing prompts to capture diverse viewpoints would more accurately reflect the varied opinions and perspectives that different individuals may have regarding the same video. Relying on a single prompt template for evaluation is limiting and does not adequately represent the range of possible reactions and insights.

3. From the content of Algorithm 1's pseudocode, it appears that there is a lack of task-specific algorithmic innovation in the application of reinforcement learning for fine-tuning. The approach seems to follow standard practices without introducing novel techniques tailored to the specific challenges of the task at hand.

4. While the proposed VIDEORM model demonstrates strong performance across various metrics, the paper lacks interpretability experiments. Including such experiments would provide deeper insights into the model's decision-making process and enhance the overall understanding of its effectiveness.
questions: questions: Weakness:

1. Although existing research has demonstrated that GPT-4 can be used for data annotation, it is essential to perform a sample check of the annotated data to assess its quality. Relying solely on the literature to support the use of GPT-4 for annotation without conducting thorough spot checks and corrections undermines the reliability of the annotated content. Ensuring the accuracy and quality of annotations through systematic verification is crucial for maintaining the integrity of the dataset.

2. Furthermore, the evaluation of generated videos should consider perspectives from multiple roles. Incorporating role-playing prompts to capture diverse viewpoints would more accurately reflect the varied opinions and perspectives that different individuals may have regarding the same video. Relying on a single prompt template for evaluation is limiting and does not adequately represent the range of possible reactions and insights.

3. From the content of Algorithm 1's pseudocode, it appears that there is a lack of task-specific algorithmic innovation in the application of reinforcement learning for fine-tuning. The approach seems to follow standard practices without introducing novel techniques tailored to the specific challenges of the task at hand.

4. Although the proposed VIDEORM model demonstrates strong performance across various metrics, the paper lacks interpretability experiments. Including such experiments would provide deeper insights into the model's decision-making process and enhance the overall understanding of its effectiveness.
limitations: limitations: The preference annotation in this study relies entirely on GPT-4, which is not advisable. While manual annotation is costly, the authors could employ individuals from diverse fields, genders, and age groups to annotate preferences according to a standardized procedure from their unique perspectives. These annotated samples could then be used as prompts for GPT-4 to perform further annotations, potentially improving the quality. Additionally, designing a richer set of prompt templates to capture preferences from multiple perspectives would enhance the dataset's inclusivity and robustness. The current approach lacks the diversity and comprehensive representation necessary for a truly inclusive dataset.

Comments: title: Response to Reviewer h97w
Response to Reviewer PNsh
Thanks
Response to Reviewer qjMG
Response to Authors
comment: Dear Reviewer h97w,

Thank you for reviewing our work and providing valuable feedback. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)

Best regards,

All Authors
Dear Reviewer PNsh,

Thank you for your detailed feedback.

We will incorporate the following content into the latest version of the paper to better address the discussion on the presence of bias in the generated data and trained models:

**Preference Bias for Specific Videos**: We have analyzed whether there is a bias in our dataset favoring real videos over synthetic ones. We found that MLLMs tend to assign significantly higher preference scores to real videos compared to synthetic ones (93% versus 70%). We appreciate your kind reminder and suggestions, and we will include further detailed bias analysis in the updated version of the paper. This will include examining potential biases towards video length, as well as any ethical biases related to gender, race, etc., to enrich our research.

**Background of Human Annotators**: Among the five participants, three were experts (in image aesthetics, video aesthetics, and image quality), and two were non-experts, ensuring a more comprehensive and accurate evaluation. During the evaluation process, the participants did not engage in any discussion or communication. The choice of combining experts and non-experts for annotation was intended to achieve unbiased labeling, avoiding the introduction of specific technical biases from having only experts or the lack of professionalism from having only non-experts. We will also add this information to the updated version of the paper.

**Further Bias Analysis**: We will conduct additional analyses to determine if there are specific biases in our dataset or model related to gender, race, or ethnicity. The conclusions from these analyses will be included in the latest version of the paper, providing a detailed reference for future research and adding value to our subsequent studies.

---

If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)

Best regards,

All Authors
Thank you for the further discussion and the detailed responses to the queries raised. At this point, I have no additional questions. I am confident that by incorporating these new discussions and additional experimental results, the quality of the final version will be significantly enhanced. I look forward to seeing the effects in the final version. Best of luck!
Dear Reviewer nZ7n

We would like to sincerely thank you for your feedback and suggestions :). We will carefully revise our paper, addressing all issues (e.g., typos and adding statistical analysis on the curated video prompts) to make the paper clearer and more complete.

Best regards,

All Authors
Thank you for providing us with detailed feedback. We have greatly benefited from your insights. Below are our responses:

>**Q1**: I am not questioning whether GPT-4 V aligns with human judgment, but rather whether the trained VideoRM has the capability to do so. The high percentage of synthetic videos in the dataset may still limit the generalizability of the reward model to practical applications. Scaling up the dataset to include more real video data might make the resulting reward model more robust in guiding the model to generate more realistic videos.

**A1**: All authors agree with your suggestions. We plan to release the second version of VideoPrefer in the future. In this upcoming version, one of our primary improvements will be to increase the proportion of real videos in the dataset, thereby making the reward model more robust. We outline the specific steps as follows:

**Step 1**: Use well-established event detection models (e.g., [mmaction2](https://github.com/open-mmlab/mmaction2) to segment existing large-scale real video datasets (such as [Kinetics-700](https://paperswithcode.com/dataset/kinetics-700)) into multiple smaller video clips.

**Step 2**: Generate captions for the obtained real video clips using a mature and reliable large model, e.g.,  GPT-4V.

**Step 3**: Input the generated captions into randomly selected video generation models to produce corresponding synthetic videos.

**Step 4**: Combine the captions, real video clips, and the generated synthetic videos to create new VideoPrefer data examples.

By implementing these steps, we will be able to increase the proportion of real videos in the dataset.

---

>**Q2**: Secondly, although the decision to enhance more video demonstrations in a future version of the paper does not help with the current review, I still hope the authors pursue this to further improve the quality of the paper.

**A2**: Thank you for the reminder. We will certainly include additional video demonstrations across various aspects (such as the assessment of VideoRM or the results of fine-tuning) in the latest version of the paper, as per your request, to further enhance the quality of the paper and improve the visualization of the dataset and experimental results.

---

>**Q3**: regarding the discussion on hyperparameters, are there any strategies to mitigate their impact? 

**A3**: One approach we have considered is to mitigate the impact of hyperparameter selection by conducting multiple annotations for each sample using GPT-4V. Specifically, we would annotate each sample N times, with the hyperparameters randomly sampled for each annotation. The final preference score for the sample would be the average of these N scores. We believe this method could help reduce the influence of hyperparameter choices to some extent. Of course, this is a very interesting research problem in itself, and we will continue exploring possible solutions to further improve the accuracy of the annotation.

---
>**Q4**:  Additionally, the summarized discussion of limitations in the response is appreciated, but it feels somewhat superficial. Could you provide more concrete examples or a deeper exploration of the potential limitations?

**A4**: Firstly, inspired by your feedback, we recognize a potential limitation of our dataset: the relatively low proportion of real videos, which might affect the robustness of the trained VideoRM (a specific solution for this can be found in **A1**). Another possible limitation we have identified is the lack of a comprehensive bias assessment for the VideoPrefer dataset in our current research. For instance, it remains unclear whether GPT-4V exhibits a noticeable preference for certain types of videos. Although we have analyzed the preference between real and synthetic videos within VideoPrefer and found that GPT-4V shows a strong preference for real videos (93% vs. 70%), further detailed analysis is needed. This includes assessing whether there is a preference for videos generated by specific models or for videos of certain styles. We plan to conduct more in-depth analyses in future research to address these potential limitations.

---

**If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)**

Best regards,

All Authors
I thank the authors for their detailed rebuttal, which addresses all my concerns. I maintain my original recommendation for acceptance and have slightly raised my score.
Thanks for the responses.
Some of the points addressed my doubts, however, the original manuscript was a bit far from publication ready.
I retain my score.
Thank you for the detailed response. While I appreciate the effort to address my concerns, several issues remain that warrant further improvement.

Firstly, regarding the representativeness of generated video quality, the analysis provided on the preference scores for real videos is informative, but it does not fully address the core concern. I am not questioning whether GPT-4 V aligns with human judgment, but rather whether the trained VideoRM has the capability to do so. The high percentage of synthetic videos in the dataset may still limit the generalizability of the reward model to practical applications. Scaling up the dataset to include more real video data might make the resulting reward model more robust in guiding the model to generate more realistic videos.

Secondly, although the decision to enhance more video demonstrations in a future version of the paper does not help with the current review, I still hope the authors pursue this to further improve the quality of the paper.

Lastly, regarding the discussion on hyperparameters, are there any strategies to mitigate their impact? Additionally, the summarized discussion of limitations in the response is appreciated, but it feels somewhat superficial. Could you provide more concrete examples or a deeper exploration of the potential limitations?

In conclusion, while the rebuttal addresses some concerns, there remain areas where the paper could be improved. I encourage the authors to consider these points carefully in the revision process.
Thank you for your response and detailed explanations. I have updated my rating to 8.