Boosting Text-to-Video Generative Model with MLLMs Feedback
Authors: Xun Wu
Shaohan Huang
Guolong Wang
Jing Xiong
Furu Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-video generation
alignment
reward model
multimodal large language model
Primary area: diffusion_based_models

Abstract: Recent advancements in text-to-video generative models, such as Sora, have showcased impressive capabilities. These models have attracted significant interest for their potential applications. However, they often rely on extensive datasets of variable quality, which can result in generated videos that lack aesthetic appeal and do not accurately reflect the input text prompts. A promising approach to mitigate these issues is to leverage Reinforcement Learning from Human Feedback (RLHF), which aims to align the outputs of text-to-video generative with human preferences. However, the considerable costs associated with manual annotation have led to a scarcity of comprehensive preference datasets. In response to this challenge, our study begins by investigating the efficacy of Multimodal Large Language Models (MLLMs) generated annotations in capturing video preferences, discovering a high degree of concordance with human judgments. Building upon this finding, we utilize MLLMs to perform fine-grained video preference annotations across two dimensions, resulting in the creation of VideoPrefer, which includes 135,000 preference annotations. Utilizing this dataset, we introduce VideoRM, the first general-purpose reward model tailored for video preference in the text-to-video domain. Our comprehensive experiments confirm the effectiveness of both VideoPrefer and VideoRM, representing a significant step forward in the field.

Boosting Text-to- Video Generative Model with MLLMs Feedback
Xun Wu',
Shaohan Huang 15 Guolong Wang", Jing Xiong Furu Wei
Microsoft Research Asia, 2 University of International Business and Economics
3 "Te University of Hong Kong
xunmu@microaott.com shachanh@microsofft
fuwei cmicrosoft.com
Abstract
Recent advancements text-to-video generative models, such as Sora [3) have showcased impressive capabilities. These models have attracted significant interest for their potential applications. However, they often rely on extensive datasets variable quality, which can result in generated videos that lack aesthetic appeal and do not accurately reflect the input text prompts, A promising approach to mitigate these issues is leverage Reinforcement Learning from Human Feedback (RLHF), which aims to align the outputs of text-to-vided models with human preferences. However, the considerable costs associated with manual annotation have led to scarcity of comprehensive preference datasets. In response to this challenge, our study begins by investigating the efficacy of Multimodal Large Language Models (MILMs) generated annotations in capturing video preferences, discovering a high degree of concordance with human judgments. Building upon this finding, we utilize MLLMs to perform aineggraneed video preference annotations across two dimensions, resulting in the creation of VIDEOPREFER, which includes 135,000 preference annotations. Utilizing this dataset, we introduce VIDEORM, the first general-purpose reward model tailored for video preference in the text-to- eext-to-vidoo domain. Our comprehensive experiments confirm the effectiveness of both VIDEO- PREFER and VIDEORM, representing significant step forward the field.
1 Introduction
Diffusion models have significantly enhanced the quality of generation across various media formats, including images 30 recent advancements in text-to-video diffusion models, exemplified by Sora [4, have achieved remarkable success in generating high-quality videos from textual prompts. However, despite recent progress, the visual fidelity of generated videos still presents opportunities for refinement (38 Video generation poses greater challenges than image generation, as it entails modeling higher-dimensional spatio-temporal output space while remaining conditioned solely textual prompt. Consequently, existing rext-to-vided generative models often produce results that are visually unappealing and inadequately aligned with the provided textual prompts.
55 and videos 13 22 In the context of text-to-video generation,
To address these limitations, pivotal solution is Reinforcement Learning from Human Feedback (RLHF), which has proven its efficacy in text-to-in -omage--mage diffusion models [51, 2 26). These methods aim fine-tune diffusion model maximize a reward function corresponding to specific aspects of image quality or alignment with text prompts. Despite this, when Reward Model trained in the image domain is transferred the video domain, can exhibit significant disparities with the objectives of video optimization (e.g., InstructVide (531). Training a reward model directly in the video domain is becoming increasingly urgent, but the scale of preference datasets the video field is small (see in Table
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table Statistics of existing preference datasets for rext-to-video generative models. denotes annotated by human while denotes annotated by GPT-4 V.
Table 2: Correlations between MLLMs and hu-
man judgment on text-video alignment on TVGE
datasets [56]. denotes image-based MILMs
while denotes Video-based MILMS.
Preference Choices
Model
Spearman
Kendall
Acc (3)
Dataset
Prompts
Videos
Viden-IJ 64
0.288
0.206
mPLUG-OWEA
0.285
VBench 115
1K
4K
44K
InstruciBLIP 10
0.322
0.246
54.33
TVGE (56
0.5K
2.5K
2.5K
mPLUGGOWLL2 12
0.358
0.257
53.36
T2VQA-DB 18
IK
45K
45K
Gerisi pro Vision
0.3921
0.2933
64.71
0.3139
VIDEOPREEER!
14K
54K
135K
GPI-4
0.486
0.360
69.65
Recognizing the importance of addressing these challenges text-to-video generative models, we first construct a large-scale fine-grained preference benchmark by utilizing MILMS as annotators, namely VIDEOPREFER. VIDEOPREFER contains following strength: (1) VIDEOPREFER is the largest open-source video preference dataset, containing 135,000 preference choices (see in Table [ (2) Different from existing video preference datasets, VIDEOPREFER contains true video captured by human, making VIDEOPREFER more generalizable.
(3) VIDEOPREFER is annotated using Multimodal Large Language Models, which is cost-effective and easily scalable.
Based VIDEOPREFER, we release the first seneral-purpose text-to-video preference reward model, V.DEORM. Unlike the preference rewards in the image domain used by previous methods for video generation alignment [53], our VIDEORM automatically captures the temporal information in videos, enhancing the modeling of quality assessment and alignment. Furthermore, we investigate the alignment of text-to-video generative models using VIDEORM and conduct extensive experiments that demonstrate its efficacy as reward model and metric for video alignment. This significantly improves the generation quality of video generation models across multiple aspects. Our main contributions are:
We systematically identify the challenges for text-to-video human preference annotation. Con- sequently, we employ MILMS for preference annotation and construct the largest and most comprehensive video preference dataset, VIDEOPREFER.
We systematically identify the lack of ceneral-purposs text-to-video preference reward model and propose VIDEORM, outperforming existing reward models for text-to-vider alignment.
Extensive experimental results validate the effectiveness of both the VIDEOPREFER and VIDE- ORM. Furthermore, through detailed discussions, we demonstrate that MLLMs are effective and cost-effective annotators for video preferences, revealing the promising future prospects of RLAIF in the domain of video preference alignment.
2 VIDEOPREFER
We introduce VIDEOPREFER, a large-scale, fine-grained video preference dataset constructed by collecting feedback from multimodal large language model annotators. In total, VIDEOPREFER contains 135K pairs of binary preference choices for 54K videos. In [2 we provide evidence demonstrating that GPT-4 V is a numan-aligned preference annotator in the video domain. The construction pipeline of VIDEOPREFER is introduced in $2.2 Detailed analysis of the statistics of VIDEOPREFER can be found in Appendix $ B.1
2.1 Why GPT-4 V(ision) can act human-aligned preference annotator?
GPT-4 has already demonstrated annotation performance aligned with human consistency in text- to-image generation 7 46 44 To further validate GPT-4 4 V's ability to provide reliable preference annotations in the video domain, we used different MILMs to annotate a subset of the TVGE dataset and calculated their correlation with the ground truth annotations, e.g., accuracy and kendall. The results are shown Table 2 We find that (1) GPT-4 V's annotation accuracy and correlation are best among all MILMs. (2) The accuracy of GPT-4 V is 69.666666%. which is very close to the previously reported agreement rate between qualified human annotators (approximately 70% [9] 46
This demonstrates that GPT-4 V is reliable annotator.

---

2.2 Construction Pipeline
The construction pipeline of VIDEOPREFER can be split into following three steps:
Step-1: Prompts Collection. We collect prompts from VidProM 44 which contains 1.67 million unique text-to-video prompts. as well as from two video-captioning benchmarks: ActivityNet(5] and MSR-VTT(49) For VidProM, we just randomly sample 12.9k prompts from its corpus. For video-captioning benchmarks, we directly utilize the provided video segment captions from the original dataset as prompts. For instance, in the ActivityNet [5], we utilize the text provided in caption corpus, comprising 0.9K thousand text instances corresponding distinct video segments. Consequently, we obtain total of 14K prompts, constituting the prompt set for our VIDEOPREFER,
Step-2: Video Collection & Generation. We generate videos with text-to-video models and collect real-world videos based on prompts collected in Step-1 for preference annotation. Through these two ways, we obtained 54K video candidates:
Model-generated videos. We selected the top-ranked text-to-video generation models on Hugging- (e.g.. ModelScopeT22 (38), well as the open-source state-of-the thr text-to-video models (e.g., Open-Sora (57)), to constitute our video generation model pool. Details regarding these models, as well the proportions and resolutions of the generated videos, can be found in Table For each prompt, we randomly sample models from the model pools, along with class-free guidance scales, generate four different corresponding videos. This approach results in a high degree of diversity, facilitating the training of a more generalizable and comprehensive preference reward model.
Real-world videos. Given the substantial quality disparity between videos generated by existing text-to-video generation algorithms and real videos, we also incorporate real-world videos to enhance the generalizability and diversity of our VIDEOPREFER, Specifically, we incorporate video segments from the two datasets introduced in Step-1 (ActivityNet [5] and MSR-VTT 44 as one of the four candidate videos, with the remaining three being generated above.
Step-3:
Preference Generation. Due to the prohibitively high cost of manual annotation, we utilize the ofate-of-thh-aar multimodal large language models (MLLMs), i.e., GPT-4 V to annotate video preferences, enabling extensive and fine-grained annotation at scale. The reliability of GPT-4 V as annotators is thoroughly discussed in & 2.1 Specifically, for each prompt and its corresponding four video candidates, we employ GPT-4 to assign preference scores on 1-to-5 Likert scale to each and Video Quality. Lower scores indicate lower
video candidate for two aspects: preference and vice versa. Finally, we obtain 135K preference choices. Detailed input instructions used for GPT-4 V's annotation can be found in the Appendix B
By performing the above three steps, we finally have 14K data items in VIDEOPREFER, while each data item contains textual prompt and four corresponding generated videos. For each video, there are preference annotations for two aspects: Prompt-Following and fideo-Quality provided. More details of VIDEOPREFER, e.g., visualization of example data item (see in Figure TO), can be found in Appendix B
3 VIDEORM: A General-Purpose Video Preference Reward Model
Based on VIDEOPREFER, we implement the reward model training and derive the first general- purpose video preference reward model, VIDEORM. We detail the architecture of VIDEORM in & 3.1 its optimization objectives in $ 3.2, and the methodology for fine-tuning video generative
models with VIDEORM in 3.3
3.1 Architecture
Existing text-to-video alignment work (53) adopts HPS v2 [45] as the reward model, which is fine- tuned from CLIP 277 and optimized upon a large image preference benchmark and demonstrates state-of-the-art preference evaluation capabilities for text-to-image generation.
However, we contend that the direct application of HPS v2 may lead to a one-sided assessment of video preferences, as it lacks the capability to evaluate the overall attributes of video, such

---

as temporal coherence and dynamics. Nevertheless, given the strong correlation between video preferences and the preferences of individual frames, the proficiency of HPS v2 45 in evaluating single-frame image preferences can still aid in enhancing the assessment of video preferences. Thus, to achieve better evaluation for video preferences, we modify the structure of HPS by integrating several temporal modeling modules, to develop a specialized reward model VIDEORM M for the video domain.
The full architecture of VIDEORM is shown in Fig- Specifically, inspired by recent advance- ments 39 21 in enhancing the temporal modeling
ure I
capabilities of the CLIP [27], we add two kinds of
temporal modeling modules into HPS v2 to modeling
miee dall
videos:
Thane
Temporal Shift 20), a parameter-free module which shifts part of the feature channels along the temporal dimension and facilitates information exchange be- tween neighboring input frames. Following [39], we insert the module between every two ViT Layers of the image encoder of HPS v2.
Figure 1: Overview of VIDEORM. By incor- porating temporal modeling modules, VIDE- ORM is capable of not only capturing the preference scores of individual frames but also modeling the temporal features of the video, thereby better evaluating the overall preference score of the video.
The sequence of frame- wise features extracted by the image encoder then fed into a temporal transformer to modeling the tem- poral features of the video.
Temporal Transformer.
3.2 Optimization
During the optimization process of VIDEORM, we freeze the text encoder and VIT layers contained in the image encoder to retain the single-frame preference modeling capability of HPS v2, while only optimizing the temporal transformer module. Thus, VIDEORM achieves the capability to model video preferences from both the perspectives of individual frames and temporal dynamics, allowing for a more comprehensive understanding of vidco content.
Similar to previous works 36 25 to train VIDEORM on VIDEOPREFER, we first average the scores of all 16 aspects for each video candidate, obtaining the final preference score for that video candidate. Then we formulate the preference score for each pair-wise video candidates as rankings. Thus, for each data item (consisting one prompt and its corresponding four video candidates V1, V2, V3. and v4), we get at most C. comparison pairs if there are no ties between two video candidates. For each comparison, ++ is better and v worse (v+ > v-), the loss function can formulated as:
loss(6) - (tt [log (or (Ro (T,v+) Ra (T, (T, )))]
where Ro (T, v) is a scalar value of preference model for prompt T and generated video v.
3.3 Fine-tuning Text-to-Vides Models with VIDEORM
Current exploration of reward reinforcement learning algorithms for fine-tuning text-to-video gen- erative models is quite limited. The only related work is InstructVideo (53), which utilizes an image-domain reward model (HPS v2 45
fine-tune text-to-video models.
Consequently, InstructVideo may have the following shortcomings: (1) Due to the inherent gap between images and vidcos, directly using image-domain reward models cannot accurately calculate the reward for generated videos, leading to visual artifacts such as structural twitching and color jittering [53] (2) Additionally, Instruct (53) calculates the reward value for all frames selected from generated videos throughout the full DDIM 34] sampling procedure, making the fine-tuning process highly sample inefficient.
VIDEORM specifically designed for the video domain. It initialized with the weights of the best image-domain reward model and trained on video preference datasets, utilizing temporal modeling mechanisms (e.g., temporal transformers) to capture temporal features of videos. This enables to evaluate the reward score of generated videos more effectively from both individual frame and

---

temporal perspectives, Additionally, unlike the maage-domain reward model used in Instruct Video, which takes single video frames as imput, VIDEORM processes the entire video as imput. Therefore, our approach is more computational efficient and can directly leverage various effective image-domain reward reinforcement learning algorithms (such as DRaFT g without the need to balance rewards between frames generated at each step, as required by InstructVideo.
Based on the above, we integrated VIDEORM into the image-domain DRaFT design a reward reinforcement learning algorithm suitable for text-to-video generative models, named DRaFT-V. The specific algorithm details are shown in Algorithm [ DRaFT-V truncates the backward pass, differentiating and computing reward score from VIDEORM through only the last K steps, making the full fine-tuning process more efficient.
algorithm and
Algorithm DRaFT-V: Reward Reinforeement Learning for Fine-tuning Text-to- -idco Models with VIDEORM
In implementation, rather than fine-tuning the full set of model parameters, we follow [53] to adopt LoRA [4] to further accelerate fine-tuning and cir- cumvent the issue of computational intensity as well as the risk of catastrophic forgetting associated with the reward loss in diffusion models.
1: Dataset: Prompt set y {y1, y/2, M..
2: Pre-training Dataset: Text-Video pairs dataset D = {(t, Vi V:)....(ta. vn.)]
3: Input: Text-to-video models Y with pre- trained parameters Bu. VIDEORM R, reward- to-loss map function Text-to-vides models pre-training loss function v,, reward re-weight scale X
4 Experiments
4:
Initialization: The number of noise scheduler time steps T', and the truncating step K
We conduct extensive experiments to validate the ef-
fectiveness of VIDEOPREFER and VIDEORM. We first train VIDEORM and evaluate it on existing
5:
for y y and (t,, va) E D do
6:
Lere + (t: Vs)
cuman-preference benchmark (....)) Next, we fine- tune existing text-to-video diffusion models for align- ing human preference by utilizing DRaFT-V with VIDFORM 4.2). Finally, we present ablation stud-
7:
wa < 10, // Update To, using fere
8:
FT NO 7) // Sample noise as latent.
9:
for j = T T...., do
10:
if > K then
11:
no grad: +j-1 + {mş}
ies ($ 4.3
12:
else
13:
with grad: 25-1 <- To, {a;}
14:
3o f 2, // Predict the original latent by noise scheduler
4.1 Task 1: Reward Modeling
15:
of c 20 // From latent image
Setup. Based on VIDEOPREFER, we develop VIDE- re-
16:
Ereward < Xo(r(u +{)) // Reward
oRM, an advanced open-source ward model that provides preferences for generated videos. Specifically, we train three versions of VIDE- ORM. (1) Firstly, to validate the effectiveness of VIDEOPREFER, we average the preference scores in each aspect of VIDEOPREFER to get final prefer- ence score, and train VIDEORM-- V with the merging version of VIDEOPREFER. (2) Then,
loss
17:
end if
18: end for
19: 0.+1 + O. // Update Yo, using Creward
20: end for
train VIDEORM-H on several open-source human crranec preference datasets listed in Table I (3) Finally, to build a stronger R.M for text-to-video generation, mix these open-source human- craffed preference datasets with VIDEOPREFER to train VIDE- ORM. The details for dataset processing can be found in Appendix A All VIDEORM series models trained in half- feff-precision 8 x 32GB NVIDIA V100 GPUs, with learning rate of le-5 and batch size of 64 in total. We set the input frames N = 8.
Compared Baselines. Due to the lack of reward models in the video domain, we compare VIDEORM with ofatc-of-the-art reward models from the image domain, i.c., CLIP ViT-H/14 (27), ImageRe- ward [48), PickScore [17] and HPS v2 (45). For all these reward models from the image domain, we calculate the scores for all video frames and then take the average as the final reward score for the video.
Preference Accuracy. The preference prediction accuracy across test benchmarks from three human- crafted preference datasets are reported in Table As we can see, the VIDEORM series outperform baseline reward models by a large margin, indicating that VIDEORM series are the best open-source reward models for text-to- text-to-video domain. We also find that VIDEORM-V which does not train on any open-source video preference datasets also surpasses all other baselines. This result validates the high quality of VIDEOPREFER enables strong out-of-distribution generalization and validate
5

---

0.5
Table 3: Pair-wise preference prediction accuracy human-crafted preference datasets. The Aesthetic Clas- sifier (simplified Aesthetic) makes prediction without seeing the text prompt. The best results are in blod and the
70
0.4
Rate
60
Reward
50
0.3
second are underlined.
40
Win Rate Reward
30
Model
TVGE [56]
VBench [15
T2VQA-DB [8
Avg
20
0.2
CLIP VIT-H/14
$7.3
52.7
52.1
54.0
4
8
Aesthetic 35
56.1
51.0
52.7
53.3
Number Samples
ImageReward 48
66.8
54,3
53.9
58.3
Figure 2: Best-of-n experiments on the
PiekSoore
64.7
53.9
61.2
59.9
55.7
59.3
T2VQA-DB [88] test benchmark. We
VIDEORM-H
72.8
60.2
64.3
65.8
sample generated videos and choose
VIDEORM-V
73.0
61.1
65,
66.4
the one with the highest reward.
VIDEORM
73.7
63.5
65.4
67.5
F-40-m
06
156
750
50%
25%
75%
100%
TVGE
56
VBench 15
T2VQA-DB 18
Figure 3: Win rates of VIDRORM compared to other reward models across three test benchmarks. On average, 72% CLIP, 67% to ImageReward, 62% PickScore and 65% to HPS v2.
the effectiveness of treating state- of-the-art MLLMs, i.e., GPT-4 4 V, as preference annotator for text-to-video generation domain.
Best-of-n Experiments. To verify that our VIDEORM could serve as good indicator of video generation quality, conduct best-of-n experiments on the T2VQA-DB [18] test benchmark. For each data item, which includes a prompt and ten corresponding videos, we calculate the reward score for each of the ten videos with VIDEORM. Thereafter, select the best-of- 1, 2, 8) responses and calculate their scores. The final results are presented at Figure 2 we find the win rate at the test benchmark increases proportionally with rewards, which validates that VIDEORM gives rigorous rewards that reflect the overall preference score.
Human Evaluation. To evaluate the ability of VIDEORM to select the more preferred videos among large amounts of generated videos, we produce human study. We randomly select 200 textual prompts from these three test benchmarks and generate 32 different videos for each prompt by utilizing ModelScope [38] Then, we perform different reward to select from those videos to get top3 results. After that, five annotators are asked identify which video
superior or if both of equal quality (denoted as "Tie') and we show the corresponding win rates against other reward models (45) at Figure Qualitative results can be at Figure [ Appendix. All of these results show that VIDEORM can select videos that are more aligned to text and with higher fidelity and avoid toxic contents.
4.2 Task 2: Fine-tuning Text-to-Vides Generative Models
Setup. Based on VIDEORM, we validate the effectiveness of DRaFT-V. We adopt the publicly available text-to-video diffusion model ModelScopeT22 (38) as base model, which is trained on WebVid10M with T= 1000 and is able to generate videos of x 256 x 256 resolution. We random sample 2K prompt-vided pair from the mixture of existing video preference datasets (TVGE (56). VBench [5] and T2VQA-DE (8] and VIDEOPREFER as the training data. Each group of experi- ment is trained in half pref--pre on 4 X 32GB NVIDIA V100 GPUs, with learning rate of Ie-5, batch size of 8, fine-tuning step of 400 and K adopted in DRaFT-V as 10 in total.
Compared Baselines. Since the only existing poward-based fine-tuning algorithm for next-to-vides models is InstructVided [53]. We compare our DRaFT-V with the the baseline text-to-video model (i.e., ModelScope 38) and instructVide implemented by ourselves. Note that in the aforemen- tioned algorithms, VistructVideo uses HPS v2 45 as the reward model. Additionally, to verify the effectiveness of using VIDEORM as reward model for fine-tuning video generation models,

---

Bose Min
2005
10%
75%
10055
06
755
90%
75%
1005
05
250
5075
755
1028
TVGE 55
VBench
15
T2VQA-DB
18
Figure 5: Win rates of text-to- text-to-video models fine-tuned with DRaFT-V compared to other baselines. Here baseline denotes the base text-to-video model without any fine-tunigg
10:
DRuFTTH
TYGE
0.71
66
Difoa 14
Accurect 50
73.23
21.00
23.25
22.59
121
60
72:2
73.00
TVGE test benchmark
Training Steps
Thaining Date: 00
Number of Input Frames
(b)
(c)
Figure 6: (a) Visualization reward model values change with the training steps for both DRaFT-V and DRaFT-H. (b) Evaluation result across three test benchmarks for the size of training data used in optimizing VIDEORM. (c) Ablation study for the number of input frames in VIDEORM.
introduced two comparison groups: (I) DRaFT-H, which uses the same algorithm as our DRaFT-V (i.e., DRaFT ()), but with HPS v2 the reward model. (2) Instruct Video V, which uses the same algorithm as InstructVideo 133 but with VIDEORM as the reward model.
Generation Quality Evaluated by Multiple Reward Models. At first, we visualize the evolution of model's generation quality across multiple reward models the model training steps increase when using our DRaFT-V at Figure 4 We observe that with the progress of training, all reward models show increasing trend, indicating that VIDEORM can serve as a reliable reward model to enable the generated videos align more closely with human preferences.
Reword Store
Traning Siepe
Figure 4: Evaluation results of the text-to-video model's gen- eration quality across multi- ple reward models when max- imizing scores from VIDE- oRM during the DRaFT-V fine-tuning process.
Human Evaluation. To evaluate the generative ability of text-to- video models fine-tuned with DRaFT-V and other compared base- lines, we produce human study. We randomly select 100 prompts from these three test benchmarks (TVGE (56). VBench (15] and T2VQA-DB [8],, and for each prompt, we generate videos using different fine-tuned models. Five annotators which video was superior or if both were of equal quality (denoted as "Tie') and we show the corresponding win rates at Figure We find that: (1) All of these results show that DRaFT-V can generate videos that are more prefered by human. (2) By comparing DRaFT-V with DRaFT-H, as well InstructVideo-V with InstructVideo, reward model for fine-tuning text-to-video models.
asked identify
demonstrate that VIDEORM is more effective
Besides, qualitative results presented at Figure 12 in Appendix We find that: (1) The base gener- ation model often fails to align videos with prompt descriptions, but fine-tuning with reward model (image video domain) improves quality and prompt consistency, demonstrating the effectiveness of reward-based fine-tuning. (2) Fine-tuning with VIDEORM (DRaFT-V and Instruct V) signifi- cantly outperforms one-tuning with image domain reward model HPS v2, indicating VIDEORM's superiority for text-to-vidd model tie-ntning.
Efficiency Evaluation. We compare the efficiency of our algorithm two aspects: (1) Convergence speed: we visualize the changes reward values with training steps for DRaFT-H and DRaFT-V in Figure We find that under the same reward-based fine-tuning algorithm, using VIDFORM M leads to earlier convergence and better performance (as demonstrated by the results above) compared to HPS v2, further validating the effectiveness of VIDEORM in fine-tuning text-to-video models. (2) Inference speed. For a single video with frames, HPS v2 requires approximately an average of seconds for inference, while VIDEORM only needs average of 1.3 seconds. Besides, Instruct requires 20 DDIM steps for a single fine-tuning step, taking approximately an average of 8.8 seconds, while DRaFT-V V only needs an average of 2.3 seconds.
7

---

4.3 Ablation Study
Scalability. To investigate the effect of training dataset sizes on the performance of VIDEORM, and to verify the scalability of VIDEOPREFER, comparative experiments are conducted. Note that VIDEORM in this experiment are only trained on VIDEOPREFER. Figure (b) shows that adding up the scale of the dataset significantly improves the preference accuracy of VIDEORM. It's promising that if we employ GPT-4 V to collect more annotation data in the future, VIDEORM will get better performance.
Number of Video Frames N Input to VIDEORM. We investigate the impact of different numbers of input video frames on the performance of the RM model. Specifically, we set the number of
input frames to N 1, 4, 8, and 12 and test corresponding model at TVGE [56] test benchmark. The corresponding results are presented at Figure O (c), we find that the VIDEORM's performance significantly decreases when is small. This is promising because too few input frames prevent the model from accurately capturing all the information and temporal features of the video, e.g., the degree of motion and temporal consistency.
Temporal Feature Modeling method in VIDEORM. We conduct experiments to explore the impact of different video frame temporal feature modeling methods on the performance of the reward model. In VIDEORM, M, we use a temporal transformer trained from scratch to model the temporal features of video frames. Additionally, we conducted ablation experiments by replacing this transformer with ID Convolutional Layer (denoted as VIDEORM') ) and LSTM (de- noted as VIDEORM*). Note that we adjusted the number of layers in the temporal feature modeling module to ensure that the number of trainable parameters remains equal. Besides, Temporal Shift are employed for all compared methods. The results presented Table 4 We find that: (1) All models outperform the image-domain reward model. (2) Replacing the temporal transformer with LSTM achieved comparable performance, indicating that VIDFORM is robust the choice of model architecture for temporal feature modeling. (3) Replacing the temporal transformer with Conv ID resulted in significant performance drop. We hypothesize that this is because Conv 1D cannot effectively model temporal features, leading to VIDEORM'SS inability to accurately evaluate video generation quality from a temporal perspective. This highlights the effectiveness of using temporal transformer for modeling temporal features in VIDEORM.
00
Traiiig Stope
Figure 7:
Ablation over K
adopted in DRaFT-V during fine- tuning text-to-vides model.
Backbone of VIDEORM. We explore the impact of different backbone the final performance of VIDEORM. In our design, VIDEORM is initialized with the weights of HPS v2 [45], which is demonstrated the best reward model for image domain. Here replace HPS v2 with other reward models in image domain, e.g., PickScore [7] (denoted as VIDFORM*) and ImageReward [48] (denoted as VIDFORM while keep the count of trainable parameter equal. The corresponding results are presented at Table 4 We find that the final performance of the corresponding VIDFORM proportional the performance of the initialized backbone weights. For example, ImageReward, which has the lowest performance among the three (see Table 3), corresponds to the worst VIDE- ORM?, while PickScore, which has the best performance among the three (see Table 3 corresponds to the best VIDEORM
Step K adopted in DRaFT-V. We investigate the impact of different K adopted in DRaFT-V on the final generative performance. The results are presented at Figure 7 we find that: (1) Performance
degrades as K increases for K > 10. (2) Even with smaller values of K, e.g., K = 1, DRaFT-V can achieve good results. Both findings are aligned with the results in DRaFT
5 Analysis
In this section, we analyze the impact of different parameter of frames (N) sampled from a single video and the temperature setting 7 of the GPT-4 V, on the accuracy of GPT-4 V-generated annotations. Due to GPT-4 V's limitation of processing a maximum of 10 frames, we analyze annotation accuracy for N = 4, 6, 8, and 10 frames. The temperature controls the diversity and randomness of GPT-4 V's outputs. We analyze annotation accuracy for T 0.3, 0.5, 0.7, and 0.9.
cirriion--speiffcalllyy the number
8

---

0-05
Table 4: Ablation study for VIDEORM. The Aesthetic
Armotation Accurasy
= 0.3 (68.8%)
(69.6%)
Classifier (simplified as Aesthetic) makes prediction with- out seeing the text prompt.
T 0.7 (67.3%)
68-
167.2%
Model
TVGE [56]
VBench [5
T2VQA-DB 18
Avg
(66.3%)
1 6
(67.2%)
(67.3%)
16.. 755
10
Number of Frames (M)
VIDEORM
70.1
58.4
61.5
63.3
Temperature (t)
VIDEORM
73,9
62.7
65,2
67.3
Figure 8: The impact of different hyper-
V
72.2
64.2
66.9
67.7
VIDEORM
72.4
63.6
64,2
66.7
parameter choices on the annotation accu-
racy of GPT-4 V.
VIDEORM
73.7
63.5
65.4
67.5
The summarized results are shown Figure g For N, accuracy initially increases with more frames but decreases beyond N = 8, achieving the highest accuracy at N = 8. This is because more frames allow for comprehensive evaluation, but too many frames can lead to overly long contexts, reducing accuracy due to GPT-4 V's limited capacity. For lower values yield higher accuracy, likely because reduced randomness leads to more comprehensive predictions, while higher randomness might cause GPT-4 V to overly focus specific frames or aspects, resulting in less comprehensive outcomes.
6 Related Work
Generative models, particularly those based on diffusion techniques, have demonstrated high-quality generation capabilities 32 28 31 30 35] by training on extensive internet-scall datasets, but the mixed quality of these datasets often leads to visually unappealing and misaligned outputs.
Aligning text-to-image generative models. Aligning text-to-image generative models [37 17 26
37 has garnered increasing attention in recent years and has shown promising results in producing
outputs that more aligned with human preference. Imagereward 48 HPS v2
47 45), and
PickScore [17 are the three most commonly used reward models for align text-to-image models. They are respectively trained on three major image domain preference datasets, namely Imagereward. ImageRewardDB 48) HPD 47 45) and Pick-a-Pic 17.. The effectiveness of these reward models has been validated across alignment algorithmss like AlignProp 26 DRAFT and ReFL 48
Aligning text-to-video generative models. Compared to text-to-image, exploration related to aligning text-to-video generative models is relatively sparse. InstructVideo [53] instruct text-to-video diffusion models with by fine-tuning with existing image-wise preference reward model HPS v2. We hypothesize that this may restrict effective preference evaluations for generated videos, image- based reward models cannot adequately capture temporal features, impairing assessments of video coherence and dynamics. Additionally, preference datasets for text-to-vides generation are also scarce. The lack of large-scale and effective open-source preference datasets severely restricts the development of research related align texi-to-video generative models.
Reinforcement Learning from AI Feedback. LLMs have been extensively used for data genera- tion 44 [24], augmentation [12] and in self trainini setups [4] 23). Some works [I] introduced the idea of reinforcement learning from AI feedback (RLAIF), which used LLMs labeled preferences in conjunction with human labeled preferences to jointly optimize for the two objectives of helpfulness and harmlessness. Recent works have also explored related techniques for generating rewards from I.MMs
These works demonstrate that I.LMs can generate useful signals for reinforcement learning fine-tuning, However, leveraging the feedback from MLLMs for aligning text-to-image generative model or for text-to-video generative model is less explored.
22 19 50]
7 Conclusion
In this paper, we identify the current obstacles faced in aligning text-to-video generative models, i.c.. lacking of large-scale preference datasets and reward models specifically tailored for videos. Thus, we introduce VIDEOPREFER and VIDEORM to address the aforementioned issues. Experimental validation confirms that GPT-4 V can act as a human-aligned preference annotator. We utilized it to label 135K video preference annotation, forming VIDEOPREFER. Based on this, we trained video- specific reward model, VIDEORM Extensive analytical usage has demonstrated the effectiveness of both VIDEOPREFER and VIDEORM.

---

References
[1] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, C. Chen, C. Olsson, Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson. E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Lar- son, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, Conerly, T. Henighan. T. Hume, S. R. Bowman, Z. Hatfield-Doddss B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
[2] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[3] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and Ramesh. Video generation models as world simulators. 2024.
[4] Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, Jing, D. Schnurr, J. Taylor, Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh. Video generation models as world simulators. 2024.
[5] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961-970, 2015.
[6] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan. Videocrafter2: Overcoming data limitations for high nigh-quality video diffusion models, 2024.
[7] J. Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang, Luo, H. Lu, and Z. Li. Pixart- ssigm:: training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.0466 046922 2024.
[8] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet. Directly fine-tuning diffusion models differentiable rewards. arXiv preprint arXiv:2309.174000 2023.
[9] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Uttrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2110.01377 2023.
[10] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, Fung, and S. Hoi. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[11] Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information P Processing Systems, 36, 2024.
[12] S. Y. Feng, V. Gangal, Wei, S. Chandar, S. Vosoughi, T. Mitamura, and Hovy, A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 968-988, Online, Aug. 2021, Association for Computational Linguistics.
[13] Ho, T. Salimans, Gritsenko, W. Chan, M. Norouzi. and D. Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 1188633-6446 2022.
[14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprini 1eXXi:2106.09666 2021.
[15] Z. Huang, Y. He, Yu, Zhang, C. Si, Y. Jiang, Y. Zhang, T. Wu, Q Jin, N. Chanpaisit, Y. Wang, X. Chen, L. Wang, D. Lin, Y. Qiao, and Z. Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
10

---

[16] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and
H. Shi. Text2video-zere Text-to-image diffusion models are zero-shot video generators, 2023.
[17] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01566 2023.
[18] Kou, X. Liu, Zhang, C. Li, H. Wu, Min, G. Zhai, and N. Liu, Subjective-aligned dateset and metric for text-to-video quality assessment. arXiv preprint S.tiv:24033//99555 2024.
[19] M. Kwon, M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. The Eleventh International Conference on Learning Representations, 2022.
[20] Lin, Gan, and S. Han, Tsm: Temporal shift module for efficient video understanding. 2019.
[21] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of clip for end to end video clip retrieval, 2021.
[22] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan. Videofusion: Decomposed diffusion models for high-quality video generation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10209-10218, 2023.
[23] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang. et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint 2rXiv:2303.17651, 2023.
[24] Y. Meng. M. Michalski, J. Huang. Y. Zhang, T. Abdelzaher, and J. Han. Tuning language models training data generators for nugmentatiion-nnaacee few-shot learning. In International Conference on Machine Learning, pages 24457-24477. PMLR, 2023.
[25] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-277444 2022.
[26] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
[27] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, G. Krueger, and Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In /CML, 2021.
[28] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):3, 2022.
[29] Roit, J. Ferret, L. Shani, Aharoni, G. Cideron, Dadashi, M. Geist, S. Girgin, L. Hussenot, O. Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306. 0000000866 2023.
[30] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (0684-10695 2022.
[31] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion mod- els with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
Denton, K. Ghasemipour, Gon-
[32] C. Saharia, W. Chan, S. Saxena, L. Li, Whang, L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.

---

[33] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman. M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 10.08402, 2022.
[34] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:0010.02502, October 2020.
[35] Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[36] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 33:3008-3022 2020.
[37] Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303. 03751, 2023.
[38] Wang, H. Yuan, Chen, Y. Zhang. X. Wang, and S. Zhang. Modelscope text-to-video technical report. arXiv preprint mrXiv:2308.06571, 2023.
[39] M. Wang, J. Xing, and Y. Liu. Actionclip: A new paradigm for video action recognition, 2021.
[40] W. Wang and Y. Yang. Vidprom: A million-scale real prompt- gaallry dataset for text-to-vides diffusion models. arXiv preprint arXiv:2403.066090 2024.
[41] X. Wang, Wei, D. Schuurmans, Q. Le, E. H. Chi, Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
[42] Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang. Y. He, J. Yu, P. Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv::3309. 15103, 2023.
[43] Z. Wang. A. W. Yu, O. Firat, and Y. Cao. Towards zero-label language learning. arXiv preprint arXiv:2109009190 2021.
[44] S. Wen, G. Fang, R. Zhang, P. Gao, H. Dong, and D. Metaxas. Improving compositional text-to-image generation with large dision-language models. arXiv preprint arXiv:2310.06311, 2023.
[45] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.093411 2023.
[46] X. Wu, S. Huang, and F. Wei. Multimodal large language model is human-aligned annotator for text-to-image generation. arXiv preprint arXiv:2404. 15/00, 2024.
[47] X. Wu, K. Sun, F. Zhu, R. Zhao, and Li. Human preference score: Better aligning text- to-image models with human preference. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2096-2105, 2023.
[48] Xu, X. Liu, Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprini 2rX5v:2304.05977, 2023.
[49] Xu, T. Mei, Yao, and Y. Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5288-5296, 2016
[50] K. Yang, D. Klein, A. Celikyilmaz, N. Peng. and Y. Tian, Rled: Reinforcement learning from contrast distillation for language model alignment, 2023.
[51] K. Yang, J. Tao, J. Lyu, C. J. Chen, Q Li, W. Shen, X. Zhu, and X. Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arXiv:2311.13222 132311, 2023.
12

---

[52] Q. Ye, H. Xu, J. Ye, M. Yan, H. Liu, Q. Qian, J. Zhang, F. Huang, and J. Zhou. mplug-ow12. Revolutionizing multi-modal large language model with modality collaboration, arXiv preprint arXiv:2311.0422 2023.
[53] H. Yuan, S. Zhang, X. Wang, Y. Wei, T. Feng, Y. Pan, Y. Zhang, Z. Liu, S. Albanie, and D. Ni. Instructvideo: Instructing video diffusion models with human feedback. arXiv preprint arXiv:2312.12490 2023.
[54] H. Zhang, X. Li, and L. Bing. Video-llama: An instruction-tune tuned audio-visual language model for video understanding. arXiv preprint 020888888. 2023.
[55] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836 1166-387, 2023.
[56] J. Zhangjie Wu, G. Fang, H. Wu, X. Wang, Y. Ge, X. Cun, D. Junhao Zhang. J.-W. Liu, Y. Gu, R. Zhao, et al. Towards better metric for text-to-video generation. arXiv e-prints, pages arXiv-2401, 2024.
[57] Z. Zheng. X. Peng, and Y. You. Open-sora: Democratizing efficient video production for all, March 2024.
13

---

A Datasets Processing
T2VQA-DB [18]. The T2VQA-DB dataset only comprises a single overall preference evaluation score for each data item. We randomly selected 2.3K samples to serve as the final test benchmark.
TVGE 56). The TVGE dataset comprises preference evaluations from 2 aspects, namely: text alignment and video quality. We just simply average the scores across these two aspects to obtain final overall evaluation score with the aforementioned datasets. We randomly selected 15K samples to serve as the final test benchmark.
VBench 15
The VBench dataset comprises preference evaluations from 16 aspects, namely; subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality, imaging quality, object class, multiple objects, human action, color, spatial relationship, scene, temporal style, appearance style, overall consistency.
Due to the fact that the videos and prompts evaluated for preference in each aspect are largely non-overlapping, we cannot simply average the scores across all aspects obtain a final overall evaluation score as with the aforementioned datasets. Instead, we mixed all samples from the 16 aspects and randomly selected 10K samples to serve as the final test benchmark.
B VIDEOPREFER
B.1 Statistic of VIDEOPREFER
In Table 3 we present the sources and distribution of the videos in our VIDEOPREFER. We find that VIDEOPREFER comprises diverse range of video sources, including videos generated by state-of-the-arr text-to-video models as well as real videos. This extensive variety video sources enhances the robustness and generalization capabilities of VIDEOPREFER.
We visualize the distribution of preference annotations across two evaluation aspects (prompt- following an video-quality in VIDEOPREFER in Figure We find that the overall score distribution of the dataset is close to the normal distribution for the Prompt-Following evaluation perspective. From the perspective of Video-quality evaluation, the overall distribution of data sets tends to be low, which indicates that the generation effect the existing text-to-video model not satisfactory.
Besides, we also visualize some data examples in VIDEOPREFER at Figure I0
Table 5: Video sources of VIDEOPREFER. denotes the realistic videos from existing benchmarks.
Source
Proportion
Video Length
Type Diffusion Diffusion
Resolution
LaVie
42
16.4%
2.0s
512x512
ModelSenpe
21.0%
2.0s
256x255
VidcoCrafter2
17.2%
1.6s
Diffusion
320x320
Open-Sore
37
2.4%
DiT
512x512
Pike
7.2%
3.0s
DiT
x088x640
Text Video-Zeno
16
15.3%
2.0s
Diffusion 512x512
ZeroScope
13.8%
Diffusion
1024x576
Gen-2
0.7%
1792x 1024
ActivityNet 5
1.7%
Human-captured Human-captured
MSR-VTT
49
0.03%
Score
Score
Score 3
Secore 2
Score
21 21.7%
1633%
18.7%
3.415
Prompt- Following
Video Saleo-Quality
Figure 9: Score distribution across two annotation aspects in PREFR..
14

---

working in his room, typing portable Prompt: computer, and cat watching him
Trompt-Following
Rating::
Rationale: sampled frmes from the vodeo show working portable with watching him, which algges perfectly with original test previded. setting appaors Irom farther supports descriptionn
Indicsted by
bookshelf backgrund, which provided.
Video-Q
Rating
Rationale: The sumpled frumess showed that there was no chrious attifuets iz this vidco whole, hut from the perspettvve of time sequenon both hunans and cats were almost inmobile, lacking the dynamic continiity of the vilio, so the final score 3.
Prompt-Fallowing
Rating::
Rationall:: The vidoo trames show cat watching and intercting with lapoop, which aligns with the text deacthing cat wotching working poctable However: lhe visible the Eranes, only his hand shown. wluch Jesis minor devialion mentioned the Overall the vodeo aligned the onginall but the riting
from the depiction the working
Vidho-Quallty
Rating:
Rationele: Oz the while, the videu rich in details, beauräffl and strung in color, wlich in with humen aesibelios bems of temporal dynamtes, the hands in the vidco have topptng movements, so can most the syname rquemments of the vodco.
Prompt-Following
actively watciung working gortable computer deviation toct, spectfiis
desoription. minor
lynn ea purtally visille watching man. Honce, the rating 1"A There" this slight
discmpprryy
Vidro-Qualtty
Rating:
Rationall:: in the sumpled frumes of this video, there are obrious object deformations such as laptng computers. In addition, temms of timing fostures the beck hes the wation of lapping the keyboandd which confoms to the timing festures but object disappoor, which not final
the continutt
Follniin
Rating::
Rationale: fromos vidao person typing on portible computer watching allymmmnt
provided toct. There zre no decations from the descrthed aoerano, thas the vidco fully aligns with the original test iqqutrements:
Vidro-Quallie
Rating:
Rationale: ovenill continet of light dark ligit the vidio is strong, and the objoct details are rich, which in with aeshetiss but the sequeace dyzamics is aot good, so the fins. sooge 3
Figure 10: Visualization of example data item in VIDEOPREFER. Here show one data item which contains prompt and four corresponding generated videos. For each video, there are two annotations from different annotation aspects (Promit-Following and Video-Quality) are provided by GPT-4 v
15

---

C Visualization of Selection Results
cat playing with butterfly
beat
small bird blooming
CLIP
IageRRoaard
HPS v2
VideoRM
Hewalian woman playing ukulelee
Two alephants playing on the beach
woman with striking blue and curly
CLIP
ImwgeR
VideoRM
Figure 11: Top-1 videos from 32 generated videos select by CLIP, ImageReward, PickScore, HPS v2 and VIDEORM. VIDEORM capable selecting higher-quality generated videos, e.g., those that better match the prompt descriptions and exhibit more dynamic content.
16

---

D Visualization Results for Fine-tuning Text-to-Video Models
person
feeding geats
pod doiphins gracefully jump ocean
Baseline
InstructVideo
DRaFT-H
InstrutWidee-V
DRaFT-V
McDonald
bear and people making pizza
Baseline
InstructVideo
DRAFT-
Mcacconit
Mcscoaal
Instruct/ideo-V
DRaFTVV
Figure 12: Visualization Results for different fine-tuning methods. We find that compared to fine- tuning with an image domain reward model, fine-tuning with VIDEORM significantly enhances the performance of text-to- ext-to-video models (DRaFT-V and InstructWideo-V).
17

---

E
Instruction Template
Preference Instruction for Frompt-Followiig
Prompt-Following
You are an Al assistant programmed to assess videos with impartial and balanced standards. video has been created based on a piece of text. Your task is evaluate how well The content of the video aligns with the original text provided as ("Input") The video evaluation based on sampled frames shown in sequence. Scoring: Rating outputs to 5:
1. Irrelevant: No alignment.
2. Partial Focus: Addresses one aspect poorly.
3. Partial Compliance:
-(I) Meets goal or restrictions, neglecting other.
(2) Acknowledges both but slight deviations.
4. Almost There: Near alignment, minor deviations.
5. Comprehensive Compliance: Fully aligns, meets all requirements.
Please present your assessment follows:
Output
Rating: [Provide the rating]
Rationale: [Explain the reason for your rating in concise sentences]
Now, review the following video and its corresponding text.
Input:
#### Text: [INSERT PROMPT HERE]
#### Frames sampled from video: [INSERT THE FRAMES OF VIDEO HERE]
18

---

Preference Instruction for Video-Quality:
Video-Quality:
You are an Al assistant trained to impartially assess temporal consistency quality, dynamic quality and aesthetic quality in videos. A video has been created based piece of text. Your task is to analyze the quality of this video based on the following guidelines and provide comprehensive evaluation. Scoring:
Rating outputs to
Bad: blurry, underexposed with significant noise, indiscernible subjects, exhibits significant inconsistencies and noticeable discrepancies in appearance of subjects.
2. Poor:
Noticeable blur, poor lighting, washed-out colors, and awkward composition with cut-off subjects, suffers from noticeable issues in maintaining uniformity of subjects and backgrounds.
3. Fair: In focus with adequate lighting, dull colors, decent composition but lacks creativity. Subjects and backgrounds maintain reasonable degree of uniformity throughout most of the video, with only minor discrepancies.
4. Good:
Sharp, good exposure, vibrant colors, thoughtful composition with clear focal point. Good video dynamics and temporal consistency.
5. Excellent:
Exceptional clarity, perfect exposure, rich colors, masterful composition with emotional impact. perfect temporal consistency and excellent dynamics.
Please present your assessment follows:
Output
Rating: [Provide the rating]
Rationale: [Explain the reason for your rating in concise sentences]
Now, review the following video and its corresponding text.
Input:
#### Text: [INSERT PROMPT HERE]
#### Frames sampled from video: [INSERT THE FRAMES VIDEO HERE]
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper includes our motivation, preference data construction details, ex- perimental settings, quantitative experimental results, and qualitative visual examples that reflect and justify the claims our abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The analysis section contains discussion of limitations our work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[Yes]
Justification:
We state the reference before illustrating formulations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed description of the dataset construction process, imple- mentation for training the reward model, and offer a multitude of visualization examples.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will public our data and code upon paper acceptance, due to the manage- ment regulations of our institution.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https//cccpbpiigg..deeeee//////e CooeeSummide ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full training and testing details in $ and Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would take too long with our available computational resources to repeat all experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the name of GPU we used for experiments in Sctup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All relevant human questions and relevant datasets have been checked for privacy compliance prior to experiments and submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work supports the alignment of generative models with human values, and its societal impact is discussed in the introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our contribution does not include new datasets pre-trained models that pose a risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Code that we derive from earlier work is properly licensed and referenced.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, papersai
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed illustration of new data benchmark and visualized exam-
ples.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
We only recruited participants for user experiments to validate the effective- ness of our model, where they were asked to choose from generated images. No human participants were involved in the dataset construction model training process.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiment solely involves measurement and does not entail behavioral manipulation; therefore, we did not apply for IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "uLdnGPDOUx", "reviewer": "Reviewer_nZ7n", "summary": "This paper introduces a new dataset called VideoPrefer, a collection of (simulated) human preferences on videos conditioned on certain language prompts. VideoPrefer utilizes GPT-4v as an automatic reward assessor, which contains videos from both machine-generated and real-world curated videos. The work then utilizes the collected preference as a reward to learn a reward model named VideoRM, constructed on top of a prior model HPS v2, extending its capabilities to temporal dimension and thus can present reward modeling in videos.", "questions": "- There are some minor typos, for example, L84 the “K” is already a thousand there. Please be more mindful when writing.\n- What or who is deciding the “win” in Figure 2? The GPT-4v assessor?", "limitations": "- The limitations of this work do not seem to be explicitly addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work collects a large scale of preference data, potentially useful for evaluating text-to-video generative models.\n- It is demonstrated that the designed reward model VideoRM is showing the most aligned preferences over a few baseline methods.\n- Finetuned text-to-video models utilizing the presented VideoRM seems effective under the designed experimental settings.", "weaknesses": "- While I appreciate the Table 2’s analysis on TVGE dataset among the tested video assessor candidates, solely performing statistical analysis on the VideoPrefer dataset is still required. People would be relying on the VideoRM which is trained on the VideoPrefer data and it remains questionable how faithful the dataset is.\n- Lack of some analysis between real-world and generated videos. Under the same prompt, are real-world videos always or more likely to be better? What heuristics can people use to expand beyond the collected video sets?\n- Is it unclear how text prompts are aligned with real-world videos in Section 2.2.\n- Lack of rigorous statistical analysis on the curated video prompts. How diverse are they? What is the type-token ratio? What are the top frequent predicates and affected entities? What are the genres? It is hard to understand how challenging and/or how underlying bias would have affected the video generation process from this curated resource.\n- Why only consider extension of HPS v2 as the base reward model? There are plenty of good video-language models that can be potentially useful (such as [1]) to be adapted as a reward model. If the work can show more unbiased learning from the dataset solely with video-based preference models, it would strengthen the work more.\n\n[1] Sung, Yi-Lin, Jaemin Cho, and Mohit Bansal. \"Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.\" CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Due to space constraints, we address your questions in the **Rebuttal** below as well as at the top of this page in the **Author Rebuttal**:\n\n---\n> **Q1**: Solely performing statistical analysis on the VideoPrefer dataset is still required. It remains questionable how faithful the dataset is.\n\n**A1**: Thank you for your insightful suggestions. To further validate the reliability of VideoPrefer, we randomly selected 2,000 samples from VideoPrefer and invited six human experts to conduct human preference scoring on two dimensions: prompt-following and video quality. We analyzed the correlation between the human experts' scores and GPT-4V's scores, which are presented in Table A. We found that GPT-4V exhibited excellent alignment with human judgment in both aspects (close to or exceeding 70%), further demonstrating the reliability of VideoPrefer.\n\nAdditionally, we provide a comprehensive statistical analysis and presentation of VideoPrefer in Section B of the appendix, including the distribution of preference scores, the distribution of video sources, and more. In the latest version of the paper, we will also include more detailed statistical analyses of VideoPrefer, such as the score distribution between real and generated videos, statistical analysis of the prompts collection, and bias analysis. These will serve as references for those using VideoPrefer.\n\n**Table A.** Correlations between GPT-4 V and human preference judgment.\n\n| Prompt-Following | Video-Quality |\n|-|-|\n| 69.65%|73.97%|\n\n---\n>**Q2**: Under the same prompt, are real-world videos always or more likely to be better? What heuristics can people use to expand beyond the collected video sets?\n\n**A2**: We conducted the following analysis: We randomly selected 500 samples containing real videos from VideoPrefer and performed a statistical analysis of GPT-4V's preference scores on prompt-following and video quality. We found that real videos had a high likelihood of receiving top scores, at 84.91% and 78.23%, respectively (compared to 25% in a random scenario). This indicates that GPT-4V tends to assign higher preference scores to real videos. To explore whether this phenomenon is reasonable, we asked six human experts to also score these samples. We found that the likelihood of real videos receiving high scores was 82.33% and 88.29% in prompt-following and video quality, respectively (25% in a random scenario). This indicates that real videos generally receive higher preference scores, also highlights that the current quality of video generation models still needs improvement.\n\nAdditionally, this inspired us to consider generating more comparison videos using video generation models for each sample containing real videos and using real videos as cases that typically receive higher human preference scores to rapidly expand the preference dataset. We will further explore the effectiveness and potential limitations of this method in future research.\n\n---\n>**Q3**: Is it unclear how text prompts are aligned with real-world videos in Section 2.2.\n\n**A3**: In fact, we use the current video caption dataset directly as the source for obtaining text prompts and corresponding real video pairs. Specifically, for each prompt in the video caption dataset, we generate three additional video candidates using a randomly sampled generative model. These generated video candidates, along with the original prompt and the corresponding real video clip, together constitute a single sample of VideoPrefer.\n\n---\n> **Q4**: Why only consider extension of HPS v2 as the base reward model? There are plenty of good video-language models that can be potentially useful to be adapted as a reward model.\n\n**A4**: The reasons for selecting the extension of HPS v2 as the base reward model are as follows:\n\n1. **High Performance and Low Bias**: HPS v2 is the best-performing reward model trained on the largest debiased preference dataset in the text-to-image domain, with minimal bias. Since video preference scores are highly correlated with the preference scores of each frame in the video, HPS v2 naturally serves as an effective reward model for the video generation domain[1]. By using HPS v2 as the initial model, we provide our reward model with a strong initial point and foundational knowledge, thus enhancing the effectiveness of the reward model.\n2. **Efficient Deployment**: During the alignment process of video generation models, the deployment cost of the reward model (in terms of parameters, computational load, and inference time) can significantly impact the alignment performance under resource constraints. HPS v2 has a much lower deployment cost compared to some video-language models, making it easier to deploy during the model alignment process.\n\nWe also finetuned the video-language models you recommended[2] (denoted as VL-Model) on both open-source human-crafted preference datasets and VideoPrefer, using the same training steps & data as VideoRM. The preference prediction accuracy is shown in Table B. We found that its performance was not as good as VideoRM, which further demonstrates the effectiveness of using HPS v2 from the text-to-image domain as the foundation for our reward model.\n\n**Table B.** \n\n|Model|TVGE|VBench|T2VQA-DB|\n|-|-|-|-|\n|HPSv2|69.5|55.7|52.8|\n|VideoRM|**73.7**|**63.5**|**65.4**|\n|VL-Model|64.44|53.38|53.09|\n\n**Reference**\n\n[1] Yuan, Hangjie, et al. \"InstructVideo: instructing video diffusion models with human feedback.\" CVPR 2024.\n\n[2] Sung, Yi-Lin, et al. \"Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.\" CVPR 2022.\n\n> **Q5**: There are some minor typos.\n\n**A5**: We apologize for the typos and will correct them in the latest version of the paper.\n\n---\n> **Q6**: What or who is deciding the “win” in Figure 2?\n\n**A6**: Pleasee see in the **Author Rebuttal**.\n\n---\n>**Q7**: The limitations do not seem to be explicitly addressed.\n\n**A7**: Pleasee see in the **Author Rebuttal**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VAFLCgRDs1", "reviewer": "Reviewer_snrb", "summary": "The paper presents a dataset and a learned reward model to better align the generated outputs of text-to-video models with human preferences. Specifically, the authors make use of GPT-4V to provide preference scores for a large dataset of videos by learning from human feedback on a smaller set of videos. The authors then develop their proposed architecture for the reward model on top of this dataset to predict preference scores given prompts and videos. They fine-tune text-to-video models with their proposed reward model and show the improvements in alignment with human feedback through quantitative evaluations, ablation experiments, qualitative results, and user studies.", "questions": "1. In their ablation study, why do the authors consider up to 12 frames? Is this limit due to computational constraints?", "limitations": "Yes, but needs more details on how the authors filter for discriminatory, harmful, or abusive content in their collected dataset.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed idea of leveraging an LLM such as GPT-4V to create a large-scale synthetic dataset for the text-to-video paradigm is useful. It makes good use of existing technology to explore previously intractable problems.\n\n2. The reward model design is technically sound and rigorous.\n\n3. The experiments presented in the paper are extensive, covering multiple LLMs for dataset creation, multiple feature representations for text-to-video models, and multiple datasets for showing model performances.", "weaknesses": "1. Some key aspects of the dataset collection process are unclear.\n\n    1.1. While GPT-4V has the best agreement with human feedback, it is still not very high (around 70%, according to the authors). Did the authors explore any prompt-tuning or other approaches to improve the agreement? Did the authors observe any potential patterns in the disagreement? Perhaps certain categories of videos have more agreement between GPT-4V and human feedback than other categories?\n\n    1.2. When collecting data from video caption datasets, do the authors process the captions in some way to make them structurally and semantically similar to video generation prompts?\n\n    1.3. Did the authors run any data filtering for discriminatory, harmful, or abusive content in the collected/generated prompts and videos?\n\n    1.4. Did the authors consider various levels of complexity in the prompts, such as foreground and background details, number of objects, actions, interactions, etc.?\n\n    1.5. How did the authors quality-check the videos generated for the proposed dataset? It is not fully clear how the proposed approach avoids a cyclic process, where improving video generation depends on a better reward model, which, in turn, requires generated videos of a certain quality.\n\n    1.6. Can the authors please clarify the specific numbers regarding the dataset size? How did they go from 14K data items to 135K preference choices?\n\n2. Some key aspects of the user study are unclear, making it hard to fully follow the results.\n\n    2.1. Is five participants a sufficiently high number from which to draw conclusions? How many videos did each participant respond to? Was there a mix of non-experts and subject matter experts such as content creators or video editors? If so, were their responses segregated in any way?\n\n    2.2. Did the authors check for participants' engagement, e.g., whether participants responded too quickly or too slowly, and whether any such responses needed to be discounted? Did the authors check for any recency bias or response drifts, where the participants may have responded differently to similar video content over time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Due to space constraints, we address your questions in the **Rebuttal** below as well as at the top of this page in the **Author Rebuttal**:\n\n---\n>**Q1**: While GPT-4V has the best agreement with human feedback, it is still not very high. Did the authors explore any prompt-tuning or other approaches to improve the agreement? Did the authors observe any potential patterns in the disagreement?\n\n**A1**: **A 70% agreement rate among qualified human preference annotators is a widely recognized benchmark across multiple fields, including preference annotations in NLP[1] and text-to-image[2].** This means that when different human experts provide preference annotations for the same sample, their agreement rate is typically around 70%. With GPT-4V achieving an agreement rate of 69.65%, we consider it a reliable preference annotator in the text-to-video domain.\n\nWe explored the impact of different input templates and hyperparameters (as discussed in Section 5) on the accuracy of GPT-4V annotations and identified the most suitable input template and hyperparameters. We believe that exploring prompt-tuning or other approaches to improve agreement is a very interesting research topic, and we plan to explore this further in future research.\n\nAdditionally, our analysis shows that in VideoPrefer samples containing real videos, the agreement between GPT-4V and human feedback is higher, reaching approximately 93% (compared to about 70% in samples without real videos). Both GPT-4V and human feedback tend to assign higher preference scores to real videos in these samples, indicating that the quality of videos generated by existing models still needs improvement compared to real videos.\n\n**Reference**\n\n[1] Cui, Ganqu, et al. \"Ultrafeedback: Boosting language models with high-quality feedback.\" ICLR 2024.\n\n[2] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" NeurIPS 2024.\n\n---\n>**Q2**: Do the authors process the captions to make them similar to video generation prompts?\n\n**A2**: We did not apply any special processing to the captions; instead, we used them directly as input prompts for the video generation model to generate videos. We believe this approach ensures that samples in VideoPrefer, composed of real and generated videos, fairly correspond to the same prompt. If we were to preprocess the prompts, we are concerned that differences between prompts corresponding to generated and real videos might introduce potential bias into the final preference results.\n\n---\n>**Q3**: Did the authors run any data filtering for prompts and videos? Needs more details on how the authors filter for discriminatory, harmful, or abusive content in their collected dataset.\n\n**A3**: The prompts in VideoPrefer are primarily sourced from the VidProM dataset, as well as the MSR-VTT and ActivityNet video caption datasets. The VidProM dataset has already implemented harmful content filtering to screen prompts (consisting of toxicity, obscenity, identity attack, etc.). Since MSR-VTT and ActivityNet are widely used video caption datasets released some time ago, we believe they contain minimal harmful information. Therefore, we did not perform additional harmful content filtering on the prompts. Similarly, the videos in VideoPrefer are mainly sourced from some open-source video generation models and video caption datasets, and we have not applied data filtering to them. Your suggestion is valuable, and we will incorporate harmful content filtering for both prompts and videos in the v2 version of VideoPrefer to ensure the dataset is more legally compliant and appropriate.\n\n---\n>**Q4**: Did the authors consider various levels of complexity in the prompts?\n\n**A4**: In this study, VideoPrefer directly uses the existing prompts from the VidProM dataset, as well as the MSR-VTT and ActivityNet video caption datasets, without considering complexity levels. Your suggestion is valuable :), and we plan to distinguish and categorize the complexity levels of prompts in future research. We will make the necessary adjustments to enhance the effectiveness of VideoPrefer.\n\n---\n>**Q5**: How did the authors quality-check the videos generated for the proposed dataset? It is not fully clear how the proposed approach avoids a cyclic process, where improving video generation depends on a better reward model, which, in turn, requires generated videos of a certain quality.\n\n**A5**: **The quality check for the generated videos primarily relies on GPT-4V, i.e., MLLMs**. VideoPrefer is essentially a preference dataset in the text-to-video domain, where each sample includes a prompt and four video candidates corresponding to that prompt. GPT-4V scores these four videos on two dimensions: prompt-following and video quality (see Section 2.2). The comparison of preference scores between high-scoring and low-scoring videos helps us train an effective reward model (see Section 3.2) to assess the degree of human preference for videos.\n\n>**Q6**: Can the authors clarify the the dataset size? How did they go from 14K data items to 135K preference choices?\n\n**A6**: Our VideoPrefer dataset contains 14k prompts, each with four video candidates. These four video candidates are scored by GPT-4V on two dimensions: prompt-following and video quality. This allows us to perform pairwise comparisons across these two dimensions, allowing for 12 pairwise preference comparisons per prompt. Therefore we obtain 14k × 12 = 168k preference choices. After filtering out comparisons where the preference scores were equal (as these cannot be used to optimize the reward model), we finalized 135k preference choices.\n\n---\n>**Q7**: Are five participants sufficient? How many videos did each participant respond to? ...?\n\n>**Q8**: Did the authors check for participants' engagement? ...?\n\n>**Q9**: In their ablation study, why do the authors consider up to 12 frames? \n\n**A**: Pleasee see in the **Author Rebuttal**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKhO69tbl7", "reviewer": "Reviewer_qjMG", "summary": "Recent advancements in text-to-video generative models, such as Sora, have shown impressive capabilities, generating significant interest for their potential applications. However, these models often rely on extensive datasets of variable quality, resulting in generated videos that may lack aesthetic appeal and fail to accurately reflect the input text prompts. To address this, leveraging Reinforcement Learning from Human Feedback (RLHF) aims to align model outputs with human preferences, but the high costs of manual annotation have limited comprehensive preference datasets. This study investigates the efficacy of Multimodal Large Language Models (MLLMs) for generating annotations, finding a high degree of concordance with human judgments. Building on this, the study uses MLLMs to perform fine-grained video preference annotations, creating VIDEOPREFER, a dataset with 135,000 annotations. Utilizing this dataset, the authors introduce VIDEORM, the first general-purpose reward model for video preference in the text-to-video domain. Comprehensive experiments validate the effectiveness of both VIDEOPREFER and VIDEORM, representing a significant advancement in aligning text-to-video models with human preferences.", "questions": "Please refer to the weaknesses.", "limitations": "This paper discusses the limitations of choosing hyperparameters, which significantly influence the annotation accuracy of GPT-4 V. In addition, it would be better to also discuss the limitations of VIDEOPREFER and VIDEORM.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Advancing Text-to-Video Generative Models with Synthetic Feedback**: This work addresses an important direction in the field by leveraging synthetic feedback to enhance text-to-video generative models. The incorporation of Reinforcement Learning from Human Feedback (RLHF) aims to align model outputs more closely with human preferences, which is crucial for improving the quality and relevance of generated videos. This approach helps overcome the limitations of variable-quality datasets and enhances the models' ability to produce aesthetically appealing and contextually accurate videos.\n\n2. **Significant Contribution through Comprehensive Preference Dataset**: The authors have compiled a preference dataset for text-to-video generative models, consisting of 14,000 prompts, 54,000 videos, and 135,000 preference choices. This dataset, named VIDEOPREFER, represents a significant contribution to the field of text-to-video generation. \n\n3. **VIDEORM Reward Model for Fine-Tuning**: Building on the preference dataset, the authors have trained a reward model named VIDEORM. This model is tailored specifically for video preference in the text-to-video domain and can significantly aid subsequent text-to-video generative models in fine-tuning with high-quality data.", "weaknesses": "1. **Representativeness of Generated Video Quality**: Over 98% of the videos in VIDEOPREFER are generated using open-source models (as indicated in Table 5). This high percentage raises a crucial question about the extent to which the quality rankings of these synthetic videos can represent the quality rankings of real videos. It is important to clarify that the concern is not whether GPT-4 V aligns with human judgment, but rather how well the preference dataset and the resulting reward model genuinely capture and represent real-world video quality.\n\n2. **Quality Demonstration through Video Demos for VIDEOPREFER**: The quality of the VIDEOPREFER dataset cannot be fully assessed through the images shown in Figure 10 alone. To better judge the dataset's quality, it is recommended that the authors provide 5-10 videos from the dataset. These demos would offer a clearer and more comprehensive understanding of how well the dataset captures human video preferences and the overall quality of the included videos.\n\n3. **Assessment of VIDEORM through Comparison Demos**: Figure 11 showcases images selected by VIDEORM, but these images alone are insufficient to evaluate the actual quality of the selected videos. To properly assess the effectiveness of the VIDEORM reward model, the authors are recommended to provide 3 comparison sets of the selected videos by different reward models. These demos would allow for a more accurate judgment of the reward model's value and its ability to select high-quality videos that align with human preferences.\n\n4. **Effectiveness of Fine-Tuning with VIDEORM Demonstrated through Demos**: The images in Figure 12 illustrating the results of the VIDEORM fine-tuned model do not adequately convey the actual video quality. To better demonstrate the effectiveness of fine-tuning with the proposed reward model, the authors are recommended to present 3 sets of comparison videos. These demos would offer a concrete demonstration of how fine-tuning with VIDEORM improves video generation quality, thus providing a more tangible proof of the model's effectiveness.\n\nBy addressing these points and providing the suggested video demos, the study could offer a more comprehensive and transparent evaluation of the proposed methods and datasets, significantly enhancing the understanding and validation of the results presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We address your feedback point by point below.\n\n---\n>**Q1**: **Representativeness of Generated Video Quality**: Over 98% of the videos in VIDEOPREFER are generated using open-source models (as indicated in Table 5). This high percentage raises a crucial question about the extent to which the quality rankings of these synthetic videos can represent the quality rankings of real videos. It is important to clarify that the concern is not whether GPT-4 V aligns with human judgment, but rather how well the preference dataset and the resulting reward model genuinely capture and represent real-world video quality.\n\n**A1**: Your suggestion is very interesting. In fact, the primary aim of our text is to demonstrate that MLLMs can provide reliable preference annotations for the text-to-video field and to present VideoPrefer, the most comprehensive video preference dataset, as well as the most effective reward model, VideoRM. Therefore, the focus of our research is on improving the quality of synthetic videos (generated by text-to-video models) and their alignment with human preferences. We introduced real videos into VideoPrefer to enhance the dataset's diversity and generalizability.\n\nMoreover, we conducted the following analysis experiment: We randomly selected 500 samples containing real videos from VideoPrefer and conducted a statistical analysis of GPT-4V's preference scores in terms of prompt-following and video quality. We found that the likelihood of real videos receiving high scores was 84.91% and 78.23%, respectively (compared to a random chance of 25%). This indicates that GPT-4V tends to assign higher preference scores to real videos. Additionally, we had six human experts score these samples for preference. We observed that the likelihood of real videos receiving high scores was 82.33% and 88.29%, respectively (compared to a random chance of 25%). This alignment with GPT-4V as the scorer demonstrates that VideoPrefer and VideoRM can effectively capture the high quality of real videos.\n\n---\n\n>**Q2**: Quality Demonstration through Video Demos for VIDEOPREFER & Assessment of VIDEORM through Comparison Demos & Effectiveness of Fine-Tuning with VIDEORM Demonstrated through Demos.\n\n**A2**: All the authors fully agree with your suggestion to provide more demonstrations comparing VideoPrefer, videos selected by different reward models, and videos generated through fine-tuning with VideoRM. In the latest version of the paper, we will enhance these demos to provide a more robust and effective evaluation and presentation of the quality of our research.\n\n---\n\n> **Q3**: This paper discusses the limitations of choosing hyperparameters, which significantly influence the annotation accuracy of GPT-4 V. In addition, it would be better to also discuss the limitations of VIDEOPREFER and VIDEORM.\n\n**A3**: Thank you for your suggestions. We will include content about the limitations of VideoPrefer and VideoRM in the limitations section. A concise summary is as follows:\n\nAlthough VideoPrefer is currently the largest preference dataset in the text-to-video domain, we believe it can be further scaled to achieve better alignment results. In the future, we plan to scale it further. Additionally, we will explore designing more optimal reward model architectures to support more effective video feature modeling, more robust preference prediction capabilities, and more efficient video processing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ynt9RF2PkM", "reviewer": "Reviewer_cM4S", "summary": "Recent text-to-video models like Sora show potential but suffer from poor video quality and misalignment with text prompts due to variable dataset quality. This study addresses these issues by leveraging Reinforcement Learning from Human Feedback (RLHF) to align outputs with human preferences. Due to the high costs of manual annotation, Multimodal Large Language Models (MLLMs) were used for video preference annotations, demonstrating strong concordance with human judgments. This led to the creation of VideoPrefer, containing 135,000 annotations, and the development of VideoRM, a reward model for video preferences. Experiments confirm the effectiveness of both VideoPrefer and VideoRM.", "questions": "Weakness:\n\n1. Although existing research has demonstrated that GPT-4 can be used for data annotation, it is essential to perform a sample check of the annotated data to assess its quality. Relying solely on the literature to support the use of GPT-4 for annotation without conducting thorough spot checks and corrections undermines the reliability of the annotated content. Ensuring the accuracy and quality of annotations through systematic verification is crucial for maintaining the integrity of the dataset.\n\n2. Furthermore, the evaluation of generated videos should consider perspectives from multiple roles. Incorporating role-playing prompts to capture diverse viewpoints would more accurately reflect the varied opinions and perspectives that different individuals may have regarding the same video. Relying on a single prompt template for evaluation is limiting and does not adequately represent the range of possible reactions and insights.\n\n3. From the content of Algorithm 1's pseudocode, it appears that there is a lack of task-specific algorithmic innovation in the application of reinforcement learning for fine-tuning. The approach seems to follow standard practices without introducing novel techniques tailored to the specific challenges of the task at hand.\n\n4. Although the proposed VIDEORM model demonstrates strong performance across various metrics, the paper lacks interpretability experiments. Including such experiments would provide deeper insights into the model's decision-making process and enhance the overall understanding of its effectiveness.", "limitations": "The preference annotation in this study relies entirely on GPT-4, which is not advisable. While manual annotation is costly, the authors could employ individuals from diverse fields, genders, and age groups to annotate preferences according to a standardized procedure from their unique perspectives. These annotated samples could then be used as prompts for GPT-4 to perform further annotations, potentially improving the quality. Additionally, designing a richer set of prompt templates to capture preferences from multiple perspectives would enhance the dataset's inclusivity and robustness. The current approach lacks the diversity and comprehensive representation necessary for a truly inclusive dataset.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. The research focuses on utilizing multimodal large models for dataset annotation, which is a highly promising direction with significant potential for real-world applications. This innovative approach leverages the strengths of multimodal data to enhance the accuracy and efficiency of dataset labeling, offering substantial advancements in the field.\n\n\n2. The paper introduces a novel reward model in the field of video preference, which effectively evaluates video quality. This approach presents a significant advancement, as it offers a robust method for assessing video content, potentially leading to improved recommendations and enhanced user experience.\n\n3. The quantitative analysis effectively demonstrates the text-to-video selection capability of the proposed VIDEORM model, highlighting its strong semantic alignment abilities. This thorough analysis underscores the model's proficiency in aligning textual inputs with relevant video content, showcasing its potential impact in the field.", "weaknesses": "Weakness:\n\n1. Although existing research has demonstrated that GPT-4 can be used for data annotation, it is essential to perform a sample check of the annotated data to assess its quality. Relying solely on the literature to support the use of GPT-4 for annotation without conducting thorough spot checks and corrections undermines the reliability of the annotated content. Ensuring the accuracy and quality of annotations through systematic verification is crucial for maintaining the integrity of the dataset.\n\n2. Furthermore, the evaluation of generated videos should consider perspectives from multiple roles. Incorporating role-playing prompts to capture diverse viewpoints would more accurately reflect the varied opinions and perspectives that different individuals may have regarding the same video. Relying on a single prompt template for evaluation is limiting and does not adequately represent the range of possible reactions and insights.\n\n3. From the content of Algorithm 1's pseudocode, it appears that there is a lack of task-specific algorithmic innovation in the application of reinforcement learning for fine-tuning. The approach seems to follow standard practices without introducing novel techniques tailored to the specific challenges of the task at hand.\n\n4. While the proposed VIDEORM model demonstrates strong performance across various metrics, the paper lacks interpretability experiments. Including such experiments would provide deeper insights into the model's decision-making process and enhance the overall understanding of its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. We address your feedback point by point below.\n\n---\n\n>**Q1**: Although existing research has demonstrated that GPT-4 can be used for data annotation, it is essential to perform a sample check of the annotated data to assess its quality. \n\n**A1**: Thank you for your insightful suggestions. To further validate the reliability of VideoPrefer, we randomly selected 2,000 samples from VideoPrefer and invited six human experts to conduct human preference scoring on two dimensions: prompt-following and video quality. We analyzed the correlation between the human experts' scores and GPT-4V's scores, which are presented in Table A. We found that GPT-4V exhibited excellent alignment with human judgment in both aspects (close to or exceeding 70%). A 70% agreement rate among qualified human preference annotators is a widely recognized benchmark across multiple fields, including preference annotations in NLP[1] and text-to-image[2], further demonstrating the reliability of VideoPrefer.\n\n**Table A.** Correlations between GPT-4 V and human preference judgment across two aspects.\n\n| Prompt-Following | Video-Quality |\n| ---------------- | ------------- |\n| 69.65%           | 73.97%        |\n\n**Reference**\n\n[1] Cui, Ganqu, et al. \"Ultrafeedback: Boosting language models with high-quality feedback.\" ICLR 2024.\n\n[2] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" NeurIPS 2024.\n\n---\n\n>**Q2**: Furthermore, the evaluation of generated videos should consider perspectives from multiple roles. Incorporating role-playing prompts to capture diverse viewpoints would more accurately reflect the varied opinions and perspectives that different individuals may have regarding the same video. Relying on a single prompt template for evaluation is limiting and does not adequately represent the range of possible reactions and insights.\n\n**A2**: All the authors agree that your suggestion is very meaningful. We will incorporate your insights into our future research to achieve a more comprehensive evaluation of generated videos. Thank you for your valuable feedback.\n\n---\n\n>**Q3**: From the content of Algorithm 1's pseudocode, it appears that there is a lack of task-specific algorithmic innovation in the application of reinforcement learning for fine-tuning. The approach seems to follow standard practices without introducing novel techniques tailored to the specific challenges of the task at hand.\n\n**A3**: The primary innovation of this paper lies in demonstrating that MLLMs can provide effective human preference annotation information for text-to-video generation. Based on this, we propose the most comprehensive video preference dataset, VideoPrefer, and the best-performing reward model, VideoRM. Algorithm 1 introduces a framework for fine-tuning text-to-video models with VideoRM, which is **not our main innovation** and indeed lacks task-specific algorithmic innovation. However, **we included it to offer a feasible approach for applying VideoRM to the alignment of text-to-video models**. This framework is **simple** and **effective**, allowing VideoRM to be used for aligning text-to-video models while avoiding the complex application strategies and high computational costs required in previous work[1] for applying image reward models to text-to-video model alignment.\n\nIn future research, we will explore and design more novel framework algorithms to achieve better fine-tuning for the alignment of text-to-video models.\n\n**Reference**\n\n[1] Yuan, Hangjie, et al. \"InstructVideo: instructing video diffusion models with human feedback.\" CVPR 2024.\n\n---\n\n>**Q4**: While the proposed VIDEORM model demonstrates strong performance across various metrics, the paper lacks interpretability experiments. Including such experiments would provide deeper insights into the model's decision-making process and enhance the overall understanding of its effectiveness.\n\n**A4**: Thank you for your interesting suggestion! We will conduct interpretability analyses and experiments on VideoRM in future research to enhance the study's comprehensiveness and completeness, and use these insights to design improved reward models for the text-to-video domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Supplementary rebuttal for Reviewer nZ7n

---

> **Q6**: What or who is deciding the “win” in Figure 2? The GPT-4v assessor?

**A6**: The "win" in Figure 2 is determined by scores from six human experts. The sample with the highest average expert score is considered the ground truth. We apologize for omitting this explanation in the paper, and we will include it in the revised version.

>**Q7**: The limitations do not seem to be explicitly addressed.

**A7**: In Section 5 and Figure 8 of our paper, we have conducted extensive comparative ablation experiments and analyses for choosing hyperparameters. We hope this can serve as a reference for future similar works when selecting hyperparameters. A relatively efficient solution, in our opinion, is to first use different hyperparameters to generate small-scale MLLM annotations and assess their quality. Once the best hyperparameter combination is identified, large-scale annotation generation can proceed (this is the method we used to construct VideoPrefer).

We will include this limitation as a separate section in the updated version of our paper. We also believe that finding efficient and effective methods for choosing hyperparameters in MLLM annotation generation is a very interesting research direction, and we plan to conduct further studies in this area.


## 
## Supplementary rebuttal for Reviewer snrb

---
>**Q7**: Is five participants a sufficiently high number from which to draw conclusions? How many videos did each participant respond to? Was there a mix of non-experts and subject matter experts such as content creators or video editors? If so, were their responses segregated in any way?

**A7**: **Based on previous related work, where three participants were used for human studies in the text-to-image domain[1] and the NLP domain[2], we believe that having five participants rate the videos is sufficient to support our conclusions.** Each participant rated all the results, and every generated video used for evaluation was scored by all participants. The final score for each video was determined by the average score from all participants.

Among the five participants, there were three experts (in image aesthetics, video aesthetics, and image quality) and two non-experts to ensure a more comprehensive and accurate evaluation. During the evaluation process, the five participants did not engage in any form of discussion or communication. Thank you for your question, and we will include these details in the updated version of the paper.

**Reference**

[1] Xu, Jiazheng, et al. "Imagereward: Learning and evaluating human preferences for text-to-image generation." NeurIPS 2024.

[2] Cui, Ganqu, et al. "Ultrafeedback: Boosting language models with high-quality feedback." ICLR 2024.

---

>**Q8**: Did the authors check for participants' engagement, e.g., whether participants responded too quickly or too slowly, and whether any such responses needed to be discounted? Did the authors check for any recency bias or response drifts, where the participants may have responded differently to similar video content over time?

**A8**: We did not check the time each participant took to respond to ensure they didn't respond too quickly or too slowly. However, we implemented a **Post-Labeling Check** mechanism for each participant to ensure the reliability of the final results. Participants were required to review 20% of randomly selected samples for quality checking and rescore them. If the scores differed by more than 25%, that participant's score for the video would not be considered.

---

>**Q9**: In their ablation study, why do the authors consider up to 12 frames? Is this limit due to computational constraints?

**A9**: We considered using up to 12 frames based on the conventional settings from previous works that employed models with clip-like structures for video-related tasks[1,2]. Subsequently, we increased the number of frames to 24 and 32 for further implementation. The experimental results are presented in Table A, where we observed that the performance of VideoRM improved with an increased number of frames. However, considering the deployment cost and speed of aligning video generation models, as well as the performance of VideoRM, we determined that using 8 frames is the most suitable choice.

**Table A. Pair-wise preference prediction at TVGE dataset for VideoRM when using different input frames.** 

| frames                  | 4    | 8    | 12   | 24   | 32   |
| ----------------------- | ---- | ---- | ---- | ---- | ---- |
| **prediction accuracy** | 72.8 | 73.7 | 73.5 | 74.0 | 74.4 |

**Reference**

[1] Luo, Huaishao, et al. "Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning." Neurocomputing 2022.

[2] Wang, Mengmeng, et al. "Actionclip: Adapting language-image pretrained models for video action recognition." TNNLS 2023.

Meta-review: The paper presents a dataset and a learned reward model for video preference to help align text2video generation models. Using GPT-4V, authors automatically label a large dataset of videos to learn this preference model, and fine-tune text-to-video models with this proposed preference model showing improvements in alignment with human feedback through quantitative/human evaluations and ablation experiments.

The paper was reviewed by four expert reviewers, all of who leaned positively towards accepting the paper. ACs concur with the reviewer consensus, and believe this is an important field of study for further improvements in text2video models. Authors must incorporate additional discussions and experimental results shared/promised during the rebuttal phase into their final paper.
Decision: Accept (poster)