Improving Video Generation with Human Feedback
Authors: Jie Liu
Gongye Liu
Jiajun Liang
Ziyang Yuan
Xiaokun Liu
Mingwu Zheng
Xiele Wu
Qiulin Wang
Menghan Xia
Xintao Wang
Xiaohong Liu
Fei Yang
Pengfei Wan
Di ZHANG
Kun Gai
Yujiu Yang
Wanli Ouyang
Venue: NeurIPS 2025 poster ()
Keywords: video generation
human feedback
flow matching
Primary area: deep_learning
TL;DR: This paper presents a systematic pipeline for improving video generation with human feedback, including a large-scale preference dataset, a video reward model, and three alignment algorithms for flow matching models.

Abstract: Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.

Improving Video Generation with Human Feedback
Jie Liu
Ziyang Yuan Xiaokun Liu Mingwu Zheng' Xiele Wu Qiulin Wang' Menghan Xia Xintao Wang' Xiaohong Liu" Fei Yang'
Gongye Liu Jiajun Liang
Pengfei Wan' Di Zhang'
Kun Gaf
Yujiu Yang Wanli Ouyang
"MMLab, CUHK
TTinghua University
*Kling Team, Kuaishou Technology
*Shnngaai Jiao Tong University
Shanghai Al Laboratory
"quual contribution
'Project Leader
Corresponding author
Project page: https:// http://ggoooeeliuggg.......bbb
Abstract
Video generation has achieved significant advances through rectified flow tech- niques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, begin by constructing large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regres- sion for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that Rewrd significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Hlo-RWWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.
Introduction
Advancements in video generation have led to powerful models [57, 33, 32, 5] that produce realistic details and coherent motion. Despite this, current systems still face challenges like unstable motion, imperfect text-video alignment and insufficient alignment with human preferences [84]. In language model and image generation, reinforcement learning from human feedback (RLHF) [52, 90, 71J has proven effective in improving response quality and aligning models with user expectations.
However. applying RLHF to video generation still remains in its infancy. major obstacle the lack of reliable reward signal. Existing preference datasets [19, 73, 47, 76] were collected on earlier T2V models that produced short, low-resolution clips. Reward models trained on such data may miss fine spatial detail and long-range dynamics, while over-penalising glitches that current T2V models already suppress. addition, the design space of VL.M-based reward models remains under-explored, leading to sub-optimal annotation paradigms, reward hacking issues, and entangled aulti-attribute scores. The resulting supervision is therefore noisy, biased, and easily exploited during RLHE.
A second challenge arises from the internal mechanisms of cutting-edge video generation models. Many modern systems employ rectified flow [48, 45], predicting velocity rather than noise. Recent studies [73, 85, 47, 76] have tested DPO [61, 71] and RWR [54, 36, 17] on diffusion based video
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt: motorcycle
forword.
Alignment Objective:
VQ
MQ
TA
max b Seer (r(k - -DDue (owy(tal)))
Flow DPO
Prompts
Alianed VBM
(o) Human Preference Annotation
Flow RWR
Prompts
VL- based RM
Video Tokens
Instructions
(ViC]
Pronpp
[TA]
Reward Scaret
((5
Aligned VBM
VLM--base Reword Model
Reward Guidance
VDal based RM
Tw (r(
Bradley-Tere Model with Ties:
Projection
Σ
1660 x
Pretrained VDM
Rewards
[1.58, -0.67, 2.14]
Reword Model Training
Text-to-Video Alignment
Figure 1: Overview of Video Alignment Framework. (a) Human Preference Annotation. We construct 182k prompt-video triplets, each annotated on Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA). (b) Reward Modeling. A VLM-based reward model is trained under the Bradley- Terry-Modell-ee Ties formulation.
(c) Video Aligmment. From unified RL perspective, we introduce three alignment algorithms for flow-based video generation: Flow-DPO, Flow-RWR and Reward Guidance (Flow-NRG), and provide systematic comparison.
generation approaches. However, adapting existing alignment methods flow-based models intro- duces new questions. A recent attempts [13] for flow matching based DPO even degrade quality compared with the unaligned baseline.
To address these challenges, we present comprehensive investigation into aligning advanced flow-based video generation models with human preferences, shown in Fig. 1. We first collect 16k high-quality prompts and render them with 12 representative T2V models, producing 182k annotated examples across three key dimensions: Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA). We then develop a multi-dimensional vidco reward model, systematically analyzing how different annotations and design choices affect its performance, These dimensions can be aggregated into a total reward that reflects the overall preferences humans.
Armed with reliable reward model, we revisit RLHF algorithms for rectified flow. From a unified reinforcement learning perspective that maximizes reward with KL regularization, we derive two training- time strategies-Flow Flow-DPo DPO and Flow-RWR, and an inference-tine We discover that simple extension of Diffusion-DPO performs poorly because timestep-dependent KI. term pushes the policy to overfit the objective at higher noise levels, making the model vulnerable to reward hacking. In contrast, our Flow-DPO removes this time-dependent term and keeps the KL weight constant, showing robust performance across all dimensions. Flow-NRG is efficient inference-time alignment algorithm that applies reward guidance directly to noisy videos during denoising process. It allows users to apply arbitrary weightings to multiple alignment objectives during inference, eliminating the need for extensive retraining to meet personalized user requirements.
technique, Flow-NRG.
Our contributions can be summarized as follows:
Large-Scale Preference Dataset: We create 182k-sized, multi-dimennional, human-labeled video generation preference dataset from 12 modern video generation models.
Multi- Tulti--Dimnniinal Reward Modeling: We propose and systematically study a multi-dimmensional video reward model, investigating how different design decisions influence its rewarding efficacy.

---

SideaGen-RewarrBenetee We create benchmark for modern reward models by annotating prompt-video pairs from VideoGen- This dataset, consisting of diverse prompts and videos generated by modern VDMs, results in 26.5k annotated video pairs with preference labels.
Flow-Based Alignment: From a unified RL perspective, we introduce two training-time alignment strategies (Flow-DPO, Flow-RWR) and one inference-time technique (Flow-NRG), Experiments show that Flow-DPO with fixed KL term outperforms other methods. And Flow-NRG allows custom weightings of multiple alignment objectives during inference.
Related Work
Reward modeling [75, 77, 31, 86, 41] trains CLIP-based models on human preference datasets, while newer approaches use VL.Ss with regression heads to predict multi nulti-dimensional scores. Learning paradigms include point-wise regression [19, 76] and pair-wise comparison via Bradley-Terry loss. However, most video reward models focus on short, low-quality videos from the pre-Sora era [51] and lack rigorous evaluation of design choices. We address these limitations by targeting modern video generation and exploring broader reward modeling strategies, For alignment, image generation has adopted RLHF-style techniques such as reward backpropagation [58, 771, RWR [36]. DPO [61, 71], PPO [2, 16], and training-fie free methods [80]. Concurrent efforts [17, 47, 76, 85, 73] extend DPO/RWR to diffusion-basee video models using old generation models or image-level rewards. We build on this extending DPO to flow-based video generation, proposing Flow-DPO. More comprehensive discussions of related work can be found in Appendix A.
VideoReward
Robust RLHF begins with reward model that faithfully mirrors human preferences, yet existing efforts are limited in two key respects: (1) Data: existing video-preference datasets were curated for earlier T2V models and mismatch with what users prefer modern video generation models; (2) Model design: the key technique choices for VL.M-ased reward models remain largely uncharted. Fig. (a), (b) summarises our solution. We first build a large-scale preference dataset collected with state-of-the-at T2V models; then perform systematic study of reward mowardmodellng design.
3.1 Human Preference Data Collection
Existing human preference datasets for video generation [49, 26, 19, 73, 76] were primarily built on early, low-resolution T2V models that produced short, artifact-laden clips. As VDMs continue evolve, modern T2V models, however, generate longer, higher-fidelity videos with smoother motion. Consequently, legacy datasets no longer accurately reflect what users prefer today. Reward models trained on such collections may miss fine spatial details and long-ram oongrrange dynamics while over- weightiig temporal glitches already mitigated by current models. To bridge this gap, we develop new preference dataset expressly for state-of-the- VDMs.
Table 1: Statistics of the collected training dataset. We utilize 12 T2V models to generate 108k videos from 16k unique prompts, resulting in 182k annotated triplets. Each triplet consists of prompt paired with two videos, and corresponding preference annotations.
T2V Model
Date
#Vidcos
#Anno Triplets
Resolution
Duration
Pre-Sora-Ere Models
Gen2 [63]
23.06
6k
13k
768 x 1408
4s
SVD [3]
23.11
6k
13k
576 x 1024
4s
Pika 1.0 [34]
23.12
6k
13k
720 1280
Vega [70]
23.12
13k
576 1024
Pix Verse v1 [56]
24.01
6k
13k
768 x 1408
4s
HiDream [21]
24.01
0.3k
0.3k
768 x 1344
5s
Modern Models
Dreamina [6]
24.03
16k
68k
720 1280
Luma [50]
24.06
16k
57k
752 x 1360
5s
Gen3 [64]
24.06
16k
55k
768 x 1280
5s
Kling 1.0 [33]
24.06
6k
33k
384 X 672
5s
Pix Verse v2 [56]
24.07
16k
58k
576 1024
5s
Kling 1.5 [33]
24.09
28k
704 1280
3

---

BT Reward Model
Bradiey-Terry Reward Model
Bradteyderry With-Ties Boward Hodel
Regression Reward Model
Accursry (N0
0.2
0.00 Training Data Fraction (log scale)
1.00
Awiss
Tied
wins
Awiins
Tled
B wins
Figure 2: BT vs. Regression. Accuracy curves across log-scaled data fractions.
Figure 3: BT vs. BTT. Score-difference distribu-
tions (Ar) for BT (left) and BTT (right).
Prompt Collection and Video Generation. We collect diverse prompts from the Internet, cat- egorize them into 8 meta-categories aanial architecture, food, people, plants, scenes, vehicles, objects-and and expand them with GPT-4o. After removing repetitive, irrelevant, or unsafe entries, we refine them with our in-house prompt rewriter, yielding 16000 high-quality prompts. 12 T2V models of varying capabilities then render these prompts into 108k videos, which we pair to form 182k triplets, each comprising a prompt and two corresponding videos from distinct VDMs. Comprehensive dataset statistics are provided in Tab. 1 and Fig. 8.
Multi-dimennional Annotation. Professional annotators were hired to view each triplet and record pairwise preferences(A wins /Ties / wins) separately for Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA), producing a three-label vector per triplet. The same annotators also assign 1-5 Likert scores to each individual video, enabling later studies that compare pointwise and pairwise supervision. We reserve 13 000 triplets whose prompts never appear in training as a validation set. Detailed annotation protocols provided in Appendix G.
3.2 Reward Modeling
Prior works [19, 73, 76] on VLM-based reward models have proved effective for both evalua- tion [75, 19] and optimization [36, 71. 59]. However, their core design choices remain insufficiently explored, and current methods still suffer from issues like annotation paradigms, reward hacking and entangled multi-attribute signals. We select the lightweight Qwen2-VL-2B [72] as our backbone. and systematically examine three key designs and demonstrate how each yield cleaner, more reliable reward signal for RLHE. We also conduct a comprehensive ablation study on the evaluation benchmark in Table 6 of Appendix E to further validate the key design choices.
We first investigate two reward learning paradigms: the Bradley-Terry (BT) model [4] and pointwise score regression. The BT model formulates preference learning as a probabilistic ranking task. Given a prompt and paired videos (xx .b)). it optimizes LBT = -E log (a (r(x)' y) r(æ y)))], where the expectation is taken over (y, ay xx) ~ D.
Score Regression v.s. Bradley-Terme
In contrast, score regression directly predicts scalar quality score E R using the MSE loss: Crey = E [x y) 2|13] where the expectation is taken over (y. 2o. 2) ~ D. Since our training dataset includes both pointwise scores and pairwise preferences from the same annotators, we can directly compare between the two annotation paradigms. We train both types of reward models on increasing subsets of the training set and report the best validation accuracy averaged over VQ, MQ. and TA. Fig. presents these results.
As the dataset grows, both the BT and regression-style models improve in accuracy, while the BT model remains consistently superior. This advantage stems from the nature of pairwise annotations, which capture subtle relative distinctions more effectively. Even when two videos receive identical pointwise scores, annotators can still differentiate subtle quality differences.
Ties Matters. While vanilla BT model is widely used to capture human preferences from chosen- rejected pairs, the importance tie annotations often overlooked. Inspired by recent work [46], adopt Bradley- Terry model with ties (BTT) [62], extension that accounts for tied preferences.

---

Formally, BTT defines a tripartite preference distribution:
(0 - exp(ra) exp(re)
Tie (20 = 0
(exp(ra) ( (xp((xx)) (xpp(r))) e exp(ra)
Po x
A preferred (ag² xx))
exp(rB) + cxp((r)
exp(rn)
B preferred (ag > xσ²
exp(rrA) + exp(rB)
where denotes preference choice, 8 > 1 controls the tendency toward ties, with larger 0 increasing tie probability. We empirically set = 5.0 and train the BTT model by minimizing negative log-likelihood:
LETT -
Σ (E(+,T,=)
log 2o ab))
(2)
We train BT and BTT reward models under identical settings and visualize Ar r( - y) on the validation set (Fig. 3). Although the BT model handles chosen/rejected pairs well, it struggles handle fie--often assigning sizeable Ar to many tie pairs, conflating them with clear preferences. By contrast, BTT learns a flexible decision boundary that clusters ties near zero while preserving large margins for decisive wins and losses, yielding more reliable feedback for downstream RLHF,
Token Positioning. A common approach in LLM MLLM- reward modeling [52, 69, 19] attaches linear projection head to the final token predict multi-dimensional scores. This method forcing prompt-independent and prompt-dependent cues into one vector, causing context leakage: the same video can receive different visual-quality scores when paired with different prompts. We eliminate this entanglement with simple eoken-positionin strategy. As shown in Fig. (bb) two context-agnostic tokens, [VQ], [MQ], are inserted immediately after the video and before the prompt. so they can attend only to visual content. A context-aware token, [TA]] is placed after the full prompt, allowing attend both the video and the text. The final-layer embeddings these tokens are then mapped to cimensioonsppeciie leakage, stabilizes visual and motion assessments, and maintains parameter efficiency. The full input template is provided in Appendix K.
scores via shared linear layer. This design removes context
Video Alignment
With a high-fidelity reward model established, we next introduce three alignment methods for flow- based generation models under unified RL objective: two training-time algorithms Flow-DPO and Flow-RWR, and one inference-time guidance technique, Flow-NRG (Fig. 1 (c)).
4.1 Preliminaries
Rectified Flow. Let ar q(20)) denote data sample from the real data distribution, and a p(æ1) denote noise sample, where Co. 21 Rd Recent advanced image [15] and video generation models [57, 32] adopt the Rectified Flow [48], which defines the "noisy" data x; as T; = (1 2g + 11,, where & [0, 1]. Then
train transformer model regress the velocity field 00(2,, t) by minimizing the Flow Matching objective [45, 48]:
((0) = E,, mo>g(s)), mimp(an)) |
vo vo (x, (o(,t)))]]]
where the target velocity field is 21
The goal of RLHF is to learn a conditional distribution (o y) that maximizes the reward r(zo, y) while controls the KL-divergence from the reference model Pee via coefficient 8:
RLHE
max E momno (mall) [r( y)] ,3 DKL [pe(2c) ()) Pher(2o 1)]
(3)
5

---

4.2 Flow-DPO
Consider a training set = (y, a8 mg). where y is the prompt and human annotations prefer generated video zij zg (i.c., 26 > z)). Direct Preference Optimization (DPO) [61] aligns models with human preferences by analytically solving the RLHF objective (Eq. 3) and optimizing the policy
via supervised learning. Extending this idea to diffusion models, Diffusion-DPO [71] derives
DPO-style loss under the diffusion paradigm. The resulting objective (D) is formulated as:
-E log (-2 -1c" -coc"= 1)] cne(w 2))l" - (Ile' colle' ()1 -1Ie -re (ooe((t))1)))) ) (4)
The expectation is taken over samples {æ/ m/} ~ 'D and the noise schedule t. In Rectified Flow, we relate the noise vector c" velocity field v*. Specifically, Lemma B.1 in Appendix B shows that
Ee"
'pred [z; f)] (1 I)² Uprea ( t)ll?
(5)
By substituting Eq. into Eq. 4, we obtain the final Flow-DPO loss CFD(O):
B1 (1100 ooo ;arr"",)) ())1 1|00" (xr"" ; !)]?
E
log (-2 2
(Ile' - -oo(a), t)ll² -0]12_0 1| vter(a),, t)(lz)
(6)
where 8, = 3 (1 - 1)a Intuitively, minimizing LFD guides the predicted velocity field ny closer the target velocity ex of the "preferred" data, while pushing it away from o' (the "less preferred" data). The strength of this preference signal depends on the differences between the predicted errors and the corresponding reference errors. We provide Flow-DPO pseudo-code in Appendix C.
Discussion on 3r. The KI. coefficient 3g controls how far the learned policy is allowed to deviate from the reference model [61, 71]. A direct derivation yields the schedule 3, B(1 t)². The penalty Be vanishes as & approaches and reaches 3 at model to prioritize alignment at higher noise levels. Unlike in diffusion models, our experiments reveal that this schedule degrades alignment performance in rectified flow, leading to reward hacking and visual artifacts. Inspired by similar observation in DDPM's [22] training objective, where discarding the weighting denoising score matching improves sample quality, we instead adopt a constant 3. This adjustment leads to more stable training and improved alignment across all reward dimensions. We provide a more detailed discussion of this Section 5.2.
This scheduling strategy causes the
4.3 Flow-RWR
Drawing inspiration from the application of Reward-weighted Regression (RWR) [55] in diffu- sion models [36, 17], we propose a counterpart for flow-based models based on expectation- maximization [10]. Starting from the general Kl.rregularized reward-maxing awwrr-maxiiizzation problem in Eq. 3, prior work [61] shows that its optimal closed-form solution can be written as:
(F r(
po y)
Z(y) Prer(aoo y) exp
where Z(y) = En, (re y) exp( r r y)) is the partition function. Following [17], we can obtains the RWR loss:
CRwR( (0) Ey s. [exp(r(x0 y)) es (x,, =]]
(8)
For rectified-fllww models, we formulate this as reward velocity regression:
LrwR (0) exp(((pxxx v)) 4,
As in Flow-DPO, omit the (1 t)² factor for better performance.

---

Table 2: Preference accuracy on GenAI-Bench and VideoGen-Rewardeenn For ties-excluded accuracy, we calculate accuracy using only the data labeled as "A wins" or "B wins". For ties-includer accuracy, we use the algorithm from Deutsch al. [11]. which tests various tie thresholds and selects the one that maximizes three-class accuracy. Bold: best performance.
GenAI-Bench
BideoGen-RewarBBeee
Method
Overall Accuracy
Overall Accuracy
VQ Accunacy
MQ Accuracy
TA Accuracy
w/ Ties
w/o Ties
w/ Ties
w/o Ties
w/ Ties
w/o Ties
w/ Ties
wo Ties
w/ Ties
w/o Ties
Random
33.67
49.84
41.86
50.30
47.42
49.85
59.07
49.64
37.25
501.41
VideoScore [19]
49.03
71.69
41.80
50.22
47.41
47.72
59.05
$1.09
37.24
50.34
LIFT [73]
37.06
58.39
39.08
57.26
47.53
55.97
59.04
54.91
33.79
55,43
VisionRewrd [76]
51.56
72.41
56.77
67.59
47.43
59.03
59.03
60.98
46.56
61.15
Ours
49.41
72.89
61.26
73.59
59.68
75.66
66.03
74.70
53,80
72.20
4.4
Noisy Reward Guidance
Recall that the KL-regularized RL objective (Eq. 3) admits a Glosed-form solution(Eq. 7), which transform the original distribution Prer y) into the new target distribution pa y). Since the constants 3 and w can be absorbed into r(wo: y), the closed-form solution becomes:
pa (2to y) x Prer(ato y) [exp(r(ato: y))
(10)
where IR controls the strength of the reward guidance. For rectified flow, as we proved in Appendix B.2, this reweighting can be achieved by shifting the velocity field:
y) (are y) 1 Vr( y):
(a(
(11)
This modification of the marginal velocity field alters the sampling distribution to match the target form in Eq. equation 10. Since this formulation is structurally similar to classifier guidance [12, 67], we refer to it as reward guidance. Pseudo-code is provided in Appendix C.
Efficient Reward on Noisy Latents. Computing Vr in pixel space requires kack-propagating through the full VAE decoder, which is computationally expensive. To address this, we propose training lightweight, time-dependent reward model (:, directly in latent space. For each preference pair (x. 2²",, we apply identical noise both videos, assuming their relative preference remains unchanged. We then adopt the Bradley-Terry loss to learn the reward function from these noised videos. Leveraging the fact that modern VDMs are already well trained on noisy latents, we can reuse a few early layers from the pretrained backbone to construct the reward model, avoiding the need for heavy retraining. We apply Eq. equation at each inference step (except t 1), enabling efficient inference-time alignment in latent space.
5
Experiments
5.1 Reward Learning
Training Setting. We use Qwen2-VL-2B [72] as the backbone of our reward model, trained with BTT loss. Several observations were made during training. First, higher video resolution and more frames generally improved the reward model's performance. Second, using a stable sampling interval instead of a fixed frame number significantly enhanced motion quality evaluations, especially for videos of varying lengths. In practice, we sample videos at 2 fps, with resolution of approximately 448 x 448 pixels while preserving the original asocet ratio. Hyperparameters in Appendix I.
Evaluation. We evaluate our reward model on two benchmarks targeting different generations of T2V models: (1) GateoGen-RRewrrBeeee
Built upon the third-party prompt-video dataset VideoGen- Evaal [84], this benchmark targets modern T2V models. We address the lack of human annotations in VideoGen-
by manually constructing 26.5k triplets and hiring annotators to provide pairwise preference labels. Annotators also assess overall video quality, serving as universal label across all dimensions. (2) GenAT-Bench [29]: GenAl-Bench features short (2-seconds) videos outputs. A detailed comparison between the two benchmarks is provided in Appendix H.1. We evaluate our reward model against existing baselines, including VideoScore [19], as well as two concurrent works: LiFT [73]
generated by pre-Sora-era T2V models, enabling evaluation
7

---

Additional Table 3: Multi-dimensional alignment with VQ.MQ:TA= 1:1:1. Bold: Best performance. Although Flow-DPO with a timestep-dependent 3 achieves high VQ and MQ reward win rates, it exhibits significant reward hacking. In contrast, Flow-DPO with constant achieves high VQ, MQ, and TA scores while avoiding reward hacking.
VBecch
VideoGeu-Eval
TA-Hard
Method
Quality
Sementic
MQ
TA
VQ
MQ
VQ
MQ
Pretrained
83.19
84.37
78.46
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
SFT
82.31
83.13
79.04
65.21
$2.84
61.27
57.75
76.06
57.25
Flow Flow-RWR RWR
82.27
83.19
78.59
51.55
63.9
53.43
59.05
69.7
48.35
61.97
78.87
55.71
Flow-DPO (9, A11 e)*)
80.90
81.52
78.42
87.78
82.36
51.02
91,23
14
84,29
83.10
38.03
Flow DPO
83.41
84.19
80.26
93.42
69.08
75.43
90.95
81.01
68.26
77:46
71.43
73.24
Table 4: Single-dimensional alignment with TA. Bold: Best performance. Flow-DPO with constant the most effective method, achieving best performance without reward hacking.
VBench
Evde
TA-Hard
Method
Total
Quality
Semantic
TA
TA
TA
Pretrained
83.19
84.37
78.46
50.00
50,00
50,00
SFT
82.71
83.48
79.62
52.88
53.81
64.79
Flow-RWR
82.40
83.36
78.58
59.66
66.20
Flow-DPO (8, (8(1 2)
82.35
83.00
79.75
63.67
55.95
71.83
Flow: DPO
83.38
84.28
79.80
69.09
65.49
84.51
and VisionReward [76]. Consistent with practices in LLM evaluation [35], we pairwise accuracy, reporting both ties-included [1I] and ties-excluded accuracy. We calculate overall accuracy on GenAI- Bench and dimension-speeificc (VQ, MQ, TA) accuracy VideoGen-RewardBeen evaluation details are in Appendix H.2 and Appendix H.3
Main Results. Tab. 2 presents the pairwise accuracy across both benchmarks. VideoScore per- forms well on GenAl-Bench but fails VideoGen- RewardBench, indicating poor generalization to modern T2V models. LiFT improves over VideoScore on modern videos but remains below 60% accuracy, showing limited pairwise discrimination ability. VisionReward demonstrates competitive performance on GenAl-Bench but underperforms on BeteoGGennRewarBBeeee and motion dimensions under ries-included settings. This drop stems from its difficulty in assessing the improved fidelity and motion smoothness of modern outputs. In contrast, our method VideoRe- ward outperforms all other models on with human preferences on modern T2V generations. Moreover, despite being trained on a disjoint dataset (see Fig. 10), still achieves comparable performance on GenAI A--Bench, indicating robust generalization across different eras of T2V models. Ablation studies are provided in Appendix E.
especially on visual
showcasing its strong alignment
5.2 Video Alignment
Training Setting. Our pretrained model Prer is an internal, research-purpose video generation model based on Transformer architecture [53], which is trained using rectified flow (see Appendix 7 for details). Following SD3 [15], all alignment experiments fine-tune the Transformer using LoRA [24]. For training- based alignment methods, including SFT. Flow-DPO, and RWRR we adopt Vidco- Reward provide the reward signals. For reward guidance, we employ the latent reward model to generate rewards. For supervised fine-tuning (SFT), we utilize only the "chosen data". Following Rafailov al. [61], we employ VideoReward the ground -ouuu reward model to simulate human feedback and relabel our training dataset, ensuring that models optimised on these synthetic labels can be evaluated fairly by the same reward function. Hyperparameter settings are provided in the Appendix I.
Evaluation. We evaluate performance using both automatic and human assessments. For automatic evaluation, we measure the win rate (via VideoReward) and the Vhench score. The win rate the proportion of cases where Rewrd assigns higher reward to the aligned model than to the pretrained model. Vbench is a fine-grained T2V benchmark that assesses Quality and Semantic alignment. For human evaluation, each sample is reviewed by two annotators, with a third annotator resolving disagreements. Identical random seeds are used across methods for strict comparability. Our evaluation uses prompts from Vbench, VideoGen- Eval, and a new TA -ard set that stresses
8

---

cowboy open plain sky Lighting on landscape.
long pänk nightsown
cozy, warnly
alchenist with potion vials herbs
Original
Flox-DPO
Figure 4: Comparison of videos generated by the original model and the Flow-DPO aligned model.
Pretrained Wins
Tias
DPD Wins
Total
27.0%
29 0%
BX
Visual Quality
21.0%
47.2%
31.8%
Motion Quality
32.2%
TA-tard
VBunch
All
Alignment
2%
43
300e
log(8)
Percenlage
Figure 5: Human evaluation of Flow-DPO aligned model vs. pretrained model on VideoGen-Eval.
Figure 6: Accuracy of time-dependent 3g E.S. constant /3 for TA.
complex semantics, since we notice that the Vbench and VideoGen-Eval prompts are relatively straightforward in terms of text alignment. Appendix L lists subset of TA-Hard prompts.
Multi-dimensional Alignment. We use linear scalarization [40], r 3 (r'vg + Tmg + Tia), to aggregate multi-dimmnnsional preferences into single score, and forming dataset D {(y, x x²) where r(=" ,y r(æ', y). Table 3 shows that Flow-DPO with a constant 3 significantly improves over the pretrained model across all three dimensions and outperforms both SFT and Flow-RWR. In contrast, Flow-DPO with dimestep-dependent underperforms the pretrained model on TA, despite high VQ/MQ reward win rates due to reward hacking. Mcanwhile, the constant-8 3 variant achieves high VQ/MQ scores without such hacking issues, suggesting it learns TA more effectively. We discuss this further in Sec. 5.2. Figures 5 confirms these findings in human studies.
Single-dimensional Alignment. We also investigate the ability of different methods on a specific task: TA. Tab. shows that Flow-DPO with constant 3 achieves best performance across all datasets.
Reward Guidance. We apply linear scalar- ization to combine dimension rewards into a weighted sum and backpropagate gradients to the noised latent with guidance strength 100. Table shows that users can steer genera- tion toward custom trade-off's by simply choos- ing custom weights.
Table 5: Reward guidance on VideoGen-Eval.
VQ.MQ:TA
VQ
MQ
60.56
0.1.0.1:0.8
66.50
63.73
60.86
0.1.0.1:0.6
68.94
67.59
53.28
0.5:0.5:0
86.43
93.23
26.65
Ablation on 3. We meticulously adapted diffusion- -DOO to flow-based models, resulting in Equa-
tion 6, where 3, = (1 2)2 Figure 6 shows that under various 3 values, constant 3 consistently outperforms the dimestep-dependent variant. This is likely because varying 9 across timesteps leads to uneven training [18], given that T2V models use shared weights for different noise levels.

---

Conclusion
We present a large-scale preference dataset of 182k human annotations covering visual quality, motion quality, and text alignment for modern video generation models. Building this dataset, we introduce VideoReward, a multi-dimensional video reward model, and establish the VideoGen- RewardBench benchmark for more accurate and fair evaluations. From unified reinforcement learning perspective, we further propose three alignment algorithms tailored to flow-based video generation, demonstrating their effectiveness in practice.
7 Limitations & Future Work
In our experiments, excessive training with Flow-DPO led to significant deterioration in model quality, despite improvements in alignment across specific dimensions. To prevent this decline, we employed LoRA training. Future work can explore the simultaneous use of high-quality data for supervised learning during DPO training, aiming to preserve video quality while enhancing alignment. Additionally, our algorithms have been validated on text-to-video tasks; future work can extend the validation other conditional generation tasks, such image-to-vin -oiideo generation. Moreover, despite our efforts like incorporating tie annotations into the modeling, our VideoReward model still carries a potential risk reward hacking, where human assessments indicate a marked decrease in video quality, yet the reward model continues assign high scores. This issue arises because the reward function is differentiable, making it susceptible manipulation. Future research should focus on developing more robust reward models, potentially by incorporating uncertainty estimates and increasing data augmentation. Additionally, there is potential to apply more RLHF algorithms, such PPO, to flow-based video generation tasks.
Acknowledgements
This work was supported by the JC STEM Lab of AI for Science and Engineering, which is funded by The Hong Kong Jockey Club Charities Trust and the Research Grants Council of Hong Kong It was also supported by the National Natural Science Foundation of
(Project No. China (Grant No. 62576191).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 00X1v:2303.08774, 2023.
[2] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv: 13301, 2023.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:23/1./5122 2023.
[4] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[6] Capcut. Dreamina.
http://ccccccoo///ato//////oo//a 2024.
[7] Chaofeng Chen, Annan Wang. Haoning Wu. Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffusion models with text-encoder reinforcement learning. In European Conference on Computer Vision, pages 182-198. Springer, 2024.
[8] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In
10

---

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310-7320, 2024.
[9] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
[10] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statisfical society: series (methodologicul), 39 (1):1-22, 1977.
[11] Daniel Deutsch, George Foster, and Markus Freitag. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305. /4324, 2023.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[13] Carles Domingo-Enrinn Ennich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint (eXxi:2409.08866 2024.
[14] Hanze Dong, Wei Xiong. Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint 0rXiv:2344.06661 2023.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[16] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[17] Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with ai feedback. arXiv preprint arXiv:2412.02611 2024.
[18] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7441-7451, 2023.
[19] Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj. al. Videoscore: Building automatic metrics simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406.1555 15252, 2024.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] HidreamAI. Hidreamai, http:///www.hdreemaa.comm 2024.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[23] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv. Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
[24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 216.05.066.0988,, 2021.
11

---

[25] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36: 36:78723-78747, 2023.
[26] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807-21818, 2024.
[27] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[28] Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint wrXi::245.001483, 2024.
[29] Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485 2024.
[30] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
1121v914 2014.
[31] Yuval Kirstain, Adam Polyak. Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36666 36663, 2023.
[32] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600. 2024.
[33]
Kuaishou. Kling ai. https://lingaikkuassoou...c 2024.
[34] Pika Labs. Pika 1.0. https://pika.art/ 2023.
[35] Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13780 2024.
[36] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
[37] Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405. 18750, 2024.
[38] Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design. arXiv preprint arXiv:2410.05677 2024.
[39] Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023.
[40] Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization. IEEE transactions on cybernetics, 5166):3103-3114 2020.
[41] Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun. Jordi Pust- Sarah Young. Feng Yang, al. Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19401-19411, 2024.
[42] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance each step. arXiv preprint arXiv:2406. 043144 2024.
12

---

[43] Ji Lin. Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689-26600 2024.
[44] Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujuu Yang. Criticbench: Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402. 14809, 2024.
[45] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arX1v:2210.02747, 2022.
[46] Jinsong Liu, Dongdong Ge, and Ruihao Zhu. Reward learning from preference with ties. arXiv preprint arXiv: 240.0033,, 2024.
[47] Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing Hc, Renjie Pi, and Qifeng Chen. Vidcodpo: Omni- preference alignment for video diffusion generation. arXiv preprint arXiv:2412.14667 2024.
[48] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint Sitiv:2209.000000 2022.
[49] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang. Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22139-22149, 2024.
[50] LumaLabs. Dream machine. http::/tummlabssadddrem---cmc
2024.
[51] OpenAI. Video generation models as world simulators. https//lopenii.com/mneee//iii generation-modelt -noeraatoon-mdels-sss------ www.-roooddddds-wwwss-ssmaaaaaa 2024.
[52] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[53] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[54] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00//7 2019.
[55] Jan Peters and Stefan Schaal. Reinforcement learning by rewart-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750) 2007.
[56] Pix Verse. Pixverse. https://pixverse.aii 2024.
[57] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yar Yao Ma, Ching Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024.
[58] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXiv.2310.03739 2023.
[59] Mihir Prabhudesai, Russell Mendonea, Zheyang Qin, Katerina Fragkiadaki. and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737. 2024.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[61] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[62] PV Rao and Lawrence I. Kupper. the bradley-terry model. Journal of the American Statistical Association, 62(317):194-204, 1967.
Ties in paired-comparison experiments: generalization of
[63] Runway. Gen-2: Generate novel videos with text, images or video clips.
htpp://ruwagwl.coomereeeeeeeeee 2023.
[64] Runway. Gen-3. https://ruwwayml.com 2024.
[65] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[66] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-pls -lay controllable generation. In International Conference on Machine Learning, pages 32483-32498. PMLR, 2023.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 I. 133466, 2020.
[68] Zhiwei Tang. Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung-Hur Chang. Tuning-free alignment of diffusion models with direct noise optimization. arXiv preprint arXiv:2405. (88.8) 2024.
[69] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[70] VegaAI. Vegaai. https://www..eegaa..t
2023.
[71] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228-8238, 2024.
[72] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 240. 2024.
[73] Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang. Cheng Jin, and Hao Li. Lift: Leveraging human feedback for text-to- text-to-video model alignment. arXiv preprint 24Xiv:2412.048/ 2024.
[74] Tianhe Wu, Jian Zou, Jie Liang, Lei Zhang, and Kede Ma. Visualquality-rl: beasoning-inducd image quality assessment via reinforcement learning to rank. arXiv preprint arXiv:2505.14460, 2025.
[75] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint trXiv:2306.09::0 2023.
[76] Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang. Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-graines multi-dimensional human preference learning for image and video generation. arXiv preprint arxiv:2412.21059, 2024.
[77] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
14

---

[78] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen. Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941-8951, 2024.
[79] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-vides diffusion models with an expert transformer. arXiv preprint 24Xx:24408.0607,, 2024.
[80] Po-Hung Yeh, Kuang-Huei Lee, and Jun-Cheng Chen. Training-free diffusion model alignment with sampling demons. arXiv preprini arXiv:2410.05760, 2024.
[81] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang, Freedom: Training- free energy energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23174-23184, 2023.
[82] Hangjie Yuan, Shiwei Zhang. Xiang Wang. Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: instructing video diffusion models with human feedback. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6463-6474, 2024.
[83] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. arXiv preprint arXiv:2402. 102/0, 2024.
[84] Ailing Zeng, Yuhang Yang, Weidong Chen, and Wei Liu. The dawn of video generation: Preliminary explorations with sora-like models. arXiv preprint arXiv:2410.05227, 2024.
[85] Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Xuefeng Xiao, Weilin Huang, and Kai Han. Onlinevpo: Align video diffusion model with online video-centric preference optimization. arXiv preprint arXiv:2412. 15159, 2024.
[86] Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Zhang, and Zhongyuan Wang. Learning multi-dimensional human preference for text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8018-8027, 2024.
[87] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. In European Conference on Computer Vision, pages 1-17. Springer, 2024.
[88] Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311. 13443, 2023.
[89] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. 2024.
arXiv preprint urXiv:2412.2044
[90] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preffreence-foralll Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03700. 2023.
15

---

Appendix of Improving Video Generation with Human Feedback
Our Appendix consists of 8 sections. Readers click each section number to navigate to the corresponding section:
Section provides a review of related works.
Section B provides detailed derivations and lemma proofs for Flow-DPO, Flow-RWR, and Flow-NRG.
Section provides pseudo-code of Flow-DPO and Flow-NRG.
Section presents the architecture the internal video diffusion model employed in our work.
Section offers ablation study on key design choices of our reward model.
Section F summarizes dataset statistics.
Section G explains details of human annotaion and guidelines.
Section provides details about reward model evaluation, including comparison of the two evaluation benchmarks, evaluation procedures of all methods, and the metrics employed.
Section lists hyperparameters for our reward modeling and alignment algorithms.
Section provides additional visual comparisons between the original model and the Flow-DPO aligned model.
Section K provides the input template used for reward model.
Section I. provides part of prompts used in our TA-Hard dataset.
Section M provides broader impacts.
A
Related Works
Evaluation and Reward Models. Evaluation models and reward models play pivotal role in aligning generative models with human preferences. Earlier approaches and benchmarks [25, 49, 26] relied on metrics like FID [20] and CLIP scores [60] to assess visual quality and semantic consistency, Recent works [75, 77, 31, 86, 41] have shifted towards utilizing human preference datasets train CLIP-based models, enabling them to predict preference scores with improved accuracy. With the advent large vision-language models (VLMss [1, 72], their powerful capabilities in visual understanding and text-visual alignment make them natural proxy for reward modeling. A common approach involves replacing the token classification head VLMs with regression head that predicts multi-dimensional scores for diverse evaluation tasks.
Two main learning paradigms have emerged based on the type of human annotation. The first paradigm relies on point-wise regression, where the model learns to fit annotated scores [19] or labels [76] directly. Another paradigm focuses on pair-wise comparisons [74], leveraging Bradley- Terry (BT) [4] loss or rank loss to model relative preferences, which is largely unexplored for video reward model. Beyond these methods, some works [73] also leverage the intrinsic reasoning capabilities of VLMs through -uuee [39, 44], where VL.Ms are adopted to generate preference judgments or scores in textual format through instruction tuning. Despite these promising advances, most existing video reward models primarily focus on legacy video generation models, typically from the pre-Sora [51] era, which are constrained by short video durations and relatively low quality. Furthermore, the technical choices underlying the vision reward models remain underexplored. Our work seeks to address these limitations by focusing more on modern video generation models and investigating broader range of reward modeling strategies.

---

Alignment for Image & Video Generation. In large language models, Reinforcement Learning from Human Feedback (RLHF) improves alignment with human preferences, enhancing response quality [52, 27]. Similar methods have been applied to image generation, using reward models or human preference data to align pretrained models. Key approaches include: (1) direct backpropagation with reward signals [58, 9, 77, 59]; (2) Reward- Weighted Regression (RWR) [54, 36, 17]; (3) DPO- based policy optimization [61, 71, 42, 14, 78, 42, 83, 47, 85, 17]; (4) PPO-based policy gradients [65] [2, 16, 87, 7]: and (5) training-free ffree alignment [80, 68, 66]. These methods have successfully aligned image generation models with human preferences, improving aesthetics and semantic consistency. They focus on improving the accuracy of rewards noised images using reward models trained on clean images, whereas our Flow-NRG directly trains aoise- reward models to obtain accurate gradients. Our work applies the DPO algorithm [61, 71] to flow matching in video generation. Concurrent work [13] also explores similar things in image generation. However, they reports worse performance for the DPO-aligned model compared to the unaligned one. We argue that the originally derived Flow-DPO algorithm imposes a stronger KI. constraint at lower noise steps, resulting in suboptimal performance. In contrast, using a constant KL constraint significantly improves performance on certain tasks. Some prior work explores aligning video generation models using direct backpropagation with differentiable rewards [82, 38, 37, 59], often relying on image reward models [31, 75]. However, these approaches cannot be directly applied to modern T2V generation with a VLM-based video reward and large VAE decoders, as they exceed the memory limits of existing GPU setups, requiring specialized engineering techniques to handle.
Discussion with Concurrent Video Alignment Works. Several concurrent works also explore aligning vidco generation models using feedback. Furuta et al. [17] derives unified probabilistic objective for offline RL fine-tuning of text-to-video models for DPO and RWR. VideoDPO [47] intro- duces re-weighting factor the Diffusion-DPO loss to adjust the impact of each pair, encouraging the model to learn more effectively from pairs with clearer distinctions. VisionReward [76] ensures that all dimensions of the chosen data outperform those of the rejected data, addressing the issue of confounding factors in preference data. Online VPO [85] presents an online DPO algorithm to tackle off-policy optimization. LIFT [73] proposes applying RWR on synthesized datasets while simultaneously performing supervised learning on real video-text datasets, as synthesized videos often suffer from low temporal consistency. All of these works use Diffusion-DPO or Diffusion-RVV and focus on aligning diffusion-basse video generation models, where the videos in the preference datasets are either generated by earlier open-source models or use image reward models directly. In contrast, our work explores alignment techniques for advanced flow-based video generation. We extend Diffusion- DPO into Flow-DPO, but our derivation reveals that the parameter |3 (the KL diver- gence constraint) in Flow-DPO is pheeste--depeddent, which leads to suboptimal performance on certain tasks. However, fixing 3 resolves this issue. SPO [42] also assumes that the preference order between pairs of images remains consistent when adding the same noise, and constructs win-lose pairs for noised images, proposing step-aware preference optimization. Our work differs from SPO in that while SPO focuses improving the training method DPO, our Flow-NRG specifically targets training noise-ware reward model on noised videos.
B Details of the Derivation
B.1 Relation beween Velocity and Noise
Lemma B.1. Let Xo q be real data sample drawn from the true data distribution and X1 P be a noise sample, where Xa, Xi E Bdd Define ve wt Xo, Xi) to be the conditional velocity field specified by Rectified Flow [48], and let of (xx) be the predicted marginal velocity field. Then the 1.2 error of the noise prediction related to the 12 ervor of the velocity field prediction by
(x,, | X.red (xt, 1)12 (1-1) 1)2 (. Xo, X1) pred (2::)1?
(12)
Proof. The Rectified Flow is time-dependent flow の 10, 1] x Rd + Rd for t [0, 1]. defined by
X1) (1 (1-t) + tX
17

---

By definition, the marginal velocity field vr()))) is
(xx))) = E(u Xo, X1) | X, = Ti!
(13)
El [((i(x Xo. X1)
Xx xz
E| E[X1 -Xo X; =x+]
X1 ((1
t) Xu + t tXi)
E
x==
1-t
X1 3t
E
Xx = Dt
t
E
X1
X1 xz
It
1 1-t
Meanwhile, the conditional velocity field vx (are
No: X1) given by
X1 Ft.
(a, Xu, X1) =
(14)
1
Substituting equation 14 into equation 13, we obtain
|X E X, xx, 11
|/ X1) - ve(xx))l*=
(1- - t)2
Assuming that
(ppo XI X. 2c.] and of vx()))).
Consequently,
|| X (ax,, 0)|2 (1 1)² 1e(x X1) (xx")
B.2 Reward as Classifier Guidance
We begin by citing a lemma from the Guided Flows paper [88].
Lemma B.2. Let pt (xlyy) be a Gaussian Puth defined by a scheduler (as, (t), i.e., pr(2)) xo)
N (xlayxo, of 1) where y & is a conditioning variable, then its generating velocity field ve (x(x))) is related to the score function v log Pt yy) by
(x by)
a.æ be V logl)
(15)
where
b. (ánor - (x,rt) Tt
a:
(16)
az
Seed Appendix A.61 the Guided Flows paper [88] for detailed derivations.
If we define
oe (xF (B)
ve (x () + w b, Vr v,(1))))
vy (z, () + w|u + b, log exp(r(ex y)) (xxlx)) w (
V log exp(r(x;, g))l
a, + (o- 1o + w log exp(r(x;, y))]
0, + b, (1g
(17)
where (x:l)) p=(x:/y))-~(((p((((r,,)) ". We change our goal from sampling from the distribution pr(0, yy) to sampling from the distribution (x, y).
We note that this analysis shows that Reward Guided Flows are guaranteed to sample from 7(-12) at time = 1 if the probability path pr(-l)) is close to the marginal probability path (x) ly), but it is not clear to what extent this assumption holds in practice. This
also mens that p. (x ly) is also marginal gaussian path defined by (:(0(11)) N(<xla,1,
18

---

Simlirly, if we define
ig [ (y) - y) + wh, Vr( y)
u, y) + wb, V log exp(r(x,,,)) y))
0,2; + b, ( + log cxp(r(xx,, y))
0,3; b, V log (2:lg)
(18)
where pr (y) x (a (((xll))(expp [exp(r( 8))l We change our goal from sampling from the distribu- tion Pr (2 yy) to sampling from the distribution pe (rx y).
Reward Guidance for Rectified Flow. Rectified Flow [48] is also a Gaussian path defined by
Ft = (1 1) + lx
(19)
where æj from normal Gaussian distribution. Then
(= ao) (a: (1 1)380, 121)
where 1 - t, o t. Then we get
b
(20)
Eq. 17 becomes
1
(xxl)) v. (xx y) + wl
1t + 1-
Vr(wi, y) (x(l)))
(21)
1-1
Eq. 18 becomes
0 (a uy (x w t
Vr(w,, y)-
(22)
We Eq. 17 or Eq. 18 guide inference through reward model.
19

---

C Pseudo-code of Flow-DPO and Flow-NRG
The Pytorch-style implementation of the Flow-DPO loss (Eq. is shown below:
def loss (model, ref __mdel, I_w, x_1, c, beta):
# model: Flow model that takes prompt condition and timestep as inputs and predicts velocity
ref_model: Frozen initialization of model
x_w: Preferred Image (latents this work)
# x_1: Mon-Preferred Image (latents in this work)
# c: Conditioning (text in this work)
beta: Regularization Parameter
timestep Carch (x_4))
noise randh_raandn_lkeee (x_w)
noisy_x_M (1 timestep) x_# timestep * noise
noisy_x_1 (1 timestep)
timestep noise velocity.H_pred model (noisy_x_H, c, timestep)
velocity_l_pred model (noisy_x_1) ____,____,, c, timestep)
velocity_ref___ppee ref_model (noisy____, velocityy_re____pre ref [tf_modellooisy___.e c, timestep) velocity_x noise x_ velocity_1 noise x_1
timestep)
model_v_err (velocity.w.pre) velocity_w) norm() (2)
model_l_err (velocity.l_pree velocity_ noom () (2)
ref _f___err_ (velocity.ref.
velocity_s) .nom p2
ref ref_l_err welloityy_rf___pre velocity_l) norm () p2w
M_diff - model_v_err ref_v_err
1_diff model_l_err ref_l_err
inside_tern -0.5
(w_diff l_diff)
loss -1 (cssigooddiiniddeeee)))
return loss
The Pytorch-style implementation the reward guidance (Eq. 11) is shown below:
def reward_ (model, reward_modee model, grompt_embedd, latents, timesteps, reigh,, sgasca cfg_scale):
model: Flow model that predicts velocity given latents conditions # reward_model: Model that evaluates the quality of latents based on prompt
embeddings and timestep
prompt_embeds: Embeddings of the text prompts
latents: Initial noise
timesteps: Sequence timesteps
# reward_weight weighting coefficient of milti-dimeneional rewards
# rg. scale: scale factor for reward guidance
cfg_scale: scale factor for classifier free guidance ***
dts timesteps
timesteps
for i, t in mumerateetimesttep
v_pred model (latents, embeda, t))
cfg_scale 1.0:
v_pred__ncond model (latents, None, t) # unconditional prediction
v_pred = v_pree__uccond + cfg_scale (v_pred v_prred_umcond
latents = latents detach requuires_grad,
(rarress_ra(((Tuue (True)
reward revard_model (1atentt,,,, grompt_embede. t) reward (reward
(oward_weight) .um
reward_guidand
torch. autograd grad (reward, latents)
if t !- 1:
V_pred = v_pred rg_scale
(1 t) reward_guidance
latents latents dtsli] v_pred return latents
20

---

D Architecture of Internal Video Diffusion Model
Our work employs an internal text-to-video foundation model, which is Transformer-basee latent diffusion model [53] with 1B parameters. The model integrates a 3D VAE to encode video data into a latent space, alongside Transformer-busee video diffusion model. Each Transformer block instantiated as a sequence of 2D spatial-attention, 3D self-attention, cross-attention, and FFN modules. Importantly, the model is trained under Flow Matching framework. An illustration of the model architecture is provided in Fig 7.
Tarrsormer Block
3D Set Actertoo
Croor Aftittoon
X
(0,)(
Decader
"A racer driving notoreycle"
T5 Eacoder
Figure 7: Architecture of Our Internal Video Latent Diffusion Model Backhone.
E
Ablation Study of Our Reward Model
Ablation of VideoReward To further validate the key design choices of our reward model discussed in Sec.3.2, we conduct ablation study the evaluation benchmark, providing quantitative supplement to our analysis in Tab. 6. We compare three reward model variants: regression-style Bradley-Terry, and Brudley-Terry-We Seetee--Trrr--th----eee Witt--idi- The BT model slightly outperforms the regression-style model, likely due to the advantages pairwise annotations, which better capture ranking relationships and are more robust to annotation noise. The BTT model matches the BT model ties-excluded accuracy but significantly improves ties-included accuracy, as its explicit handling tied pairs helps it learn more robust decision boundary, capturing neutral relationships in ambiguous cases. Additionally, we find that using separate tokens for each reward attribute, instead of a shared last token further improves performance. This design better represents distinct reward aspects, enhancing alignment with human preferences.
Table 6: Ablation study on reward model type and seprate tokens. Bold: Best Performance.
GenAI-Bench Overall Accuracy w/o
VideoGen VideoGen-RewardBendd
Variants
RM Type
Soparate Tokens
Overall Accuracy
Accuracy
Accurccy
TA Accuracy
Regression
48.28
71.13
58.39
70.15
54.23
73.61
61.16
65.56
52.60
70.95
II
BT
47.74
71.21
61.22
72.58
52.33
77.10
59.43
73.50
53.06
71.62
BTT
48.27
70.89
61.50
73.39
60.52
76.31
64.64
72.40
53.55
72.12
61.26
75.66
72.20
Ablation on Noisy-Latent Reward Training. We apply reward guidance using only MQ rewards on TA-Hard prompts. When trained with noised latents, the reward model successfully guides generation, yielding an MQ win rate of 74.6. By contrast, the model trained on clean latents fails to offer meaningful guidance for intermediate noised latents, achieving only 38.6 win rate and underperforming even the unguided baseline. Following Yu et al. [81], we also directly predicting an from æ, and then backpropagating via r(wo: y). However, this method produces unusable videos without meaningful content.
21

---

Statistics of Training Data
Tab. I and Fig. 8, we provide a comprehensive statistics of our 182k-sized training dataset.
An beruur
Ecity
milding backgroum ound
Sense
seeni
Food
panda
detaii
eye v1ew ew
fan shot
2030
Ing ree
table
Camerr
filledi appear
1330
Rumen
LEPT
seem
wea 10g
r
video captures Light
water ho Wholdia ding sky
Part
face video distance m.
Nurbber
Phomet
(a) Prompt Catagory
(b) Word Cloud in Prompts
(c) Prompts Lengths
Mbess
Gens
Wiiiss
Wins
Fowarra
viku Quulity
Droamiia
lame
Powarse
1g
Percentage (50
Durot on (secenits)
(d) Vidaos Duration and Resolution
(e) Human Preference Distrbution
Figure Statistics of our training data.
Previous reward models (e.g.. VideoScore [19], LIFT [73]) were trained on datasets with mostly low resolutions (s480p) and short durations (2s), whereas ours include recent T2V models with higher resolutions (2720p) and longer clips (3-6s). The dataset statistics are shown in Tab.
Table 7: Comparison of video preference datasets in terms of spatial resolution and temporal duration.
Dataset
Resolution
Duration
Existing Preference Datasets
VideoFeedback [19]
256 256 576 x 1024
VidcoDPO [47]
320 x 512
2s
LiFT-HRA [73]
480 720
Our Datasets
Ours
384 x 672 768 x 1408
3s 6s
G
Details of Human Annotation
We provide additional details regarding the annotation process. First, annotators are provided with detailed scoring guidelines and undergo training sessions ensure they fully understand the criteria: Tab summarizes the key points for each dimension as outlined in the guidelines. Reference examples are provided to help annotators better grasp the evaluation standards. Each sample is evaluated by three independent annotators. For training and validation sets, annotators provide pairwise preference annotations and pointwise scores for Visual Quality (VQ), Motion Quality (MQ), and Tempotal RewardBench, annotators evaluate the same three aspects along
Alignment (TA). For with an additional Overall Quality, using only pairwise preferences. In cases where the annotators disagree on sample, an additional reviewer is tasked with resolving the discrepancy. The final label determined on the basis of the reviewer's evaluation, ensuring consistency across the dataset. Furthermore, during the annotation process, all annotators are instructed to flag any content deemed unsafe. Videos identified as unsafe are excluded from the dataset, ensuring the safety of the data used for training and evaluation.
22

---

Table 8: Key points summary outlined in annotation guidelines for each evaluation dimension.
Evaluation Dimension
Key Points Summary
Considering the following dimensions introducted by non-dynamic factors:
Image Reasonableness: The image should be objectively reasonable.
Clarity: The image should be clear and visually sharp.
Visual Quality
Detail Richness: The level intricacy in the generation details.
Aesthetic Creativity: The generated videos should be aesthetically pleasing.
Safety: The generated video should not contain any disturbing or uncomfort-
able content.
Considering the following dimensions in the dynamic process of the video:
Dynamic Stability: The continuity and stability between frames,
Dynamic Reasonableness: The dynamic movement should align with natural physical laws.
Motion Aesthetic Quality: The dynamic elements should be harmonious and not stiff.
Motion Quality
Naturalness of Dynamic Fusion: The edges should be clear during the dynamic process.
Motion Clarity: The motion should be easy to identify.
Dynamic Degree: The movement should be clear. avoiding still scenes.
Considering the relevance to the input text prompt description.
-Subject Relevance Relevance to the described subject characteristics and
subject details.
Dynnamic Information Relevance: Relevance to actions and postures described in the text.
Text Alignment
Environmental Relevance: Relevance of the environment to the input text.
Style Relevance: Relevance to the style descriptions, if exists.
Camera Movement Relevance: Relevance the camera descriptions, if
exists.
mansiif-uute
Made Scape
Gerd
Lasie
lurr
720x1260
T2yilurb
578x1024
Opendora
Oper
harclann
394x672
394w672
100w512
cegiidaat
255x258
256x256
Duration (seconds)
Duration (seconds)
(a) GenAI Bench
(b) VidenGen- ReswardBBench
Figure 9: Video Duration and Resolution in GenAl-Bench and VideoGen- Bench
H Details of Reward Model Evaluation
H.1 Evaluation Benchmarks
We evaluate our reward model using two benchmarks: GenAL-Bench [29] and VideoGen- RewardBench, GenAI-Bench employed to assess the accuracy of the reward model in evaluating pre-SOTA-era T2V models, while RedeoGennRewaarBBench is used to evaluate its performance on modern T2V models. In this subsection, we describe both benchmarks, highlighting key parameters and differences in Fig 9 and Tab. 9. We also visualize the model coverage across the training sets of different baselines and the two evaluation benchmarks, as shown in the Fig 10.
GenAl-Bench collects data from 6 pre-SOTA-era T2V models and 4 recent open- source T2V models. Human annotations for overall quality are obtained through GenAT-Arena, resulting in benchmark consisting of 10 T2V models, 508 prompts, and 1.9k pairs. As the videos
23

---

Table 9: Comparison between GenAT-Bench and VideoGen- ReebGenn-ewwrBBnchh Eariler Models indicates that pre-Sora-era T2V models, and Modern Models indicates that T2V models after Sora.
Prompls und Sumpled Videus
Human Preference Annotations
Benchmark
@Samples
#Prompts
#Earlier Models
#Modern Models
#Duration
#Ammotations
#Dimensions
GenAI-Bench
3784
508
(Open-Source)
3 (Open-Source)
2s 2,58
1891
3 Opeen-Sorree)
VideoGen- Benc
26457
9 (Close Source)
2022.11
Teat2Vidao--er
2023.06
Gand ZeroScope
Training
AntimaoDDIt
ModelSoopo
PrexSore-Era Models
Lavle
VisoonReward Training Set.
Hotehot-XX
2023.12
SVD
Pika
Our Training Set
VideoCrafle22
Plavense
Modam Opan Source Midelss
2024.02
OpenSora-1.0
Dreamin
2024.06
TZwTTubo
OpenSore
Models
VidkoGGer-RowweeBennhh
Kling
Dingying
Miuu
EasyAnimmate vi
CogViceoX
2024.09
Kiing
Warslang
Hallio
Luma 1.5
Figure 10: The model coverage across the training of different baselines and the two evaluation benchmarks. VideoScore, VisionReward, and GenAl-Bench primarily focus on pre-SoRA-Aee models, while our training set and MideoGen-Rewardae concentrate on oftte--f-thearrr T2V models.
in GenAI-Bench predominantly originate from earlier video generation models, they typically have lower resolutions (most around 320x512) and shorter durations (2s-2.5s). We consider GenAI-Bench benchmark assess the performance of reward models early-generation T2V models.
VideoGen-Rewenn RevardBBcn VideoGen-Eval [84] has open-sourced dataset containing videos gen- erated by 9 closed-source and open-source models, designed qualitatively visualize performance differences across models. Due to its high-quality data, broad coverage of the latest advancements in T2V models, and third-party nature, we leverage VideoGen-Eval
create fair benchmark. for evaluating reward models' performance on modern T2V models. We manually construct 26.5k video pairs and hire annotators assess each pair's Visual Quality. Motion Quality, Text A Alignment, and Overall Quality, providing preference labels. Ultimately, VideoGen- RewardBench includes 12 T2V models, 420 prompts, and 26.5k pairs. This benchmark represents human preferences for -ate-off-hheart models, with videos featuring higher resolutions (480x720- 576x 1024), longer durations (4s 68), and improved quality. use BeeeoGeenRewwadBeenne as the primary benchmark to evaluate reward models' performance modern T2V models.

---

H.2
Comparison Methods
Random To eliminate the influence of metric calculations and benchmark distributions on our evaluation results, introduce special baseline: random scores. Specifically, for each triplet (prompt, video A, video B), we randomly sample rA and rB from a standard normal distribution, denoted rA, FB (0, 1). We then calculate accuracy in the same manner as for the other models. The mathematical expectation of random scores for ties-excluded accuracy is E(ace) = 2. and the mathematical expectation of ties-included accuracy is E(ace) max(3 3. p(c= "Ties")).
VideoScore VideoScore [19] adopts 1erris--defis22-88 -BB [28] as its base model and trains with point- wise data using MSE loss to model human preference scores. Since VideoScore predicts scores across multiple dimensions, and its dimension definitions differ from those in TideoGen-RewarBeenne RewardBench, we compute both the overall accuracy and the dimension-speciff (VQ. MQ. TA) accuracy by averaging the scores of five dimensions when conduct evaluation GenAl-Bench and BeeeeoGGeeeewwdBBnnnneee consistent with the evaluation strategy outlined their paper. The training data for VideoScore predominantly comes from pre-SOTA-era models, which explains its relatively better performance on CennA--Bench while accounts for the significant performance drop on VideoGen-RewardPee RewardBench.
LiFT LiFT [73] adopts VILA-115-40B [43] as its base model and employs a VLM-as-a-judee approach. The reward model is trained through instruction tuning with inputs, preference scores along with a critic. The model generates video scores and reasons through next-token prediction. LiFT evaluates videos across three dimensions: Video Fidelity, Motion Smoothness, and Semantic Consistency, which are similar to the dimensions defined in VideoGen VideoGen-RRwardeee RewardBench. We calculate the overall accuracy using the average scores of these three dimensions and compute the dimension- specific accuracy using the corresponding dimensional scores. LiFT predicts discrete scores on a 1-3 scale, which often leads to ties in pairwise comparisons. When calculating accuracy without ties, we randomly convert the predicted tie labels to chosen/rejected with a 50% probability, indicating that the model is unable distinguish the relative quality between the two samples.
VisionReward VisionReward [76] adopts CogVLM2-Vidde 12B[23] as the base model and is trained to answer set of judgment questions about the video with a binary "yes" or "no" response using cross-entropy loss. During inference, VisionReward evaluates 64 checklist items, providing converted into 1/0 scores. The final score is computed as the weighted average of these individual responses. We use the final score to calculate both the overall accuracy and the VQ/MQ/TA accuracy. VisionReward's 's training data includes models from the pre-SOTA era models [8] as well as recent open-source T2V models [89, 79]. It performs well on GenAl-Benc and demonstrates reasonable capabilities VideoGenn-RewarBeenn
Our Reward Model We adopts QWen2-VL-2B [72] as the base model and train it with pair- wairewise data using BTT loss in Eq. 2. Scores are normalized on the validation set and averaged to obtain overall scores for evaluation and optimization. When evaluating sample videos at 2 FPS and a resolution of 448 x448, consistent with the training settings. We calculate the overall accuracy by averaging the scores across the three dimensions, and compute dimension specific accuracies using the respective scores. For GenAl-Bent FPS and resolution of 256x 256, as the minimum resolution in GenAl-Bench is 256x256. Given the significant disparities in visual quality and motion between the GenAl-Ban A--Becch videos and our training data, we utilize only the predicted TA scores to calculate the overall score.
we
we sample videos at
H.3 Evaluation Metrics
Similarly to VisionReward [76], we report two accuracy metries: ties-included accuracy [11] and ties-excluded accuracy. For ties-excluded accuracy, we exclude all data labeled as "ries" and use only data labeled as "A wins" "B wins" " for calculation. Since all competitors predict scores based on pointwise samples, we compute the rewards for each pair, convert the relative reward relationships into binary labels, and calculate classification accuracy. For ties-included accuracy, we adopt the tie calibration algorithm proposed in Algorithm 1 by Deutsch et al. [11]. This method traverses all possible tie thresholds, calculates three-class accuracy for each threshold, and selects the highest accuracy as the final metric.
25

---

I Hyperparameters
In all alignment experiments, applied LoRA to fine-tune the transformer models' linear layers, as our findings indicate that full parameter fine-tuning can degrade model's performance or potentially lead to model collapse.
Table 10: Hyperparameters for alignment algorithms
dggorith--ggnosti hyperparameters for SFT, Flow-RWR, Flow-DPO
Training strategy
LoRA [24]
LoRA alpha
128
LoRA dropout
0.0
LoRA R
64
LoRA target-modules
o.projikproojx--rrooooou
Optimizer
Adam [30]
Learning rate
5c-6
Epochs
I
Batch size
64
GPUs
NVIDIA A800@80G
Flow-DPO
8
500
Table 11: Hyperparameters for reward modeling.
VLM
Full training for vision encoder
Training strategy
LoRA for language model
LoRA alpha
128
LoRA dropout
0.0
LoRA
LoRA target-modules
Linear layers in language model
Optimizer
Adam [30]
Learning rate
2c-6
Epochs
Batch size
32
GPUs
NVIDIA AS00@80G
0 in Eq.
5.0
VDM
Training strategy
Full training
Optimizer
Adam [30]
Learning rate
5e-6
Epochs
Batch size
144
Reward Dimension
GPUs
NVIDIA AS00@80G
Additional Qualitative Results
present additional qualitative results generated by both the original model and the Flow-DPO aligned model, as shown in Fig. 11 and 12g.
26

---

eyes taliiing baskethall play graffiti urban
A person stands elone by the lake, xatching the sunset with their reflection nirrored the water.
powered knight guards anciant castle whirring and escaping it the crystal golen glowing crystal deep cavern.
A hooded figure with glowing runes reditates in moonlit stone circle, nagic swirling around their floating fome
crystal
fleating tower library.
A plague doctor with Elockwork enhancements concocts herbal remedies as green snoke swirls in redieval apothecary.
stargaze hilllop: points
through telestope.
dolphin and sea turtle explore vibeant coral reef, swisning colorful corals and narine life.
Blonde woran in blue blouse pours water frox rug a modern kritchon under ware Lighting.
Original
Flow-DPO
Figure 11: Additional visual comparison of videos generated by the original model and the Flow-DPO aligned model.
27

---

shot, boy right hapoily
Candle burns and nelts as adjacent ice cube melts into water.
Moran shoulder- Length blande hair washing rachine softly laundry
A colossal nech towers over futuristic cityscape, weapons ready, conveying epic scale and intensity.
long green swoater sways headphones, cozy living roon.
through
glowing sparkling nagical creatures.
In fog-shrouded Victorian town, trench-coated detective investigates eerie disappearances under gas lamps,
quirky navigates colorful futuristic city, dodping flying rescuing toy.
robot with LTDAR and cameras navigates cluttered warehouse, dynanically avoiding obstacles.
Blonde woran in blue blouse pours water frox rug a modern keitchon under ware lighting.
Original
Flow-DPO
Figure 12: Additional visual comparison of videos generated by the original model and the Flow-DPO aligned model.
28

---

K
Input Template for Reward Model
Full Input Template
[VIDE0] You tasked with evaluating a generuted video based three distinct criteria: Visual Quality. Motion Quality, and Text Alignment. Please provide rating from to 10 for each of the three categorics, with being the worst and 10 being the best. Each evaluation should be independent of the others.
+VVisual Quality:**
Evaluate overall visual quality of the vidco, focus static factors. The following sub-dimensions should be considered:
""Rcssonableness*** The vidco should not contain any significant biological or logical errors, such as abnormal body structures or nonsensical environmental setups.
**Clarit:** Evaluate the sharpness and visibility of the video. The image should be clear and easy interpret.
with no blurring or indistinct areas.
*eDetail Richness:** Consider the level of detail textures, materials, lighting, and other visual elements (e.g.,
hair, clothing, shadows).
*** Aesthetic and Creativity:*** Assess the artistic aspects of the video, including the color scheme, composition, atmosphere, depth field, and the overall creative appeal. The scene should convey sense harmony and balance. Safetty: The video should not contain harmful or inappropriate content, such as political, violent, or adult material. If such content is present, the image quality and satisfaction score should be the lowest possible.
Please provide the ratings of Visual Quality: <I <\ END
*Motoo Quality:**
Assess the dynamic aspects of the video, with focus dynamic factors, Consider the following dimensiioon:: **Stability:*** Evaluate the continuity and stability hetween frames. There should no sudden, unnatural jumps, and the video should maintain stable attributes (e.g.. no fluctuating colors, textures, or missing body parts).
**Naturalness:** The movement should align with physical laws and be realistic. For example, clothing should flow naturally with motion, and facial expressions should change uppropriately (e.g.. blinking, mouth movements).
*** Aesthetic Quality:** The movement should be smooth and fluid. The transitions between different motions camera angles should seamless, and the overall dynamic feel should visually pleasing.
#*Fusion:*# Ensure that elements in motion (e.g., edges of the subject, hair, clothing) blend naturally with the background, without obvious artifacts or the feeling of cut-and-paste effects.
* *Claarity of Motion:* The video should be clear and smooth in motion. Pay attention any areas where the vidco
might have blurry unstcady sections that hinder visual continuity.
** "mplitude:: s* If the video largely static or has little movement, assign a low score for motion quality.
Please provide the ratings of Motion Quality: < ([Q.rrward)>>> END
**Text Alignment:**
Assess how well the video matches the textual prompt across the following sub-dimensions:
**Subject Relevance** Evaluate how accurately the subject(s) in the vidco (c.g.. person, animal. object) align with the textual description. The subject should match description
of number, appearance, and behavior. **Motion Relevance: Evaluate dynamic actions (e.g, gestures, posture, facial expressions like talking or blinking) align with the described prompt. The motion should match the prompt in terms of type, scale, and direction.
** Eavirrnneen Relevance:** Assess whether the background and seene fit the prompt. This includes checking if real-world locations seenes accurately represented, though some stylistic adaptation is acceptable. **Style *clevance:** If the prompt specifies particular artistic or stylistic style, evaluate how well the video adheres to this style.
**Camera Movement Relevance:** Check if the camera movements (e.g., following the subject, focus shifts) are consistent with the expected behavior from the prompt.
Textual prompt [PROMPT]
Please provide the ratings of Text Alignment: <ITA reward|>
END
29

---

L Prompt Subset of TA-Hard
A rabbit and a turtle racing on a track. The rabbit sprinting ahead, while the turtle is steadily moving along. Spectators are checring from the sidelines. and a finish line is visible in the distance.
A lion and zebra playing soccer on grassy field. The lion is dribbling the ball, while the zebra is trying to block it.
The field is surrounded by trees, and other animals are watching the game.
fox and an owl stargazing logether on hilliop. The fox is lying on its back, pointing at the stars, while the owl perched on a nearby branch, looking through a telescope. The night sky is clear, with countless stars twinkling. A dolphin and a sea turtle exploring coral reef. The dolphin is swimming gracefully. while the sea turtle is gliding slowly beside it. The coral reef is vibrant with colorful corals and various marine life.
dolphin and whale singing together in the occan. The dolphin leaping out of the water, while the whale producing deep, melodic sounds, The ocean is vast and blue, with the sun setting on the horizon. A fox and a rabbit playing a duet on a piano in a forest clearing. The fox is playing the melody, while the rabbit is accompanying with harmony. The forest is alive with the sounds of nature, and other animals are gathered to listen. squirrel and chipmunk building treehouse in large oak tree. The squirrel hammering nails, while the chipmunk holding a blueprint. The tree is tall and studyy, with branches full leaves.
A robot with glowing blue eyes and a human with a cybernetic arm playing basketball in a futuristic gym. The robot is dribbling the ball with precision, while the human is preparing to block the shot. The gym is equipped with advanced technology and holographic scoreboards.
knight in shining armor and a wizard with long. flowing beard practicing archery a medicval courtyard. The knight is aiming at a target with a longbow, while the wizard is using magic to guide the arrows. The courtyard surrounded by stone walls and blooming flowers.
A talking apple with eyes and mouth, and singing banana with legs hosting talent show in vibrant theater. The apple the judge, giving feedback contestants, while the banana is the host, entertaining the audience with jokes and songs. The theater filled with colorful lights and excited spectators. pirate with a wooden leg and a mermaid with shimmering tail playing a duet on a grand piano in an underwater cave. The pirate is playing the melody, while the mermaid is accompanying with harmony. The cave is illuminated by biolumineseent sea creatures, creating magicul atmosphere.
superhero with cupe and detective with magnifying glass solving mystery bustling city. The superhero is flying above the strects. scanning for clues, while the detective is examining evidence on the ground. The city alive with activity, with skyserapers lowering overhead.
A chef with tall hat and robot with multiple arms cooking gourmet meal in ofate-of-the-art arrt kitchen. The chef chopping vegetables with precision, while the robot is sinultaneously stirring, frying, and baking. The kitchen equipped with the latest culinary technology. creating a seamless cooking experience.
painter with a beret and poct with a quill creating art a sunlit studio. The painter is working on a vibrant canvas, while the poet is writing verses inspired by the artwork. The studio is filled with natural light and creative energy, with art supplies scattered around.
A spider with a square face and green-furee pverred puppy having playful fight in whimsical garden. The spider using its web to swing around, while the puppy is playfully nipping at the spider's 's legs. The garden is filled with oversized flowers and colorful mushrooms.
A talking teapot with mustache and a dancing teacup with legs performing a tea ceremony in an enchanted forest. The teapot pouring tea, while the teacup is twirling and dancing around. The forest is magical, with glowing plants and twinkling lights.
A robot with television screen for a head and toaster with arms and legs having a cooking competition in retro kitchen. The robot is displaying recipes on its screen, while the toaster is popping out perfectly toasted bread. The kitchen is styled with vintage appliances and checkered floors.
A pair of animated scissors with eyes and mouth and roll of lape with tiny arms and legs wrapping presents in festive workshop. The scissors are cutting wrapping paper with precision, while the tape is sealing the packages with a smile. The workshop decorated with holiday lights and ornaments.
A pair of animated sncakers with eyes and a mouth and a talking basketball with a face playing a game of one-on-one on an urhan baskethall court. The sneakers dribbling and making quick moves, while the basketball is bouncing and trying score. The court surrounded by graffiti-covered walls and cheering spectators. A paper airplane with a scarf and a paper boat with captain's 's hat racing in the rain. The airplane glides through the air while the bout suils through puddles.
A basketball with a mohawk and a soccer ball with a bandana playing hopscotch in playground. The basketball bounces high while the soccer hall rolls smoothly.
mechanical knight with steam- rewwered joints standing guard an ancient castle gate. Gears whir sofily as its head turns to scan the surroundings, while steam occasionally escapes from its armor joints. A wandering alchemist with potion-filled vials clinking on their belt, gathering herbs in an enchanted forest where mushrooms glow and flowers whisper secrets.
mystcrious plague doctor with clockwork enhancements pecking through their dark robes, mixing herbal remedies a medieval apothecary shop as green smoke swirls from bubbling vials.
30

---

M Broader Impacts
Our work aims improve video generation with human feedback. While the primary goal is to improve performance, we recognize that video- generation technologies have broader societal implications. Positive applications may include enhanced creative expression and accessibility tools, thereby benefiting diverse communities. However, the same methods can be misused for creating deceptive or harmful content (e.g., deepfakes), leading to misinformation, reputational harm, and erosion of public trust.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contribution are detailed in Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. for limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Yes. We provide the detailed derivations and lemma proofs for Flow-DPO, Flow-RWR, and Reward Guidance in Sec 4 and Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail the experimental settings in Sec and evaluation settings in Ap- pendix H. We also provide the hyperparameters in Appendix I, input template of reward model in Appendix K. pseudo-coo of Flow-DPO and Flow-NRG in Appendix C. We believe our experimental results can easily reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
33

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We include the training code for our VideoReward model in the Supplementary. The preference dataset and alignment code will be released in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines futtps://nigg cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cwtppblcc/guuiiicccccccccccoocat httpc//guiiee//ooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail the experimental settings in Sec and evaluation settings in Ap- pendix H. We also provide the hyperparameters in Appendix I, input template of reward model in Appendix K, pseudo-code of Flow-DPO and Flow-NRG in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the resource limitation, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the GPU resources utilized during training Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:: htttpp:///euris.... cc
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix M.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We detail the human annotation process in Appendix G. We ask all annotators to manually filter out unsafe prompts during the dataset construction process.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, anperwwithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We include the training code for our VideoReward model the Supplementary, along with detailed installation and usage instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We hired annotators at the standard local market rate to label the data. All annotators received training and were provided with detailed annotation guidelines (see Appendix G).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study involved annotation of Al-generated videos. No personal sensitive information was collected, and the task posed minimal risk to participants. This type of annotation work does not require IRB or equivalent approval.
Guidelines:
37

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: htt////uursccc//ooaeeeeeee http://sc..ccooeeeee/ee/e/////// for what should or should not be described.
38


Reviews:
[{"review_id": "92Mafyet5z", "reviewer": "Reviewer_sgc3", "summary": "The paper provides a complete solution for improving the visual quality of flow-based video generation models through human feedback. This includes:\n* A large-scale multi-dimensional human preference dataset,\n* A video reward model trained on the dataset,\n* A reward model benchmark dataset containing 26.5K video pairs,\n* 3 algorithms for improving video generation with the help of the reward model.\nEach individual component contain their own innovations and empirical justifications:\nThe preference dataset is collected from more recent video generators to raise the standard, and includes both point-wise and pair-wise annotations.\nNext, several design decisions for reward modeling are explored, with the Bradley-Terry model with ties achieving the best performance.\nFinally, the reward model is used to improve a flow-based video generator using newly-derived Flow-DPO, Flow-RWR, and Flow-NRG formulations. Flow-DPO improves the generation over the base model both quantitatively and visually.", "questions": "* How does Flow-NRG. compare with other methods?\n* The paper mentions reward hacking issues with Flow-DPO when using time-dependent $\\beta$. Could the authors elaborate how this reward hacking manifests visually in the generated videos?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* This paper provides a complete guideline for building a full RLHF workflow for flow-based video generation models and more. It serves as a valuable resource for future research.\n* The paper includes comprehensive ablations, such as the comparison between point-wise and pair-wise rewards, as well as the scaling factors for Flow-DPO. They are good references for future works.\n* The proposed reward model achieves SOTA performance, while the video generator show promising improvement after DPO.\n* The paper is well written and logically well-structured, including just the right amount of detail for such an involved pipeline without overwhelming the reader. For example, \n\n### Weaknesses\n* The quantitative evaluation for Flow-NRG seems to be missing.", "was_revised": true, "final_justification": "I maintain my rating of accept. The paper is well-written with a lot of useful detail, and I have no major concern over the validity of the methods.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.\n\n---\n\n` [W1 & Q1] The quantitative evaluation for Flow-NRG. `\n\nTable 5 in the paper reports Flow‑NRG’s quantitative evaluation (via quantitative) on VideoGen‑Eval under custom reward weights. For convenience, the numbers are reproduced below. The three columns (VQ, MQ, TA) are the win rates (%) against the vanilla model when we steer the generator with different reward-weight triplets $w_{\\text{VQ}}:w_{\\text{MQ}}:w_{\\text{TA}}$, where $r=w_{\\text{VQ}}*r_{\\text{VQ}}+w_{\\text{MQ}}*r_{\\text{MQ}}+w_{\\text{TA}}*r_{\\text{TA}}$.\n\n$$\n\\begin{array}{c|ccc}\n\\hline\n\\text{$w_{\\text{VQ}}:w_{\\text{MQ}}:w_{\\text{TA}}$} & \\text{VQ} & \\text{MQ} & \\text{TA} \\newline\n\\hline\n0.0:0.0:1.0  & 60.56 & 46.48 & 70.42       \\newline\n0.1:0.1:0.8  & 66.50 & 63.73 & 60.86       \\newline\n0.1:0.1:0.6  & 68.94 & 67.59 & 53.28       \\newline\n0.5:0.5:0.0  & 86.43 & 93.23 & 26.65       \\newline\n\\hline\n\\end{array}\n$$\n\n\nThe results show Flow‑NRG reliably steers generation toward the weighted objectives and improves alignment. Compared with other methods in Table 1, Flow‑NRG achieves performance comparable to SFT and somewhat below Flow‑DPO, which is reasonable given that Flow‑NRG is purely inference‑time.\n\n\n---\n\n\n` [Q2] The visual manifestations of reward hacking`\n\nThank you for the suggestion! We observe that Flow-DPO with time-dependent $\\beta$ is more prone to hacking issues, **including abnormal color shifts, unnatural brightness and contrast, and reduced diversity** (e.g., the model tends to enlarge the main subject to avoid penalties from missing local details). These artifacts degrade the general quality yet still yield higher rewards. We will add qualitative examples of these hacking issues in the revision."}, {"review_id": "im3yZ369WU", "reviewer": "Reviewer_B8CS", "summary": "This paper introduces an approach to enhance video generation by integrating human feedback, aiming to overcome issues such as unsmooth motion and misalignment between video content and prompts. The authors construct a comprehensive human preference dataset, focusing on advanced video generation models and incorporating pairwise annotations across multiple dimensions. The paper presents three alignment algorithms tailored for flow-based models: Flow-DPO, Flow-RWR, and an inference-time method, Flow-NRG.\nExperimental results demonstrate that VideoReward outperforms existing reward models and that Flow-DPO achieves superior performance over Flow-RWR and conventional supervised fine-tuning methods. Additionally, Flow-NRG offers users the flexibility to assign custom weights to multiple objectives during inference.", "questions": "I will consider whether to adjust my score based on how my concerns are addressed and the comments of other reviewers.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written with clear motivation, addressing two key challenges: the quality of reward data in previous methods and their focus on non-Flow Matching paradigms.\n2. The paper conducts extensive experiments, thoroughly verifying the performance of the proposed method, leading to reliable experimental conclusions.\n3. The design of Noisy Reward Guidance to replace classifier-free guidance is intriguing.\n\nWeaknesses:\n1. In the Discussion on βt section, the paper ultimately opts for a time-independent constant β. There seems to be no significant difference between Equations 6 and 4, as they merely change the prediction target from diffusion noise to the FM velocity field. Consequently, is there a substantial distinction between Flow-DPO and Diffusion-DPO? Is it necessary to differentiate them? Similarly, for Flow-RWR, Equations 9 and 8 present similar issues, raising questions about the necessity of distinguishing between Flow-RWR and diffusion-based RWR.\n2. The novelty of the paper is relatively limited. Although the introduced Noisy Reward Guidance is interesting, the core design concepts, such as Flow-DPO, do not seem to represent significant improvement or innovation over previous work.\n3. I am curious about the performance difference of the proposed method when using the original classifier-free guidance instead of Noisy Reward Guidance.\n4. According to the quantitative results on VBench, there does not appear to be a significant improvement in the total metric compared to the pre-trained model.\n\nMinor Weaknesses:\n1. The Efficient Reward on Noisy Latents section lacks detailed information, which can cause confusion. For instance, the statement \"We then adopt the Bradley–Terry loss to learn the reward function from these noised videos\" is unclear whether the loss training is based on noisy latents or pixel space noisy videos.", "was_revised": true, "final_justification": "The rebuttal solved most of my concerns. However, the technical novelty and theoretical explainability of the work remain unresolved. Therefore, my final score is borderline reject.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.\n\n---\n\n` [W1] Is it necessary to differentiate between Diffusion-DPO and Flow-DPO?` \n\n\nThanks for asking this clarification question! A straightforward way to extend Diffusion-DPO to flow matching is indeed to replace the diffusion's noise prediction term with the FM's velocity prediction term. However, **this heuristic lacks theoretical grounding, so whether it works is unpredictable**.\n\nWe first utilize the connection between diffusion and flow matching and derive a vanilla Flow-DPO objective that naturally carries a timestep-dependent $\\beta_t$. Our analysis shows that this form induces a timestep-dependent KL constraint, which biases alignment towards high-noise (early) steps and triggers noticeable reward hacking on certain dimensions. Motivated by the DDPM simplification from Eq.12 to Eq.14, we then test a variant that replaces $\\beta_t$ with a time-independent constant $\\beta$; this version performs better overall and we term it as \"Flow-DPO\".\n\nThus, although Flow‑DPO looks similar to Diffusion‑DPO in form, **we provide the theoretical rationale and empirical evidence for why the constant‑$\\beta$ variant is preferable to the timestep‑dependent one**. Throughout the paper, we stress that **Flow‑DPO is an extension of Diffusion‑DPO to flow‑matching models**. Given that leading image generation models (e.g., SD3 [1], FLUX [2]) and video generation models (e.g., Sora [3], Wan2.1 [4], Seedance [5]) now adopt flow matching in place of traditional diffusion, we believe this distinction is necessary to keep the theory reasonable and making the framework more intuitive for the community.\n\n[1]. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.  \n[2]. FLUX.  \n[3]. Sora: Video generation models as world simulators.  \n[4]. Wan: Open and Advanced Large-Scale Video Generative Models.  \n[5]. Seedance 1.0: Exploring the Boundaries of Video Generation Models.  \n\n---\n\n\n` [W2] The novelty of the paper is relatively limited.`\n\nWe respectfully disagree with the assessment that the novelty of our work is limited. While it is true that some components were built upon existing paradigms in other fields, we emphasize that human preference alignment in video generation remains in its early stage, with several initial challenges yet to be adequately addressed. These include: (1) the mismatch between existing preference datasets and modern T2V models; (2) the underutilized potential of VLM-based reward modeling; and (3) the limited applicability of alignment algorithms like DPO. **Our work aims to propose a comprehensive alignment pipeline tailored to flow-based T2V models and validate its effectiveness in improving video generation**.\n\nImportantly, our contributions go beyond straightforward adaptations. For example, we show that directly extending Diffusion-DPO to flow-based models by leveraging the connection between diffusion and flow matching leads to reward hacking and suboptimal performance. Our Flow-DPO addresses this with a constant KL constraint, resulting in more stable and effective alignment. While Flow-DPO is similar to Diffusion-DPO in form, **we provide both theoretical rationale and empirical evidence to support its effectiveness.** Likewise, our proposed Flow-NRG offers a novel inference-time method for distribution alignment without additional training. **Beyond the alignment algorithms, our work also contributes insights into dataset construction and reward model design**, offering a comprehensive pipeline to preference alignment for video generation.\n\n\n---\n\n` [W3] The performance difference between CFG and Flow-NRG. `\n\nThank you for your question, and we apologize for the confusion. Noisy Reward Guidance and classifier-free guidance (CFG) are orthogonal techniques. In all our experiments, the model first uses CFG to obtain an initial velocity, followed by Noisy Reward Guidance to refine it.\n\nAs shown in Table 5, we directly compare the full method (CFG + Noisy Reward Guidance) with the CFG-only baseline. The reported win rates (%) reflect the performance gains brought by Noisy Reward Guidance on top of CFG. We will clarify this point more explicitly in the revision to avoid further confusion.\n  \n\n---\n\n` [W4] VBench results show no significant overall improvement over the pre-trained model. `\n\n\nWhile the VBench score shows a modest increase from 83.19 to 83.41 after applying DPO (Table 3), this improvement is more significant than it appears. We argue that in generative AI, relying solely on automatic metrics to assess model capability remains challenging and can mask perceptual gains. **Beyond VBench, our user study (Fig. 5) shows 44.0% wins for the DPO‑tuned model versus 27.0% for the pretrained baseline (29.0% ties).** These consistent advantages indicate that DPO significantly improves visual quality and alignment despite the modest increase in the VBench score. We encourage the reviewer to view the video examples in the supplementary materials for a visual comparison of the improved generation quality.\n\n---\n\n` [Q1] Clarify reward training on noisy latents or pixel space`\n\nWe apologize for the confusion caused by the lack of detail. For the \"Efficient Reward on Noisy Latents\" section, the loss is computed on noisy latents. As noted in line 208, we train a time-dependent reward model $r_\\theta(·, t)$ directly in **latent space** to avoid the costly backpropagation through the full VAE decoder required for computing ∇r in pixel space."}, {"review_id": "n0e6HBKtwO", "reviewer": "Reviewer_Y3iN", "summary": "This work introduces VideoReward, a three-dimensional reward model for video generation. To evaluate its effectiveness, the authors further propose three strategies: two training-time strategies (Flow-DPO and Flow-RWR) and one test-time strategy (Flow-NRG), all tailored for flow-based video generation models. Experimental results demonstrate the effectiveness of VideoReward and the proposed strategies on their internal VDM.", "questions": "Please refer to the weaknesses for detailed concerns.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strenth:**\n1. Proposes three novel optimization strategies for flow-based models, covering both training and test-time stages.\n2. Contributions in building a preference dataset, reward model, and reward benchmark, showcasing significant workload and effort.\n\n\n**Weaknesses:**\n1. **Unclear Motivation:** The motivation for introducing three strategies for flow-based models within a reward model is not clearly articulated. It would be helpful to clarify why these strategies are necessary.\n2. **Limited Preference Aspects:** The human annotations only consider three preference aspects (VQ, MQ, and TA), which seem insufficient for evaluating video generation. Additional aspects, such as temporal coherence, should be considered, or the authors could provide further justification for focusing solely on these three aspects.\n3. **Triplets Construction:** According to Line #99, preference video pairs are selected from distinct VDMs. However, due to the inherent quality differences between VDMs, selecting video pairs from the same VDM (with different random seeds) might result in more comparable positive and negative videos, enabling finer-grained preference modeling.\n4. **Video Alignment Evaluation:** The evaluation is conducted exclusively on the authors’ internal pretrained model, which is insufficient to validate the generalizability of both the reward model and the proposed optimization strategies. Broader evaluations on more base models are necessary.\n5. **Reference or Experimental Support:** Some claims lack adequate references or experimental support. For instance, Line #27 regarding previous reward models and Line #251 regarding the effects of full fine-tuning vs. LoRA fine-tuning require more evidence or citations.", "was_revised": true, "final_justification": "The authors' response has successfully addressed my concerns. Thus, I raise my rating accordingly.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.\n\n\n---\n\n` [W1] Clarify the motivation for introducing three strategies for flow-based models within a reward model.`\n\nThank you for your insightful comments. The three alignment algorithms are not isolated; they **all stem from a unified reinforcement learning framework** that aims to maximize reward with KL regularization. In other words, these strategies are three different implementations of the same optimization objective.\n\nThese strategies were introduced to address the different practical needs in applying human preference alignment to flow-based models. Specifically, Flow-DPO and Flow-RWR are designed for training time, with our results showing Flow-DPO to be more effective. Flow-NRG, on the other hand, is designed for inference time scenarios where retraining is impractical.\n\n---\n\n` [W2] Limited Preference Aspects: Additional aspects, such as temporal coherence, should be considered, or the authors could provide further justification for focusing solely on these three aspects. `\n\n\nIn our paper, we primarily focus on three aspects: Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA). Section F provides a detailed explanation of the specific factors considered under each aspect. **Regarding temporal coherence, we treat it as a component of Motion Quality** (see Table 7). This categorization is inspired by the design of image reward models like ImageReward [1], which typically consider VQ and TA. Building on this, we introduce MQ to account for the temporal dimension of video preferences. We believe this three-aspect formulation is reasonable, and we agree that adopting a more fine-grained decomposition could be a valuable direction for future work.\n\n[1]. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS 2023.\n\n\n---\n\n` [W3] Selecting video pairs from the same VDM (with different random seeds) might result in more comparable positive and negative videos, enabling finer-grained preference modeling `\n\nThank you for the valuable suggestion. We agree that sampling pairs from the same VDM could yield finer-grained preferences. In our current setup, although the pairs are drawn from different VDMs, the models are comparable in capability, with similar mean MOS scores (see the table below). Here, MOS refers to the mean human opinion score, where annotators rate individual videos on a scale from 1 to 5. So this choice does not, in practice, prevent effective preference modeling. We acknowledge that intra‑model pairing is a promising direction and will explore it in future work.\n\n\n$$\n\\begin{array}{l|ccc}\n\\hline\n\\textbf{Model} & \\textbf{MOS VQ} & \\textbf{MOS MQ} & \\textbf{MOS TA} \\newline\n\\hline\n\\text{Dreamina}  & 2.45 & 2.04 & 3.49 \\newline\n\\text{Luma}      & 2.23 & 2.11 & 3.45 \\newline\n\\text{Gen3}      & 2.42 & 2.16 & 3.26 \\newline\n\\text{Kling 1.0} & 2.34 & 2.20 & 3.53 \\newline\n\\text{Kling 1.5} & 2.37 & 2.22 & 3.54 \\newline\n\\hline\n\\end{array}\n$$\n\n\n---\n\n` [W4] Broader evaluations on more base models are necessary. `\n\nAs shown in Section C and Figure 7, our base model follows a standard VDM recipe—a 3D VAE coupled with a Transformer-based latent diffusion model. Most contemporary T2V models (e.g., Sora [2], Hunyuan Video [3], Wan2.1 [4]) adopt the same formulation. Additionally, our method is not tied to any architecture‑specific components. We believe that the results on this architecture are representative, and our approach can be applied to other T2V models without modification.\n\n\n[2]. Sora: Video generation models as world simulators.  \n[3]. Hunyuanvideo: A systematic framework for large video generative models.  \n[4]. Wan: Open and Advanced Large-Scale Video Generative Models.  \n\n---\n\n` [W5-1] Reference or Experimental Support: Line #27 regarding previous reward models. `\n\nPrevious reward models (e.g., VideoScore [5], LIFT [7]) were trained on datasets consisting primarily of low resolutions (often $\\le$ 480p) and short durations (2s), whereas ours cover recent T2V models (most $\\ge$ 720p) with higher resolutions and longer clips (3s -6s). The statistics are listed below. We will add these statistics in the revision.\n\n$$\n\\begin{array}{lcc}\n\\hline\n\\textbf{Dataset} & \\textbf{Resolution} & \\textbf{Duration} \\newline\n\\hline\n\\textit{Existing Preference Datasets} & & \\newline\n\\text{VideoFeedback [5]} & 256\\times256\\ - \\ 576\\times1024 & \\text{1s}\\ -\\ \\text{3s}\\newline\n\\text{VideoDPO [6]}      & 320\\times512                          & \\text{2s} \\newline\n\\text{LiFT-HRA [7]}      & 480\\times720                          & \\text{6s} \\newline\n\\hline\n\\textit{Our Datasets} & & \\newline\n\\text{Ours}             & 384\\times672\\ - \\ 768\\times1408 & \\text{3s}\\ -\\ \\text{6s} \\newline\n\\hline\n\\end{array}\n$$\n\nFurthermore, as shown in Table 2, while prior reward models perform well on GenAI-Bench, they perform significantly worse on VideoGen-RewardBench compared to ours. GenAI-Bench videos are shorter (2–2.5 seconds) and lower in quality, while VideoGen-RewardBench features longer clips (4–6 seconds) and higher resolution generated by modern video models.\n\n[5]. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. EMNLP 2024.  \n[6]. Videodpo: Omni-preference alignment for video diffusion generation. CVPR 2025.  \n[7]. LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment. arXiv 2024.  \n\n---\n\n` [W5-2] Reference or Experimental Support: Line #251 regarding the effects of full fine-tuning vs. LoRA fine-tuning `\n\nThank you for your helpful suggestion. Regarding line #251 (\"full-parameter updates were observed to hurt quality or even collapse the model\"), our preliminary ablation shows that full-parameter fine-tuning more often degrades overall quality and is more susceptible to reward hacking issues than LoRA-based tuning. Recent works like SD3 [8] also favor LoRA over full fine-tuning in their post-training stage. We will add the supporting evidence in the revision.\n\n\n[8]. Scaling rectified flow transformers for high-resolution image synthesis. ICML 2024"}, {"review_id": "2cDwOgRn0n", "reviewer": "Reviewer_zALS", "summary": "This paper introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices\nimpact its rewarding efficacy. It brings three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods.", "questions": "Regarding the reward model methodology, do you plan to open-source your dataset and pre-trained reward model? (This won’t affect my evaluation of the paper—it’s merely a suggestion.) For methods like DPO or GRPO, the quality of the constructed dataset often has a more significant impact than the algorithmic details themselves. Open-sourcing the code and data could greatly benefit future research and applications by enabling others to build upon your work.\n\nCurrently, training solely on synthetic data introduces the limitations I mentioned earlier for the reward model. However, using real data (e.g., actual video clips) directly often leads to failures in DPO/GRPO training. Have you explored solutions to this issue? Specifically, have you attempted training with real data, and if so, does your method still perform effectively?", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n- Constructed a relatively comprehensive dataset (as listed in Table 1).\n- Proposed three effective algorithms specifically designed to enhance video generation quality for flow-matching methods.\n- Provided extensive experimental results, effectively demonstrating both the limitations and standout performance of the new reward model.\n\nWeaknesses:\n\nThe paper relies solely on model-generated data to construct the training set, which may be problematic. Based on our experience, even state-of-the-art models (e.g., Keling, Veo3, etc.) often fail to produce video clips of fully satisfactory quality, even after multiple sampling attempts. Consequently, we believe this approach makes it difficult for the trained reward model to surpass the performance ceiling of existing models.", "was_revised": true, "final_justification": "Thank the author for the response. After considering the opinions of other reviewers, I have decided to maintain my score.", "rebuttal": "We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.\n\n---\n\n` [W1]: Limitations of using model-generated data for reward model training. `\n\n\nWe appreciate the reviewer’s observation and agree that current SOTA video generation models still fall short of producing video clips that match the fidelity of real-world videos. However, we would like to clarify the reasoning behind our choice of using model-generated data for training the reward model:\n\n1. **The goal of the reward model is to capture human preferences among outputs with similar generation quality, rather than to distinguish between real and generated videos.** As the reviewer rightly pointed out, given the current gap between real and generated videos, assessing the quality difference between real and generated videos is often a trivial task. This distinction may not provide meaningful learning signals for the fine-grained preference modeling required in our setting. Constructing such incomparable pairs might hinder the model's ability to learn subtle preference differences.\n2. **Our reward model is primarily used to guide post-training algorithms such as Flow-DPO or other RL-based methods.** In this context, the reward model needs to capture preference distributions among outputs that are within the capability range of the base generation model, as these are the samples it will be asked to evaluate during alignment. For this reason, we intentionally construct the reward model training set using generations from recent, high-quality T2V models. This ensures the reward model can provide relatively accurate reward signals during the alignment phase.\n\n\n---\n\n` [Q1] Do you plan to open-source your dataset and pre-trained reward model? `\n\n\nYes. We will open‑source our reward model, including the model weights as well as the code for training and evaluation. The dataset is subject to a stricter internal review; we are actively working through that process and will update it once approval is obtained.\n\n\n---\n\n` [Q2] Challenges of Real-Data Training in DPO/GRPO Settings? ` \n\nYes, we have explored the use of real videos in DPO training, for example, by using real videos as the *chosen* samples and generated videos as the *rejected* ones. However, this setup often leads to undesirable effects, such as over-sharpened outputs and the emergence of grid-like patterns in generated frames. We attribute this to the significant distributional gap between real and generated videos produced by current T2V models. Such a mismatch may result in suboptimal alignment outcomes. \n\nTo mitigate this, possible solutions include (1) incorporating an auxiliary supervised fine-tuning (sft) loss to stabilize training and (2) increasing the $\\beta$ coefficient in the DPO loss to enforce stronger KL regularization. Additionally, if the primary failure is degraded visual quality (assuming DPO focuses on aligning other aspects such as motion naturalness), we have verified that a short sft stage (around 50 steps) on high-quality real videos after DPO training can effectively restore visual fidelity. In summary, **if DPO leads to noticeable degradation in visual quality, a subsequent SFT stage on high-quality data can recover fidelity with very few steps, while still preserving the alignment benefits of DPO**."}]

Author rebuttal:

Meta-review: This paper proposed a complete solution of improving video generation with human feedback, including a large-scale multi-dimensional human preference dataset, a video reward model and video reward benchmark data set, and three methods (Flow-DPO, Flow-RWR, and Flow-NRG) to improve video generation model.

Advantages pointed out by reviewers are 1) multiple contribution of a full RLHF workflow including preference data set, video reward model, 3 methods to enhance video generation  2)comprehensive experiments with ablation studies, showing the promising results for the proposed reward model 3) Some novel design like noisy reward guidance to replace classifier-free guidance.

After rebuttal, the major concern remained is methodological novelty, i.e., Flow-DPO and Flow-RWR are direct adaptations of Diffusion-DPO and RWR to flow matching. However, consider the contributions of other aspects and the advantages, I will recommend "Accept (poster)".
Decision: Accept (poster)