DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models
Authors: Ziyi Wu
Anil Kag
Ivan Skorokhodov
Willi Menapace
Ashkan Mirzaei
Igor Gilitschenski
Sergey Tulyakov
Aliaksandr Siarohin
Venue: NeurIPS 2025 spotlight ()
Keywords: video generation
diffusion model
direct preference optimization
Primary area: applications
TL;DR: We propose an improved DPO method tailored towards video diffusion models

Abstract: Direct Preference Optimization (DPO) has recently been applied as a post‑training technique for text-to-video diffusion models.
To obtain training data, annotators are asked to provide preferences between two videos generated from independent noise.
However, this approach prohibits fine-grained comparisons, and we point out that it biases the annotators towards low-motion clips as they often contain fewer visual artifacts.
In this work, we introduce DenseDPO, a method that addresses these shortcomings by making three contributions.
First, we create each video pair for DPO by denoising corrupted copies of a ground truth video.
This results in aligned pairs with similar motion structures while differing in local details, effectively neutralizing the motion bias.
Second, we leverage the resulting temporal alignment to label preferences on short segments rather than entire clips, yielding a denser and more precise learning signal.
With only one‑third of the labeled data, DenseDPO greatly improves motion generation over vanilla DPO, while matching it in text alignment, visual quality, and temporal consistency.
Finally, we show that DenseDPO unlocks automatic preference annotation using off-the-shelf Vision Language Models (VLMs): GPT accurately predicts segment-level preferences similar to task-specifically fine-tuned video reward models, and DenseDPO trained on these labels achieves performance close to using human labels.

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models
Ziyi Wu Wu 12,33 Anil Kag', Ivan Skorokhodov', Willi Menapace's Ashkan Mirzaei', Igor Gilitschenski2..* Sergey Tulyakov Aliaksandr Siarohin
Snap Research, Universiiyy of Toronto, VVector Institute
Abstract
Direct Preference Optimization (DPO) has recently been applied a post-training technique for text-to-video diffusion models. To obtain training data, annotators are asked to provide preferences between videos generated from independent noise. However, this approach prohibits fine- sne-ggained comparisons, and we point out that it biases the annotators towards low-motion clips they often contain fewer visual artifacts. In this work, we introduce DenseDPO, a method that addresses these shortcomings by making three contributions. First, we create each video pair for DPO by denoising corrupted copies of a ground truth video. This results in aligned pairs with similar motion structures while differing local details, effectively neutralizing the motion bias. Second, we leverage the resulting temporal alignment to label preferences on short segments rather than entire clips, yielding a denser and more precise learning signal. With only one-third of the labeled data, DenseDPO greatly improves motion generation over vanilla DPO, while matching it in text alignment, visual quality, and temporal consistency. Finally, we show that DenseDPO unlocks automatic preference annotation using off-the-shelf Vision Language Models (VLMs): GPT accurately predicts segment-level preferences similar to task-specifically fine-tuned video reward models, and DenseDPO trained these labels achieves performance close to using human labels. Additional serrch..ittuun github. ioo DDDDsseDDOO
results are available at https
Introduction
Recent advances in diffusion models [23] have enabled high-qualin eext-guided video generation [2, 24, 35, 58, 66, 71]. Despite tremendous progress, existing video generators still fall short on temporal coherence, visual fidelity, and prompt alignment [89], impeding their industry-level applications.
Inspired by the success of learning from human feedback in language models [3, 52] and image diffusion [5, 17, 70], recent works have explored preference alignment in video diffusion [40, 60, 87]. Among them, methods based on Direct Preference Optimization (DPO) [61] stand out as they bypass the need for an explicit reward model [10, 47, 50, 65]. However, existing DPO methods for video diffusion are largely adapted from their image-based counterparts, without addressing the unique challenges inherent to video generation. Typically, these methods first generate videos from independent noise maps, followed by human preference labeling to construct comparison pairs. Yet, human preferences in video influenced by multiple, sometimes inversely correlated, factors, such as the visual quality (i.e., pixel-level fidelity) and the dynamic degree (i.c., strength global motion). Indeed, current video generation models excel at producing high-quality slow-motion videos, while struggling to synthesize more challenging dynamic scenes [8]. As a result, when annotators are asked express preferences, they often favor artifact-free slow-motion clips. Applying DPO training such preference data further reinforces video generators' bias toward slow-motion content, ultimately suppressing the model's ability generate dynamic and motion-rich videos.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"A young aduit male doing handstand on the beach"
"A woman doing push-up exercise beach sunsef"
"A monkey performs jump skateboard at the skate park, landing smoothly"
*A panda breakdancing in neon-lit urban alley'
Figure I: Text-to-video results with our DenseDPO aligned model. Our method improves both visual quality and temporal consistency of the model, enabling generation of challenging motion.
A natural approach to enhance competing factors, drawing inspiration from Pareto optimization, to fix some attributes within each video pair while varying others. Motivated by the guided image synthesis approach in SDEdit [51], generate a pair by introducing different partial noise to ground-truth video and perform denoising. The resulting videos each pair share high-level semantics and motion trajectories while differing in local visual details [8, 72], allowing to reduce spurious correlations. However, guided sampling inherently reduces the diversity across generated video pairs, leading degraded DPO performance [53, 63). A straightforward solution might be to annotate more data pairs. Instead, we propose extracting richer and more accurate supervision from each video pair by collecting segment-level preference labels.
Prior works show that multi-dimensional scores are superior to a single label in preference align- ment [42, 80, 93]. Unlike images, videos have a unique temporal dimension [78]. In practice, observe that human preferences over video pairs often vary across time, as artifacts may appear at different timestamps in each video, leading inconsistent preferences. This issue is more severe with modern video generators as they produce longer videos. Therefore, we split videos into short segments (e.g., 1s slices), and collect per-segment preference labels. Thanks to temporally aligned videos from guided sampling, there is clear one-to-one correspondence between segment pairs, sim- plifying the annotation process. Segment feedback also reduces the amount of ties when both videos contain artifacts, and provides more accurate supervision. In addition, it allows us to apply existing vision- languaee models (VLMs) [1, 4] which can produce reliable judgment on short segments.
Our main contributions are threefold: (i) A DenseDPO framework tailored towards video generation, with improved data construction and preference granularity over vanilla DPO; (ii) DenseDPO retains the motion strength of the base model while matching other metrics vanilla DPO, with significantly higher data efficiency; (iii) We show that existing VLMs fail label preference over long videos (e.g., 5s), but they perform well in segment-level preference, achieving results close to human labels.
Related Work
Preference learning for image diffusion. Inspired by the success of human feedback learning in language modeling [3, 52], similar approaches have been adapted to image generation. One major line of work focuses on training reward models from human preference labels [33, 75, 76, 79, 93], these models can be used as loss functions to optimize generators by direct gradient backpropagation [9, 13, 31, 59, 77, 79] or policy gradients [5. 17]. Another line work utilizes predicted rewards on training data to re- re-weight the diffusion loss [38], or trains only on high-scoring samples [14, 15, 39]. However, all these methods require an explicitly trained reward model, and may suffer from the reward hacking issue [13, 79]. In contrast, Diffusion-DDP [70] and D3PO [83] directly optimize the model on

---

pre-collected human preference pairs, bypassing the need for online reward feedback. Building on this paradigm, subsequent works have explored improving comparison data pairs [28. 32, 88. 90], better DPO objectives [29, 41, 73], and credit assignment over denoising timesteps [43, 94].
Preference learning for video diffusion. Early approaches preference learning in video diffusion directly borrow techniques from image diffusion, such as direct reward optimization [40, 56, 60, 68, 87] and training loss re-weighting [19]. However, they often rely on image reward models [33, 75] to provide supervision. Recent papers thus focus on developing better video reward models [46, 47, 80]. One strategy aggregates multiple video quality assessment metrics [21, 30] a final score [47, 91]. However, existing metrics are only effective for short videos [46, 80), limiting their applicability for modern video generators that produce long videos [7. 71, 84]. To address this limitation, LiFT [74], VesionReward [80]. and VideoAAlign [46] collect large number of videos from advanced video generators, label human feedback, and fine-tune VL.Ms to predict preferences. With a powerful video reward model, they apply weighted training [74] or DPO [46, 80] to improve video generation. In contrast to prior works, we focus on DPO for video diffusion using direct human annotations, i.e. without an explicit reward model. Analogous to the verbosity bias observed in language model preference learning, where annotators favor longer outputs [64, 67], we identify a motion bias in video preference labels, where slow-motion videos are often preferred. To mitigate this, we propose better data pair construction strategy to address this bias via guided video generation.
Rich feedback for alignment. While early preference alignment methods treated human feedback as single binary label, recent works begin to exploit rich, multi-dimensional feedback [42, 80, 92]. In image generation, MPS [93] learns reward model that evaluates images on four dimensions including aesthetics, semantics, detail, and overall quality, improving its alignment with humans. On the other hand, Liang et al. [42] curates dataset that localizes regions of artifacts and misaligned words in the text prompt, leading to better DPO performance. Multi-aspect feedback even more critical for video generation due to its inherently higher dimensionalityy Recent works all explicitly model dimensions such as visual fidelity, text relevance, and motion consistency [46, 47, 80J. However, a notable limitation is that they still aggregate feedback at the whole-video level, neglecting the fine- grained temporal dimension of preferences. In contrast, our DenseDPO partitions videos into short, temporally aligned segments, and collects preferences for each segment. This is conceptually similar to the sentence- preference label used in language models [34, 86]. By localizing feedback to brief windows, obtain more accurate and denser supervision signals for DPO training.
3 Method
We build upon diffusion models and the standard Direct Preference Optimization (DPO) framework, which we refer to as VanillaDPO (Sec. 3.1). We discuss the motion bias inherent in using this naive approach for video generation and introduce StructuralDPO, a method that optimizes human preferences on structurally similar video pairs (Sec. 3.2). To address the reduction of diversity induced from using structurally similar videos, we propose DenseDPO, which enables fine- men-grained human preference alignment along the temporal axis of videos (Sec. 3.3).
3.1 Background: Video Diffusion and DPO
Rectified-flow diffusion models. Let æ E HxW denote a video sample of length T with spatial dimensions H x W. We follow the rectified flow framework [44, 48), which learns transport map from the standard normal distribution e ~ No I) to the distribution of real videos x ~ Pdata with a denoiser. The forward diffusion process produces a noisy input x; at time t E [0. 1] via linear interpolation with noise e: 2, = (1 - t)zo + te. The denoiser Go t, c), implemented as a neural network parameterized by 0, trained to reverse this process with the following objective:
min Fx (o...)) - ..) G, 0
where p(t) is the distribution of noise levels (following [16], we adopt the logit-norma one) and refers to the auxiliary conditioning variable such as text embeddings.
Van.llaPP.. In the direct preference optimization framework [61, 70], generative model is trained to align its outputs with human preferences. Typically, these preferences are defined by a dataset D [(c, æh, xt, (I)}, where each sample consists of two videos {z", x per input condition and
3

---

[a] Vanilla DPO
Seed
VDM
Prompt: "a woman doing exercise on blue mat"
VDM
Seed
(b) Dense
Seed
VDM
malviseo
Prompt: doing
Seed
exercise on blue mat""
Figure 2: Comparison between VanillaDPO (top) and DenseDPO (bottom). VanillaDPO compares two videos generated from independent random noises and only assigns single binary preference, biasing the annotators toward flo-mootion videos. In contrast, DenseDPO generates structurally similar videos from partially noised real videos, and label segment-level dense preferences.
their preference label / +11 The preference function is defined as:
+1,
x > x (i.e., x preferred over x x > x (i.e., x² is preferred over æ²)
1(x x²1)
(2)
1,
The Bradley-Terry (BT) model [6] defines pairwise preference using reward function r(x, c) which computes the alignment score between the sample æ and the input condition c. The corresponding probabilistic preference can be expressed as:
PBr( > x [c) = (r((=", c) r(x', cc); Per(x' > x²|c) = a(r(x' [))(=x.cc-ac - r000 c)),
.c
(3)
where (.) is the sigmoid function.
Rafailov et al. [61] defines the binary preference optimization explicitly optimizing the binary reward objective log a (I(=", x²) * (r(=",c) .) r(x' c))) in conjunction with a Kullback-Leibler
(KL) divergence regularization to control the deviation from reference model. Wallace et al. [70] re-formulated the preference optimization framework for diffusion models assuming the presence of reference model Gres, which is further extended to rectified flow models in [46]. Given sample (e, x:, a²,1). ,I), the denoiser Go. and the reference model Gref, we can define an implicut reward as:
s(x c,t, ||(c" xxx) - Go c) ||(c" - -x)c=~=>)>> - G, t, c)ll2
(4)
where at = (1 t)x=" + tE", " No, I) is a noisy latent for input a= (either a: or xx time t. With the implicit reward function, the VanillaDPO objective is defined as follows:
( = - E
(e,0" a DDDD DNwD, [log a (-B * I(x xx)) * (s(x", c,t, (s(x,,c,tt, eN1(01)
s(x' c,,, c,,,,))))] (5)
3.2 StructuralDPO: Preference Learning over Structurally Similar Videos
Motion bias in VanillaDPO. In the standard VanillaDPO pipeline, preference pairs are created by independently sampling two videos (xx x!) from different noise seeds under the same conditioning (see Algo. 1), followed by human preference annotation. While this approach works reasonably well for images, its direct extension to videos introduces new issues due to the presence of the new temporal dimension. Independent noises often result in videos with significantly different motion patterns and global layouts (see Fig. 2 (a)). For example, in typical preference pair, one video may be nearly static but visually clean, while the other contains the desired motion but also introduces artifacts, such distorted limbs or flickering.

---

No Guidance
n=0.85
n=0.75
n=0.65
GT video
Figure 3: Guided video generation with different 1j. Lower y means more guidance. We sample one frame per video for visualization. 7 = 0.75 is enough to maintain the motion trajectory and high-level semantics of the ground-truth video. For slow-motion videos (top), a high suffices to generate artifact-free videos, while videos with challenging motion (bottom) require more guidance.
We empirically observe that this is a common bias in generated video data. Indeed, video models often excel at producing high-quality slow-motion clips, while dynamic videos usually contain visible artifacts [8, 89]. Since humans typically perceive clean static videos as more realistic than artifact- prone dynamic ones, this often leads to preference dataset that systematically over-represents static content. Consequently, a model trained with DPO on this dataset would produce videos with reduced motion. In our preliminary DPO experiments, using our base model (see Tab. and Tab. 2). The same issue has also been observed in prior works, e.g., Tab. 10 of [80] using Cog VideoX [84] and Tab. of [47] using VideoCrafter-v2 [11].
observed substantial drop in dynamic degree
StructuralDPO. To address the static bias in VanillaDPO, we propose StructuralDPO, which modifies the data curation strategy using guided generation [51] to obtain pairs of videos with similar motion trajectories. Specifically, in VanillaDPO, we start from independent noise, i.e., æ&y NO0, I); ø'y ~ (N( I) and then denoise from step N till step 1. Instead, propose to de- noise from a partially noised real video to generate video pairs. Concretely, given a ground truth video and the guidance levely) 17 E [0, 1], we obtain the corrupted noisy video at step round(1) * N) as:
x = (1 (1-n) n) + n xt = (1 - n)æ + ret; where e ~ 1).,
(6)
The corrupted videos are then denoised from step n to 1, as outlined in Algo. 2. Here, 7 governs the structural similarity between the two samples. Since early diffusion steps control the global motion [8], this approach preserves the overall dynamics while allowing variations in local details. We compare videos of varying motion strengths generated with different vy in Fig. 3.
StructuralDPO applies the standard DPO formulation (Eq, (5)) on this structurally consistent dataset, which helps to focus the preference on the temporal artifacts and visual inconsistencies, anchoring other dimensions like dynamic degree. Additionally, guided sampling simplifies the generation task, allowing the model to produce artifact-fre free highly dynamic videos sampling reduces data construction costs as it requires fewer sampling steps.
reliably. Finally, guided
Algorithm 1 Vanilla Paired Video Generation
Algorithm Guided Paired Video Generation
Input: Denoiser Ge. Input Condition c, Infer-
Input: Denoiser Ge. Input Condition e, Inference
ence Steps N
Steps N. Real Video æ. Guidance Level 17 E [0,1]
Init: A =
Init: A, N. n = round(1 * N)
Init: æb ~ (0,
Init: x. = =1 - 1) + ne c. ~ ~N(0, I)
Init: x
I)
Init: x, = (1-n)æ + nc! c ~ ~N(O
for N to 1 do
for i = n to do
1=
1 N
< x + Go(zf.t,c)) A.
x x!
t, A.
x x! + t, A,
x _|_ = x + Go(z),t, A.
end for
end for
return Video Pair (zf. x/)
return Video Pair (zf z/)
5

---

3.3 DenseDPO: Rich Temporal Feedback with Segment-Level Preferences
Although StructuralDPO effectively preserves dynamic degree, models trained with tend to gen- erate videos with lower visual quality and weaker text alignment compared to VanillaDPO. This performance gap because video pairs are structurally similar, which reduces diversity in the curated DPO dataset, and, as we discuss in Appendix B, can unintentionally drive the model to diverge from the real data distribution [53, 63]. straightforward solution is obtain more labeled data, but increases the annotation cost compared to VanillaDPO.. Instead, we explore an alternative approach to increase data and annotation diversity without increasing the number of labeled video pairs.
DenseDPO. In VanillaDPO and StructuralDPO, a scalar preference label / E {-1,+1] is obtained for the entire video of length T. Instead, DenseDPO annotates preferences on shorter temporal segments. Since guided video generation (Algo. 2) yields structurally similar video pairs, the same time period in both videos has a clear correspondence, making comparison feasible. We show an example in Fig. 2 (b) with the intervals being a single frame: for frame 1 and 2, the first video is better, while for frame 3, the second video better. Formally, given two videos (ax", .²") and the interval length s, we can break down videos into F' = ceil(? 4-) temporal segments of length by splitting along the time dimension. The resulting video pairs a)) f=)) are annotated with preferences over each segment, yielding segment-level dense preference labels I E +1,,+1+1 i.e., 1(x x²) = [1(=). x 2/)]/=1 Thus, following Eq. (5), we can formulate the DenseDPO objective as:
Uaf.. * (s(x",c,t,,) t, 0) s(==sss=='eee (i.aa,t...1))) (7)
[(0)= -E (c,v" st 1)DDD log (rp(t)) (0.1)
-B Σ
where s(-) is the implicit reward value on the f-th video segment.
In our collected dense preference data, we find that over 60% of video pairs have both winning and losing labels in I. In regular preference annotation such pairs will either be treated as ties or choose the video with fewer artifacts. In the latter case, this encourages the model to minimize loss videos with artifacts in Eq. (5), degrading the model performance. In contrast, DenseDPO assigns preference labels more accurately over time, only optimizing models on segments with clear difference.
Segment preference annotation with VLMs. Another benefit of the DenseDPO that it allows us use aff-the-shelf VLMs for automatic preference labeling. Prior works point out that existing VLMs struggle at assessing long videos (e.g., 5s) [21, 80]. often requiring task-specific fine-tuning and large-scale human annotations to train effective video reward models. Instead, we show that pre-trained VL are already capable of processing short clips (e.g.. 1s). Given two temporally aligned videos, we feed in pairs of segments into VLM, and ask it identify the better one. As will demonstrate in the experiments (Tab. 3), GPT-o3 [1] achieves high accuracy on segment-level preferences, leading DPO results competitive with using human preference labels.
4 Experiments
Our experiments aim to answer the following questions: (i) How does DenseDPO perform against VanillaDPO? (Sec. 4.2) (ii) Can we leverage existing VLMs to produce high-quality preference labels? (Sec. 4.3) (iii) What is the impact of each component in our framework? (Sec. 4.4)
4.1 Experimental Setup
We list some key aspects of our experimental setup here. For full details, please refer Appendix A.
Preference learning data. We curate a high-quality video dataset from existing datasets, resulting in around 55k videos. This is done by filtering the length, visual quality, and motion score of videos similar to [58], and prompting GPT-4o [3] to classify if the text prompt contains events of meaningful dynamics. This naturally gives text prompts and corresponding ground-truth videos.
Baselines. Our pre-trained text-to-video generator is a DiT [5s]- latent flow model. We fine- tune it on the curated high-quality data, termed the SFT baseline. For VanillaDPO. we randomly select 30k text prompts from the curated dataset, generate 2 videos of 5s per prompt with Algo. 1, and ask human labelers to annotate preferences. This leads to around 10k winning-losing pairs after

---

Table 1: Quantitative results on VideoJAAM-enchh [8]. We report automatic metrics from VBench [30] and VesionReward [80]. DenseDPO significantly outperforms Vanilla DPO in dy- namic degree, while achieves similar performance in other dimensions.
VBench Metries
VilloReward Metrics
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background Consistency
Motian Smoothness
Dynamic Degree
Text Aligoment
Visual Quality
Temporal Consistency
Dynamic Degres
Pro-trained
54.65
55.85
88.29
91.50
92.40
84.16
0,770
0.192
0.354
0.680
55.19
53.26
87.71
91.52
92.72
83.25
0.773
0.205
0.675
Vanilla DPO [46]
57.25
60.38
21.21
93.94
93.43
80.25
0.867
0.371
0.636
0.535
Structural DPO
56.38
59.78
90.21
92.34
92.94
84.69
0.843
0,341
0.602
0.652
DPnseDPO
56.99
60.92
91.54
93.84
93.56
85.38
0.863
0.376
0.632
0.680
Table 2: Quantitative results Motcon We report automatic metries from VBench [30] and VesionReward [80]. DenseDPO achieves similar motion smoothness compared to Vanilla DPO, while consistently outperforms it in visual quality, dynamic degree, and text alignment.
VRench Metries
VsionReeaard Metrics
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background Consisteacy
Motion Smoothness
Dynamic Degree
Text Aligument
Visual Quality
Temporal Consisteacy
Dynamic Degree
Pre-trained
56.21
56.26
88.23
91.67
93.56
83.69
0.261
0.112
0.154
0,840
SFT
56.16
55,54
87,94
92.42
94.44
84.93
0.273
0.105
0.129
0.845
Vanilla DPO [46]
57.51
61.20
91.45
93.49
97.43
72.55
0.355
0,172
0.239
0.709
Structural DPO
57.46
59.84
90.98
93.13
79.95
0.347
0.152
0.839
DenseDPO
57.54
61.52
91.60
93.72
97.33
84.73
0.359
0.179
0.232
0.858
removing ties, For StructuralDPO, we use the same 30k prompts from VanillaDPO, and label human preferences on videos generated using Algo. 2. The guidance level r) is randomly sampled from [0.65, 0.8] to obtain video pairs with similar motion. This again leads to around 10k winning-losing pairs. Both VanillaDPO and StructuralDPO apply the Flow-DPe loss in Eq. (5). Following prior works [46, 80], we set 3 to 500 and apply LoRA [27] with rank 128 to fine-tune the video model. We train with the AdamW optimizer [49] and global batch size of 256 for 1000 steps.
DPO implementation details. For fair comparison with baselines, we only take 10k video pairs from the StructuralDPO training data label dense preferences, which costs similar amount of human annotation time. The segment length 8 is set to 1s. Overall, more than 80% of video pairs have at least non-tie segment and can be used in DPO training, greatly improving the data efficiency over using global preferences. All other hyper-parameten are the same as DPO baselines.
Evaluation datasets. We utilize two benchmarks evaluate the performance of text-to- text-to-video gener- ation. VideoJAM- [8] contains 128 prompts focusing on weal-world scenarios with challenging motion, ranging from human actions to physical phenomena. We also construct MotionBench, which collects more diverse prompts from existing prompt sets [35, 58, 74] such as MovieGenBench.. We
run GPT-4o to select prompts with dynamic human actions, resulting in 419 prompts.
Evaluation metries. We aim to measure the visual quality, text alignment, and motion quality of videos. Specifically, we want to evaluate both the smoothness and strength of the motion. Therefore, we adopt VBench [30] and a state-of-the-art video quality assessment model, VisionReward [80].
4.2 DPO with Human Labels
Tab. I and Tab. 2 present the quantitative results on VideoJAM and MotionBench. Fig. 4 shows a qualitative comparison. VanillaDPO greatly improves the pre-trained model and the SFT baseline in all dimensions except dynamic degree due to the motion bias in video preference data. StructuralDPO retains the motion with paired video generation, while compromising the visual quality and text alignment. With the rich temporal feedback, DenseDPO consistently outperforms StructuralDPO. In addition, it matches all aspects of VanillaDPO and scores a significantly higher dynamic degree, despite using only one-third of labeled videos (10k 30k).
Please refer to Appendix C.I for comparison with more baselines including online RL-based methods. For more qualitative results, please check out Appendix C.2 and our project page for video results.
7

---

*In a studio, popping dancer creates precise isolation movements
DenseDPO StructuralDPo VanillaDPO Pre-trained
Figure 4: Qualitative results. Pre-trained model generates deformed limbs. VanillaDPO fixes it but generates almost static motion. StructuralDPPO retains dynamics but produces oversaturated frames. DenseDPO is the only method that generates correct limbs, large dynamics, and high quality visuals. Please check out our project page for video results baselines and our methods.
DanselPO wins
Ties
StructuralDPo wins
DenseDPO wins
Ties
Varilla DPO wins
TA-
TA
VO
TC-
Figure 5: Human evaluation DenseDPO vs. StructuralDPO (left) and VanillaDPO (right). TA,
VQ. TC, DD stand for text alignment, visual quality, temporal consistency, and dynamic degree.
Human evaluation. We conduct user study using all prompts from VideoJAM- in Fig. 5. We ask the participants to express their preference when presented with paired samples from our method and each baseline. DenseDPO consistently outperforms StructuralDPO in all dimensions. Compared to VanillaDPO, we achieve significantly higher dynamic degree. and are on par other aspects. Please refer to Appendix C.2 for more user study results.
4.3 DPO with VLM Labels
VLM labelers. As discussed in Sec. 3.3, we aim to evaluate the effectiveness of pre-trained VLMs in video preference learning. We take VLMs designed specifically for the video quality assessment task, VideoScore [21], LiFT [74], VideoReward [46], and VisionReward [80]. They have been fine-tuned on large-scale human preference labels. We also utilize the state-of-the-arr visual reasoning model, GPT o3 [1]. to explore the limits of models without task-specific training. Finally, we design GPT o3 Segment that partitions long videos into short segments to process separately, and aggregates results via majority voting. Please refer to Appendix A.6 for more implementation details.
Evaluation setup. We first test the preference prediction accuracy of VLMs two types of videos: Short Segment partitions videos into Is clips, and compares them separately. We report the accuracy on the 10k human preference labels used in DenseDPO. Long Video directly runs the model on the entire video except for GPT o3 Segment that aggregates segment-level results. We report the accuracy on the 30k human preference labels used in StructuraIDPO. In addition, we conduct DPO training using these VLM-generated labels, and report the VisionReward score on VideoJAM M-bbncc. Notably, run VLMS label video pairs generated from all 55k training data for better performance.
Results, Tab. 3a presents the results of preference prediction. With task-specific fine-tuning, state- of-the-art art video reward models outperform the advanced GPT o3 in assessing short clips. Yet, their

---

Table 3: VLMs results on human preference learning. (a) Existing VLMs excel at assessing short segments, yet their performance on long videos is still unsatisfactory. In contrast, by aggregating results over short segments, GPT o3 achieves the best accuracy without task-specific fine-tuning. (b) DPO training with GPT-based segment-level labels achieves performance close to using human labels, significantly outperforming other VLMs that predict single binary preferences.
Label Source
TA
VQ
TC
DD
Short Segment
Long Vidco
VLM
Pre-trained
0.770
0.192
0.354
0.680
VideoScore [21]
61.23%
51.23%
VesionReward [80]
0.785
0.347
0,521
0.622
LiFT [74]
65.03%
55.45%
StructuralDPO
VedeoReward [46]
71.89%
59.65%
GPT o3 [1]
0.759
0.284
0.506
0.621
StructuralDPO
VesionReward [80]
72.45%
62.11%
GPT o3 [1]
70.03%
53.45%
GPT o3 Segment
0.842
0.368
0.598
0.672
DenseDPO
GPT o3 Segment
70.03%
70.59%
Human Label
0.863
0.376
0.632
0.680
(a) Preference accuracy of VLMs on Is short segments and 5s long videos.
(b) DPO results with VLM labels on VideoJAM-bende [8].
Table 4: Ablation different variants of preference labels used for DPO training. We report VBench [30] and Rewion [80] metrics on VideoJAAM-bnne [8].
VBench Metrics
VisionReward Metries
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background Consistency
Motion Smoethness
Dynamic Degree
Text Aligament
Visual Quality
Temporal Consistency
Dynamic Degree
Pre-trained
$4.65
$5.85
88,29
91.50
92.40
84.16
0.770
0.192
0.354
0.680
StructuraIDPO
56.38
59.78
0.602
Majority Voling
56.48
59.72
90.15
92.35
92.98
84.54
0.847
0.342
0.608
0.645
Flip 40% Label
55.10
55.42
88.85
92.04
92.60
84.27
0.782
0.250
0.556
0.668
Flip Label
56.23
90.85
92.85
84.85
0.846
0.602
0.5 Label
55.62
56.13
89.23
92.48
92.99
84.25
0.806
0.305
0.589
0,655
2x Label
57.56
62.02
92.85
95.06
93.85
85.55
0,882
0.395
0.654
0.682
56.94
0.636
0.678
DenseDPO
$6.99
60.92
91.54
93.84
93.55
85.38
0.863
0.376
0.632
0.680
performance on long videos degrades drastically. Our further analysis in Appendix C.3 reveals that these models might be biased towards video content rather than temporal motion. Thanks to the temporal alignment of video pairs, we can run GPT to compare short segments individually, and then aggregate the results. This leads to higher accuracy in long video preference prediction.
We further show the DPO alignment results in Tab. 3b. Due to a higher preference accuracy, DenscDPO with GPT o3 Segment labels outperforms StructuralDPO with binary preferences from other VLMs. It even matches DenseDPO trained with human labels on text alignment, visual quality, and dynamic degree. Yet, please note that GPT label has 5.5x more videos than human label.
4.4 Ablation Study
We study the effect of each component in DenseDPO. All results are evaluated on VideoJAAM-bench.
Human labeling bias. We first study if there a systematic bias in the annotation pipeline, e.g., labelers may produce higher quality preference labels on short segments than long videos. To verify it, we aggregate labels of all segments within video to obtain its binary preference label via majority voting, and then train StructuralDPO on these labels. Tab. shows that it achieves similar results compared to StructuralDPO trained on globally labeled preferences. This proves that DenseDPO's superior performance comes from segment-level preference supervision instead of labeler bias.
Human label quality. We study how label noise affects DenseDPO performance. As shown in Tab. 4, we randomly flip 20% and 40% winning losing labels. This results clear drop in all the metrics.
Human label quantity. We ablate different amounts of segment preference labels used in DenseDPO. Tab, 4 shows that scaling to x labels leads the best results across all metrics. Interestingly, results

---

Table 5: Ablation on segment length s of dense preference labels. We report VBench [30] and VisionReward [80] metrics VideoJAM-bench [8]. All models are only trained 5k videos.
VBench Metrics
VisionReward Metrics
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background
Motion
Dynamic Degree
Text Aligament
Visual Quality
Temporal Consistency
Dynamic
Consisenccy
Smoothness
Degree
trained
54.65
55.85
0.192
0.354
0.5
55.57
57.03
89.88
92.76
92.52
84.25
0.811
0.326
0.601
0.643
(Ours)
55.62
56. 13
89.23
92.48
92,99
84.25
0.806
0.305
0,589
0.635
55.40
56.10
89.04
91.96
92.54
84.15
0.795
0.291
0.557
0.623
Table 6: Ablation on DenseDPO training with different amount of GPT o3 segment-level labels. We report VBench [30] and VisionReward [80] metrics on VideoJAM-bench [8].
VBench Metries
VisionRevand Metrics
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Text Aligument
Visual Quality
Temporal Consistency
Dynamic Dogree
Pre-trained
54.65
55.85
88.29
91.50
92.40
84.16
0.770
0.192
0.354
0.680
10k labels
55.21
56.83
91.45
92.38
84,20
0.782
0.247
0.360
0.668
35k GPT lahels
55.89
58.75
89.68
92.25
92.70
84.89
0.817
0,316
0.498
0.670
55k GPT lahels
56.23
60.15
90.75
93.01
92.99
85.21
0.842
0,368
0.598
0.672
trained with 0.5x labels are clearly better than results trained with 40% labels flipped. This may indicate that the quality of labels has a larger impact than the quantity of labels.
Ground-trute video in guided generation. In our experiments, we randomly sample 10k videos from the 55k high-quality dataset serve guidance videos. Here, randomly sample another set of 10k videos without overlap with the previously selected ones and conduct DenseDPO training. Tab. 4 (New GT Video row) shows that both runs achieve similar results, proving that our method is robust to the selection of ground-truth data as long as they are high-quality videos.
Dense label granularity. We study the impact of the segment length in dense preference labels. By default, s = 1 is used in all our experiments. Here, we tested 0.5 and & 2. Due to the high annotation cost, we only label 5k videos for each segment setting in this study. Tab. 5 compares DPO trained on 5k videos using different s. s = consistently outperforms s = due to more fine-grained preference annotation. Interestingly, hypothesize that this is because 0.5s is too short-a a longer context window is needed to assess the temporal aspect of videos. In addition, labeling dense preference at 0.5 is 2x expensive compared to s = 1. Therefore, we choose to label 1s segments in our experiments.
0.5 performs similarly to
1. We
VLM label quantity. Finally, we study DenseDPO performance when scaling up automatic VLM labels in Tab. 6. Since VLM labels are noisy, DenseDPO with 10k GPT labels achieves only marginal improvement compared to the pre-trained model. Nevertheless, the amount of VLM labels grows, we see consistent improvement in all metrics. This trend reveals that the quantity of VLM labels compensates for their quality, and our method scales well with the quantity of automatic labels. Since VLM itself an actively evolving field, we view this as a promising future direction to further scale up automatic preference learning with more data and better off-the off-the-shelf VLMs.
Conclusion
We present DenseDPO, an improved preference optimization framework for video generation. We address two critical aspects of video opo-ccoparison data curation and preference labeling. Our guided video generation mechanism and fine-grained preference labeling significantly improve over vanilla DPO. Furthermore, we show that segment-level labels unlock automatic annotation with off-the-shelf VLMs without task-specifi training. We discuss our limitations in Appendix D.
Acknowledgments
We would like thank Zhengyang Liang, Weize Chen, and Tsai-Shien Chen for valuable discussions, and Maryna Diakonova for help with data preparation and user studies.
10

---

References
[1] OpenAl o3.
card/, 2025. 6, 8,
2025.
26
[2] Kling2.0. https://klingai.com 2025.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023. 2, 6, 24, 26
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL. technical report. arXiv preprini arXiv:2502. 13923, 2025.
[5] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024. 1,2
[6] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 1952.
[7] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. OpenAl technical reports, 2024. URL https: //openai. con/research/vvides ott.raarrrrrriieeeeeeeeeerirooaa
penneration-modelle as-lldd--s www.oroddddmmmms.......mmmo 1,3
[8] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-ne motion representations for enhanced motion generation in video models. In ICML, 2025. 2, 5, 7, 9, 10, 26, 31
[9] Chaofeng Chen, Annan Wang. Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Enhancing diffusion models with text-encoder reinforcement learning. In ECCV, 2024. 2
[10] Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, et al. SkyReels-V2: Infinite-length film generative model. arXiv preprint arXiv:2504. 13074, 2025.
[11] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality vidco diffusion models. In CVPR, 2024. 5
[12] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsual Yang, et al. Panda-70M:: Captioning 70m videos with multiple cross cross- modality teachers. In CVPR, 2024. 24
[13] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 13000 2023. 2
[14] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 2, 24
[15] Hanze Dong. Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. TMLR, 2023. 2
[16] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024. 3, 24, 25
11

---

[17] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee. and Kimin Lee. DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models. NeurIPS, 2023. 2, 21
[18] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards ana- lyzing and understanding the limitations of dpo: A theoretical perspective. arXiv preprini arXiv:2404. 04626, 2024. 29
[19] Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with ai feedback. arXiv preprint arXiv:2412.02617, 2024.
[20] Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar. Jun- Yan Zhu, and Ha--Bin Huang. On the content bias in fréchet video distance. In CVPR, 2024. 32, 33
[21] Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, al. VideoScore: Building automatic metries to simulate fine-grained human feedback for video generation. In EMNLP, 2024. 3, 6, 8, 2,26
[22] Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022. 25
Classifier-free diffusion guidance. arXiv preprint
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 1, 29
[24] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen Vidco: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023033 2022.
[25] Jiwoo Hong, Sayak Paul, Noah Lee, Kashif Rasul, James Thorne, and Jongheon Jeong. Margin- aware preference optimization for aligning diffusion models without reference. In ICLR Workshop, 2025. 29
[26] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. CL2V Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024. 26
[27] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 7,25
[28] Zijing Hu, Fengda Zhang, and Kun Kuang. D-Fusion: Direct preference optimization for aligning diffusion models with visually consistent samples. In ICML, 2025.
(29) Qihan Huang, Long Chan, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, and Jie Song. PatchDPO: Patch-level dpo for finetuning-free personalized image generation. In CVPR, 2025. 3
[30] Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprchensive benchmark suite for video generative models. In CVPR, 2024. 3, 7.9, 10, 26, 31
[31] Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. CoMat: Aligning text-to-image diffusion model with image-to-text concept matching. NewrIPS, 2024. 2
[32] Shyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov, Jian Ren, and Anil Kag. Scalable ranked preference optimization for text-to-image generation. arXiv preprint 1r /8013, 2024."
[33] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-Pic: An open dataset of user preferences for text-to-image generation. NeurlPS, 2023. 3
12

---

[34] Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. SDPO: Segment- levell direct preference optimization for social agents. arXiv preprint arXiv:2501.0182/ 2025.
[35] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuan Video: systematic framework for large video generative models. arXiv preprint arXiv:2412.03603, 2024. 1, 7, 24, 26
[36] PKU-Yuan Lab and Tuzhan AI etc. Open-Sora-Plan, April 2024, URL https:://oi.orr//:
5281/zenodo 10948109. 24
[37] Black Forest Labs. Flux. http:///gg.ttt.occo////aoottttre
2024. 29
[38] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023. 2
[39] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402. 17245, 2024. 2
[40] Jiachen Li, Qian Long. Jian Zheng, Xiaofeng Gao. Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2V-Turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design. In ICLR, 2025. 3,3
[41] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. NeurIPS, 2024.
[42] Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. In CVPR, 2024. 3
[43] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. In CVPR, 2025. 3
[44] Yaron Lipman, Ricky TQ Chen, Heli HamuH Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023. 3, 24
[45] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-GRPO: Training flow matching models via online rl. arXiv preprint 2xXiv:2505.05477 2025. 33
[46] Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, Xintao Wang. Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, and Wanli Ouyang. Improving video generation with human feedback. arXiv preprint arXiv:2501.1391 2025. 3, 4, 7, 8, 19, 24, 25, 26, 27, 31, 32, 33, 36
[47] Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. VideoDPO: Omni-preference alignment for video diffusion generation. In CVPR, 2025. 3, 5, 19,29 29
[48] Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023. 3, 24, 25, 29
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019. 7, 25
[50] Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Mtep-Video-T2V technical report: The practice, challenges, and future of video foundation model. arXiv preprint 20224.250.. 2025.
[51] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wa, Jun-Yar Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 2, 5,
13

---

[52] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022. 1,2
[53] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.1322 13228, 2024. 2, 6, 29
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, nigh-peeformance deep learning library. NeurIPS, 2019, 25
[55] William Peebles and Saining Xie. Scalable diffusion models with transformers. ICCV, 2023. 6,
24
[56] Jun-cheng Chen Po-Hung Yeh, Kuang-Huse Lee. Training-free diffusion model alignment with sampling demons. In ICLR, 2025. 3
[57] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penma, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 24
[58] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoory Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie Gen: A cast of media foundation models. arXiv preprint WxXiv:2410.137200 2024. 6, 24, 26
[59] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:23/0.03739, 2023.
2
[60] Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737 2024. 1,3
[61] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly reward model. NeurIPS, 2023. 3,
[62] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020. 24
[63] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional Unalignment: Likelihood displacement in direct preference optimization. In ICLR, 2025. 2, 6, 29
[64] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. In NeurIPS Workshop, 2024.
[65] Team Seawead, Ceyuan Yang. Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo, Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Meng Wei, Zhiwu Qing, Fei Xiao, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Liangke Gui, Sheng Bi, Jiashi Li, Yuxi Ren, Rui Wang, Huixia Li, Xuefeng Xiao, Shu Liu, Feng Ling, Heng Zhang, Houmin Wei, Huafeng Kuang, Jerry Duncan, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Xiaobin Zhuang, Xiaojie Li, Xin Xia, Xuyan Chi, Yanghua Peng, Yuping Wang, Yuxuan Wang, Zhongkai Zhao, Zhuo Chen, Zuquan Song, Zhenheng Yang, Jiashi Feng, Jianchao Yang, and Lu Jiang. Seaweed-7B, Cost-effective training of vidco generation foundation model, 2025. 25
[66] Abhishek Sharma, Adams Yu, Ali Razavi, Andeep Toor, Andrew Pierson, Ankush Gupta, Austin Waters, Daniel Tanis, Dumitru Erhan, Eric Lau, Eleni Shaw, Gabe Barth-Maron, Greg Shaw, Han Zhang, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jakob Bauer,
14

---

Jeff Donahue, Junyoung Chung, Kory Mathewson, Kurtis David. Lasse Espeholt, Mare van Zee, Matt McGill, Medhini Narasimhan, Miaosen Wang, Mikolaj Binkowski, Mohammad Babaeizadeh, Mohammad Taghi Saffar, Nick Pezzotti, Pieter-Jan Kindermans, Poorva Rane, Rachel Hornung, Robert Riachi, Ruben Villegas, Rui Qian, Sander Dieleman, Serena Zhang, Serkan Cabi, Shixin Luo, Shlomi Fruchter, Signe Nørly, Srivatsan Srinivasan, Tobias Pfaff, Tom Hume, Vikas Verma, Weizhe Hua, William Zhu, Xinchen Yan, Xinyu Wang, Yelin Kim, Yuqing Du, and Yutian Chen. Veo, 2024. URL https: pestpa:/ddeepnee //epppideddd.d.. beceepinnd.gogleetteeeee
[67] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. In COLM, 2024. 3
[68] Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, and Tsung- Hui Chang. Tuning-free free alignment of diffusion models with direct noise optimization. ICML Workshop, 2024. 3
[69] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 11X.v:18/2.01117, 2018. 32
[70] Bram Wallace, Mcihua Dang, Rafael Rafailov, Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In CVPR, 2024. 1, 2, 3, 4, 19, 28, 29
[71] Ang Wang, Baole Ai, Bin Wen, Chaojic Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20311 2025. 3,2.2
[72] Binxu Wang and John J Vastola. Diffusion models generate images like painters: an analytical theory of outline first, details later. arXiv preprint arXiv:2303.02499, 2023.
[73] Fu Yun Wang, Yunhao Shui, Jingtan Piao, Keqiang Sun, and Hongsheng Li. Diffusion-NPOL Negative preference optimization for better preference aligned generation of diffusion models. In ICLR, 2025.
[74] Yibin Wang, Zhiyu Tan, Junyan Wang, Xiaomeng Yang, Cheng Jin, and Hao Li. LiFT: Leverag- ing human feedback for text-to-video model alignment. arXiv preprint arXiv:2412.04814 2024.
3, 7,7,,,922 26
[75] Xiaoshi Wu, Yiming Hao. Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv: 09341, 2023. 2, 24
[76] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human Preference Score: Better aligning text-to-image models with human preference. In ICCV, 2023. 2
[77] Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffusion models. In ECCV, 2024. 2
[78] Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, and Sergey Tulyakov. Mind the Time: Temporally-coontrllled multi-event video generation. In CVPR, 2025. 2
[79] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Rewae: Learning and evaluating human preferences for text-to-image generation. NeurlPS, 2023. 2, 24
[80] Jiazheng Xu, Yu Huang. Jiale Cheng, Yuamming Yang. Jiajun Xu, Yuan Wang. Wenbo Duan, Shen Yang. Qunlin Jin, Shurun Li, Jiayan Teng, Zhuoyi Yang, Wendi Zheng, Xiao Liu, Ming Ding, Xiaohan Zhang, Xiaotao Gu, Shiyu Huang, Minlie Huang, Jie Tang, and Yuxiao Dong. VesionReward: Fine-grained multi-dimensional human preference learning for image and video generation, arXiv preprint arXiv:2412.2 2024. 2, 3, 5, 6, 7, 8, 9, 10, 19, 24, 25, 26, 31, 32, 33
15

---

[81] Hongwei Xue, Tiankai Hang, Yanhong Zeng. Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, 2022. 24
[82] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. DanceGRPO: Unleashing grpo on visual generation. arXiv preprint arXiv: 078888, 2025. 33
[83] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In 31
CVPR, 2024.
[84] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer. In ICLR, 2025. 3,5, 24
[85] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizerr is key to visual generation. arXiv preprint arXiv:2310.05737, 2023, 24
[86] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen Hc, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. RLHF-V: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In CVPR, 2024. 3
[87] Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. InstructVideo: Instructing video diffusion models with human feedback. In CVPR, 2024. 1,3
[88] Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. NeurIPS, 2024.
[89] Ailing Zeng, Yuhang Yang, Weidong Chen, and Wei Liu. The dawn of video generation: Preliminary explorations with sora-like models. arXiv preprint arXlv:2410.05227, 2024. 1,
[90] Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, et al. SePPO: Semi-policy preference optimization for diffusion alignment. arXiv preprint arXiv:2410.05255, 2024. 3, 25, 31
[91] Jiacheng Zhang, Jie Wu, Weifeng Chen, Yatai Ji, Xuefeng Xiao, Weilin Huang, and Kai Han. OoOi Align video diffusion model with online video-centric preference optimization. arXiv preprint 24xiv:24/2..5/111 2024. 3
[92] Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In ICCV, 2023. 3
[93] Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimennional human preference for text-to-image generation. In CVPR, 2024.7 3,3
[94] Tao Zhang, Cheng Da, Kun Ding, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model as a noise-aware latent reward model for step-level preference optimization. arXiv preprint
0tt::2502.00055.1 2025. 3
[95] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all, March 2024. URL https: / //ggtttttuuttt www.ggtb...//pccaiiteeoeoeoeeeee hwtp./////accaaiccc/eepeeeaa-ee. Sora. 24
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose DenseDPO, improved direct preference optimization framework for video diffusion models. We have verified it with extensive experiments in Sec. 4 and Appendix C.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification: The main paper does not include theoretical results. We have an analysis of a potential failure cause of StructuralDPO in Appendix with detailed derivations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all implementation details about dataset processing, model archi- tecture, and training and inference in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not have enough time to clean up the code at submission time. For data, we provide all details about our processing steps in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. ccccccpppoo wwtppblcc//aaeccccccccccccccccct mublic/guides /ilee//CodSubbii (CodeSubmissionoollll ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide all implementation details about dataset processing, model archi- tecture, and training and inference in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars following common practice in the field [46, 47, 70, 80]. In our preliminary experiments, we ran with three random seeds, and empirically observed small variance in the automatic metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all required information in Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttps://neurrie cwtpurips.ccpppppeepe cttppccccpcc/cccttcccccccccccllt csGuidelines"
Answer:
[Yes]
Justification: Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential positive and negative society impacts in Appendix D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We discuss safeguards on our video generator in Appendix D.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We provide the license information in Appendix A.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.ttcoe...m com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new asset is introduced in the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide details about human preference annotation collections in Ap- pendix A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Research with human subjects is main part of the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
22

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the prompts used for LLMs in Appendix A.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / /ttpa/rppp.cc/oooeeeeeeeeeeeeeee /ttpc//oeeeencce///22/////////// for what should or should not be described.
23

---

A
Detailed Experimental Setup
In this section, we provide full details on the datasets, baselines, evaluation settings, and the training and inference implementation details of our model.
A.1 Training Data
Following common practice in diffusion model post-training [14, 35, 71], we curate a high-quality data subset from existing large-scale video datasets [12, 81]. We mostly follow [58] to filter the resolution, duration, aesthetic quality, and motion strength videos. Inspired by VideoAlign [46], further apply GPT-44 [3] to only retain prompts with non-trivial motion.
GPT-4o Prompt Filtering Template
Please help me classify if text prompt contains challenging dynamics. Ignore camera motion and description of the background in the text prompt. Only focus on foreground objects. In general, we are looking for complex motion that requires precise, coordinated movement such as doing sports.
Here some good examples:
skateboarder performs jumps.
2. On rainy rooftop, a pair of hip-hop dancers lock and pop in perfect syne.
3. A figure skater executes powerful leap.
4. woman transitions gracefully
aerial hoop under golden hour light. acrobat holding handstand on mat.
martial artist performs a spinning hook kick in misty bamboo forest.
Additional rules:
Only keep real-time videos, and remove any video that dlow-motion, time-lapse, or acrial shot; Remove videos with more than five major subjects in the scene, such as sports games or a group of people doing something;
3. Remove videos with any of the following content: screen-in-screen, screen recording, special effects from video editing, cartoon, animation, TV news, and video games;
When not violating the previous rules, you can keep videos with any the following content: eating, cutting, or any action that causes big deformation the main object, such as "A person takes a bite of a hamburger / cuts a steak squeezes sponge makes a dough";
5. Be conservative. If you are uncertain about prompt, please classify it as "no".
Please reply "yes" or "no" in the first line of your response. Then, plcase explain your decision in the second line. I will now provide the prompt for you elassify: [PROMPT]
The GPT-4o template we used is shown above. Overall, our final dataset contains around 55k high-quality text-video pairs, which we used to generate preference data.
Data processing. The training dataset contains videos of different lengths, resolutions, and aspect ratios. Following common practice [57, 95], we use data bucketing, which groups videos into fixed set of sizes. Overall, we sample videos up 512 resolution and 5s during training.
A.2 Baselines
Pre-trained model. Our base text-to-video model is a latent Diffusion Transformer framework [55]. leverages a MAGVIT-v2 [85] as the autoencoder and stack of 32 DiT blocks as the denoiser Ge.
The autoencoder is similar to the one in CogVideoX [84], which downsamples the spatial dimensions by 8x and the temporal dimension by 4x. x. Each DiT block is similar to the one Open-Sora [36], which consists of 3D self-attention layer running all video tokens, cross-altention layer between video tokens and T5 text embeddings [62] of the input prompt, and an MLP.
The base model adopts the rectified flow training objective [44, 48]. We mostly follow the design choice of Stable Diffusion 3 [16], e.g., Logit-Normal Normal distribution of noise levels.
REFL [79] directly maximizes the reward from reward model via gradient ascent. We tried REFL our preliminary experiments with HPSv2 [75] and Vewion [80] as the target reward models. However, observed severe reward hacking, c.g., the video frames become oversaturated and the

---

temporal motion gets static. It also requires huge GPU memory and computation as we need to decode latent tokens to raw pixels to upply these VLMs and backpropagate gradients through them. Since the results are clearly worse than DPO, we did not continue this direction and do not report their results in the paper.
SFT fine-tunes the pre-trained model on the 55k high-quality dataset described in Sec. A.1 for 5k iterations. We did not observe a clear difference between full-model and LoRA [27] fine-tuning, and thus choose LoRA fine-tuning as it is more efficient.
D3PO [83] shares similar objective function VanillaDPO. We adapt their official codebase' to our video setting, and use the same preference labels as in VanillaDPO.
DPOK [17] is an online RL-based method. We also adapt their official codebase to our video setting, and leverage VisionReward [80] to provide online reward feedback. In our experiments, we found that DPOK is unstable to train and easily results in reward hacking. Thus, we use low learning rate of 10-6 and a high KL. weight to prevent training divergence.
SePPO [90] can be viewed as a combination of DPO and SFT, as its preference data come from samples generated by past checkpoint of the current model and pre-collected high-quality data. Since no training code is available for SePPO, we re-implement it based on its paper.
VanillaDPO follows the common direct preference optimization (DPO) practice for video diffusion models [46, 65, 80]. We randomly select 30k text prompts from the curated dataset, and generate 2 videos of 5s per prompt to obtain a comparison pair. We then ask human annotators to choose a better video or a tie by considering text alignment, visual quality, and temporal consistency. labelers assigned
pair, with reviewer to correct any potential errors. This leads to around 10k winning-losing pairs after removing ties. Indeed, human preferences in video are influenced by multiple, sometimes inversely correlated, factors, making it hard obtain a clear preference.
StructuralDPO uses the same 30k prompts as VanillaDPO to construct comparison pairs. For a text prompt, sample a guidance level 17 ~ (0.65, 0.8), and 2 Gaussian noises from 2 random seeds. Then, we add the noise to the ground-truth video corresponding to the text prompt according to n, and denoise from it to generate video. We perform the same human preference annotation process, again leading to around 10k non-tie data pairs.
We have tried using different 7) when generating the video pair. Usually, the video generated with more guidance is preferred over the other one (e.g., has fewer artifacts). However, we observe that the model quickly achieves low DPO loss when trained on this data, yet the quality of generated videos is not improved. We hypothesize that different ry leads to statistical differences in generated videos, and the model learns shortcuts using such cues instead of truly understanding human preferences.
DPO training hyper- paremmttraeesss. Both VanillaDPO and StructuralDPO adopt the Flow-DPO loss with a constant 3, as prior work shows it achieves the best performance [46]. Following prior works [46, 80], we use B = 500 and apply LoRA [27] with rank 128 to fine-tune the video model. We train with the Adam optimizer [49] and a global batch size of 256 for 1K steps. The peak learning rate set to 10-5 and is linearly warmed up from in the first 250 steps. A gradient clipping of 1.0 is applied to stabilize training. We implement all models using Py PyTorch [54] and conduct training on 64 NVIDIA A100 GPUs, which takes around 16 hours.
Inference hyper-parameters. We use the rectified flow sampler [48] with 40 sampling steps and a classifier free guidance [22] scale of 8 to generate horizontal videos of 512 288 resolution. A timestep shifting [16] of 5.66 is applied to further improve results.
A.3 DenseDPO Implementation Details
For a fair comparison with baselines, we only label dense human preferences on 10k randomly sampled video pairs from the StructuralDPO training data, which costs a similar amount of human annotation time compared to labeling 30k global binary preferences. The segment length 8 set to Is. Overall, more than 80% of video pairs have at least non-tie segment and can be used in DPO training, greatly improving the data efficiency over using global preferences. Since the DPO loss is now averaged over all non-tie tokens within training video pair, we sample video clips that have
httr:::://gcchabbcomm/////s/// Phttps://githuu
25

---

more than 20% non-tie segments to avoid a small effective batch size. All other training and inference hyper-parameters are the same as DPO baselines.
A.4 Evaluation Datasets
At the beginning the project, we experimented with the prompt suite VBench [30]. However, noticed that the model can achieve high scores with good visual quality and even degraded motion dynamics. Since our main focus is improving the temporal quality of the pre-trained video model and VanillaDPO, choose to evaluate on two motion-rich prompt sets.
Video.JA.--bene [8] contains 128 prompts focusing on real-world scenarios with challenging motion, ranging from human actions to physical phenomena.
MotionBench collects more diverse prompts from existing prompt [35, 58, 74]. We GPT-4o to select prompts with dynamic actions as described in Sec. A.I, resulting in 419 prompts. This prompt contains more diverse scenes, subjects, and action types.
Note that, to ensure a fair comparison between methods, we use the original text prompt without any prompt rewriting extension process.
A.5 Evaluation Metrics
Inspired by prior works [46, 80], identify three key aspects in text-to-video generation: visual quality, text alignment, and motion quality. Since VanillaDPO has the motion bias issue that leads visually pleasing videos with reduced motion strength, want evaluate both the smoothness (i.e., temporal consistency) and the strength (i.c., dynamic degree) of the video in a disentangled way. Thus, we cannot use metrics like VideoReward [46] as it only contains "motion quality" dimension.
VBench [30] is a comprehensive benchmark that tests different aspects video generation model. When testing custom prompt sets, it supports six dimensions covering the visual quality, temporal consistency, and dynamic degree. Following the official evaluation protocol, we rum each model to generate 5 videos using each prompt with 5 random seeds.
Vewid [80] is a state-of-the-art video reward model that fine-tunes pre-trained Vision Language Model (VLM) [26] large-scale human video preference data. It breaks down the human preference into 64 aspects, which can be categorized into 9 dimensions. We take the "Alignment" dimension text alignment, merge "Composition", "Quality", "Fidelity" dimensions into visual quality, merge "Stability", "Physics", "Preservation" dimensions into temporal consistency, and take the "Dynamic" dimension as dynamic degree to report in the final result.
A.6 VLM-based Preference Labeling
VLM labelers. We aim to evaluate the effectiveness of existing VLMs in video preference learning. This includes both off-the-shelf models and models fine-tuned for the preference prediction task. For fine-tuned VLMs, we input both videos and take the one with a higher score the winning data:
VideoScore [21] is fine-tuned on human-labeled scores 1-3s short videos. We average all five dimensions of its output as the overall score video;
LiFT [74], VideoReward [46], and VisionReward [80] are all fine-tuned on >5s long videos generated by modern video models. We average their output dimensions take the overall dimension (if presented) as the final score of a video.
For off-the-shelf VLMs, we take GPT o3 [1] as we find it to outperform GPT-4o [3] due to the visual reasoning capability. We follow the official guide² to prompt the model follows:
http://ccookoook..oeo.eoooomacap oisos for video siiieoo_nnrsttddinggg
26

---

GPT o3 Video Preference Labeling Template
Please help me compare two videos generated by our rext-to-video diffusion model. I will provide you with frames sampled from the videos. The two videos structurally similar (e.g. global layout and motion are similar), so I only want to compare their details. Please assess which one has higher quality, i... the video that contains fewer artifacts. Pay close attention to visual artifacts such as:
Additional fingers or legs, deformed human limbs, morphing human faces or body parts:
Blunyy or distorted objects, slight motion blur is fine, but the object should not be completely distorted; Abrupt changes in the object, such as objects appearing/ daaerringgdspapariinng unexpectedly, or anything that should not happen in the real world, c.g., rigid object deforming or melting.
Please only answer "tie" (two videos have equal quality). "first" (the first video has fewer artifacts), and "second" (the second video has fewer artifacts), followed by simple explanation. Be conservative in your answer. If you see similar amounts of artifacts in both videos, please choose "tie". Only select "first" or "second" when one video is clearly better than the other.
These are frames from the first video, sampled at FPS:
Video frames
These are frames from the second vidco, sampled at 8 FPS:
Video 2 frames
Please compare the two videos and tell me which one is better.
To improve accuracy, we apply a self-consistency check by reversing the order of Video 1 and Video 2. If the predictions on both orders are the same, we keep Otherwise, we treat it tie. This simple strategy improves the accuracy on short segments by around 10%. We note that there might be better strategies in prompt construction, such as concatenating paired video frames side-by-side, or organizing frames into a grid. We leave further investigations for future work.
Finally, design GPT 03 Segment that partitions long videos into short is segments process separately. This gives the dense preference labels compatible with our DenseDPO framework. To obtain a global preference for the entire long video, simply apply majority voting.
Preference prediction setup. Prior work [46] pointed out that existing VLMs excel at processing short videos, while falling short on long videos. Therefore, we evaluate two cases:
Short Segment that predicts human preferences on Is clips. We report the accuracy on the 10k dense human preference labels used in DenseDPO training;
Long Video that predicts human preferences on the entire video, except for GPT o3 Segment that aggregates segment-level results. We report the accuracy on the 30k binary human preference labels used in StructuralDPO training, which a superset of the previous case.
When calculating the prediction accuracy, we skip tie labels and only compute results on segments or videos with non-tie ground-truth preference labels.
DPO training setup. We apply StructuralDPO on binary preference labels produced by GPT o3 and Rewrd as it achieves the highest accuracy. We also apply DenseDPO on dense preference labels produced by GPT o3 Segment. Notably, we run VLMs to label video pairs generated from all 55k training data to explore the limit of automatic preference learning performance.
B Caveat of Guided Sampling
B.1 Analyzing the Learning Signal
As discussed in Sec. 3.2, guided sampling is attractive since it fixes the structure in the preference pair. This neutralizes the motion bias between videos and focuses the comparison on visual artifacts. However, StructuralDPO with guided sampling achieves inferior results. Analyzing its learning signal reveals that it can paradoxically push the model to "unlearn" the real data distribution. Intuitively, this happens because the learning signal from losing sample typically dominates over the winning one in those regions of a video, which correspond to the real data distribution. In this section, we investigate this phenomenon and discuss potential remedics to the issue.
27

---

For our analysis, we will use the original DPO notation [70] to simplify the exposition and emphasize that the argument applies for the most general diffusion DPO setup. Diffusion DPO training loss is formulated as:
[())- - Eaay (n (0,7( mala"" (mn'), m) mpm-g(a(mm)) (mb)) logo -3T(
(8)
+ DKL (g(x) ax,,)) || pe (x x²(")) - DKL (g(=)11 xb,,)) Prot (2 2²
(9)
- -DRL (1(2)-1 xb,1)) || pa (1 x))) + DKL (g(a4-1 ab,t) Pier( xx))
(10)
where (at 2/)) is the winning-losing preference pair, T is the number of diffusion steps, f ~ (00 T) the noise level distribution, and q( zu) = Max ayTo, oTJ). In Diffusion- DPO [70], the objective is further simplified to:
((0) = -m mD) (-,7( of a(( (ng at (=) log (oggo--TTTarl --BTw(t(
(11)
|| - ca( 1) | ( 12 - ||e' co 1)112 + |||e e 1) ).
(12)
where (t) is weighting function. Let's denote:
A ||e" = 1)112
A ||e' cD (0 111
(13)
A a ||e.0 (xr( .)]1
A = ||e' Grel Grer 1)12
(14)
Then, the loss gradient is:
Vol(0)=
VoL [log (-STw(t) (A - - (A - A())))
(15)
EE [[L (T)) Vo(A7 A (4/ -a)
(16)
= E [C Vo - -b)
(17)
where C' C- (1(1)o(()
One can show that C > O since o..) E 1) and B, T, (tt) > 0. Since C > 0, the per-pixel direction of the incoming gradient w.r.t. the model outputs is determined entirely by the interplay between V.Ay and V.d. This fact becomes crucial for StructuralDPO with guided sampling for the following reason.
In guided sampling (see Algo. 2), we generate a winning/losing sample x; = (1 - (1-7) -7 + nE using real video 22g. This makes them carry similar structure, which means that the winning and losing samples share many pixels. Moreover, these shared pixels normally correspond to the ground-truth data distribution. Let's split the pixels into two sets, Ismee and Tunique:
Isamo (x0" (x",x') = {p x [p] R ablp se volp]
x²)
(18)
Zunique (x) x1) = {p p o Isame).
(19)
where p is a pixel location (e.g., typically, triplet (i, k) denoting the frame, height, width indices). In this way, Isame stores pixel locations which remained intact during the forward diffusion process of guided sampling and subsequent denoising with Gg. Fig. 2 and Fig. 11 illustrate this: many pixels in the winning and losing samples are identical and correspond to the original ground-truth video. In this way, one can argue that Tame corresponds to the real data distribution.
Let's denote:
A 1 - (i)( t)
(20)
i... the diffusion loss in particular pixel locations I. Now, if AT" samee] < A samee] (meaning that the diffusion error in ground-truth pixel locations of a losing sample is higher than that of the winning one), then the gradient will be dominated by the negative contribution of A [amee and the model will be unlearning real data distribution. Turns out, this is exactly what is happening in practice:
Prior to DPO, the model undergoes extensive supervised fine-tuning (SFT), so it rea- sonable to expect that at initialization, we have DKL (g(x)" xx,e) |l (x xx)) <
28

---

DKL (q(x)-1 ahet) Il po
x))), meaning that the model is closer in distribution
to winning pairs and interprets the entire image as unlikely when there are artifacts
present in some part of it. The condition DyI. (g(#1_1 x.,e)) l| (po(a]/_ [=(x)=)=+))))))) <
DKL. (1(0)_1 æb.e) (o xf)) implies AØ < A) (see Appendix 2 of [70]).
The model suffers from an internal distribution shift in the presence of artifacts and its predictions in good pixel locations deteriorate in the presence of artifacts. Besides, it might be shifting its capacity towards rectifying the artifacts, so the rest of the output suffers.
Given the above two observations, we conclude that the model will be unlearning the real data distribution in the StructuralDPPO setting. While the above analysis is outlined for diffusion models with e- e-prediction parametrization [23]. it holds for t- o-prediction as well (used in both our and many contemporary works that align rectified flow models [48]) with argument derivation being basically the same. We also emphasize that even marginal domination of the gradient from a losing sample over the one from the winning sample results in such "unlearning" behavior.
Moreover, practitioners commonly use the same noise seed (i.e., e" = e') for both the winning and losing samples, following the original DiffusionDPO implementation." 4 This exacerbates the problem: the learning signal from a losing sample no longer dominates merely in expectation per pixel, but at each training step, entirely suppressing the learning signal from the winning sample.
Several prior works observed similar behavior in DPO on language models [18, 53, 63]. For example, DPO-Positive [53] shows that on datasets with short edit distances between winning and losing samples, DPO may lead to a reduction in the model's s likelihood on the preferred examples.
Den is a natural way to eliminate this shortcoming of StructuralDPO since it is designed to provide dense per- per-pixel DPO objective and would allow to treat the pixels from Isame (i.e., similar pixels) as ties, thus removing any loss on them. In the ideal world, we would love to have per-pixe pixel preference labels for DenseDPO training, but, as work demonstrates, even coarse-level temporal ones allow us to recover and improve the DPO performance.
One can also investigate other strategies to mitigate the issue by taking into account the similarity between pixels (e.g., uncertainty-base or margin-aware DPO [25, 47J) or reformulating the DPO in some novel way without maximizing the loss of losing samples. We leave this for future work.
B.2 Empirical Verification
While the previous section presents our theoretical argument, must verify empirically whether the assumption regarding the dominance of the losing sample's loss over the winning sample indeed holds. Here, we address this question through empirical analysis using the Flux-dev [37] model. We deliberately chose a popular, open-source e-prediction model to ensure our conclusions remain general and reproducible rather than specific to our internal video model.
We constructed a synthctic dataset containing controlled artifacts to facilitate this analysis. Specifi- cally, we selected 5, 195 real-world images with a resolution of 1024² and artificially corrupted them by applying blur to their central patches. Each image is encoded using the FluxAE encoder, resulting in latent tensor of dimensions 128 128 x 16. We then applied increasing levels of corruption to the central 32 x 32 patch (representing 6.25% of the image) with the following blur intensities:
1. No corruption (0% blur)
2. Blur with k = 2 (6.25% of the patch size)
3. Blur with A: 4 (12.5% of the patch size)
4. Blur with & = 8 (25% of the patch size)
5. Blur with k = 16 (50% of the patch size)
The resulting dataset comprises five image variants, progressively more corrupted, visualized in Fig. 6 (top). This setup allows separate measurement of losses in corrupted and uncorrupted regions. Subsequently, we randomly sample a timestep L ~ 4[00 1]. estimate velocities Vpinx (2 and
coa http:///hheeeeeeeeeeeee///aeaaaa ott.a////.oo..oooo.o....ooo.o.o.
29

---

Blur: 0%
Bluc: 6%
Blur: 12%
Blur: 25%
Bluc: 50%
10000
0.008
0.006
0.002
0.000
Figure 6: Visualization of losing sample loss dominance in uncorrupted regions. Top: example images with progressively increasing blur in the central region (note, that the visualization corresponds to an equally-corrupte RGB image, rather than the decoded corrupted latent tensor). Bottom: the per-pixel loss difference &g between losing and winning samples averaged across latent channels. Positive values indicate the dominance of losing sample losses, driving the model to unintentionally degrade predictions in artifact-free areas. For clarity, clamp maximum values in the visualization to the maximum loss difference observed in uncorrupted regions, preventing extreme outliers from dominating the heatmap.
Loss Difference per Blur Kernel
Blur Kernel
Figure 7: Average loss difference mean(62) in uncorrupted regions as a function of artifact severity. Increasing blur severity amplifies the losing sample's 's negative learning signal in good (uncorrupted) regions, illustrating the risk of inadvertently unlearning correct predictions.
Uflux t), and compute per-pixel losses (averaged over the 16 latent channels C):
C UFiux (x)" (x),, tcl - (c- (<->>")<c)) (c))
(21)
1 (Wrix (z/, t)|d (c =c)(
C' c'=
(22)
where (.)2 denotes element-wise squaring, Next, for each sample, we calculate the loss difference
dc E 18 hx as follows:
82 L ".
(23)
5c indicates the extent to which the losing sample's 's loss surpasses that of the winning sample. this dominance occurs in Iaaee it implies the model is unlearning these areas due to the negative contribution from the losing sample.
Loss computations use the same noise seed for the corrupted and uncorrupted images (as is usually done in practice). Fig. (top) illustrates representative example of corruption. Visualizations in Fig. 6 (bottom) clearly demonstrate the loss dominance of the losing samples in the uncorrupted
30

---

Table 7: Quantitative comparison with more baselines on VideoJAM-bench [8]. We report automatic metrics from VBench [30] and VisionReward [80].
VRtechh Metries
VisionRewand Metrics
Method
Aesthetic Quality
Imaging Quality
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Text Alignment
Visual Quality
Temporal Consistency
Dynamic Dogree
Pre-trained
54.65
55.85
88.29
91.50
92.40
84.16
0,770
0.192
0.354
0.680
D3PO
82.53
0.322
0.602
DPOK [17]
54.99
56.28
89.14
92.36
92.95
78.65
0.795
0.337
0.518
0.457
ScPPO [90]
55.83
57.42
89.93
92.65
92.93
82.85
0.841
0.326
0.554
0.587
Vanilla DPO [6]
57.25
60.38
91.21
93.94
93.43
80.25
0.867
0.371
0.636
0.535
DenseDPO
91.54
85.38
0.376
0.680
MrructuraDPS wins
Ties
VaPillaDPO wins
DenseDPO wins
Ties
Pre-trained wins
47.3%
20.7%
72.5%
34.6%
41.4%
$1.0%
33.3%
15.1%
36.8%
14.0%
45.0%
10.4%
39.8%
42.7%
17.4%
38.5%
23.9%
37.7%
(a) StructuralDPO vs. VanillaDPO
(b) DenseDPO vs. pre-trained model
Figure Human evaluation on the VideoJA.M- dataset. TA, VQ. TC, DD stand for text alignment, visual quality, temporal consistency, and dynamic degree.
regions. As we maximize the loss of the losing sample and minimize that of the winning sample, this results in the model increasing loss in uncorrupted image regions when trained with DPO to mitigate blurring artifacts.
Fig. 7 provides quantitative analysis, demonstrating how increasing artifact severity intensifies the negative learning signal from losing samples in uncorrupted image areas.
C Additional Experimental Results
C.1
Comparison with Additional Baselines
In Tab. 7, we compare with additional baselines on VideoJAM--ench [8]. D3PO [83] achieves similar results as VanillaDPO since they are both DPO-based methods. DPOK [17], an online RL method, slightly improves the visual quality and temporal consistency compared to the pre-trained model, yet significantly degrades the dynamic degree. This is because the reward model uses, Rewid [80], is biased towards per-frame visual quality instead of temporal motion (see more analysis C.3). Thus, training with such online feedhack leads to static motion. SePPO [90] achieves better dynamic degree than VanillaDPO, while performing worse visual quality, Instead of using offline- geniereee losing samples, SePPO generates online samples with reference model, and designs filtering method (AAF) assess its quality. However, we noticed that AFF is based on model denoising loss, which sometimes unreliable. In those cases, the model optimized on low-quality videos, leading degraded visual quality.
C.2 DPO with Human Labels
Human evaluation. Fig. presents an additional user study on VideolAM-benchh. Fig. 8a shows that StructuralDPO outperforms VanillaDPO in dynamic degree it performs DPO on video pairs with similar motion. Yet, it underperforms other dimensions. Fig. 8b shows that DenseDPO consistently beats the pre-trained model in all dimensions.
Qualitative results. show more videos generated by DenseDPO Fig. 9. Fig. compares DenseDPO with baselines. Overall, DenseDPO aligned model generates videos with high visual quality, rich motion, and precise text alignment. Please check out our project page for video results of baselines and our methods.
31

---

"A men exercising with battle ropes at a gym
transitions gracefully aerial hoop under golden hour light
"A weightiffter performs a deadlift with perfect form in a concrete garage gym
"A giraffe stepping gingerly along tightrope above city plaza, drawing gasps from the crowd below"
"A raccoon rolierblading in skate park, performing small jumps off the ramps"
"A raccoon riding massive with style"
"llow motion of water drop crown formation"
Figure 9: Text-to-video results with DenseDPO aligned model. Here, show generation of challenging human activities, novel animal actions, and physical phenomena. Please check out our project page for video results of baselines and our methods.
C.3 DPO with VLM Labels
Motion bias in VLMs. As discussed in Sec. 3.2, there is a motion bias in human preference annotation human labelers tend favor artifact-free dlw-motion clips over dynamic clips with artifacts. Ge et al. [20] pointed out that video metrics such as FVD [69] are also biased towards per-frame visual quality rather than temporal motion quality. Here, we study whether more advanced VI.M-based video reward models also suffer from this issue. We test two state-of-the-art models, VesionReward [80] and VideoReward [46]. We randomly sample 10k videos from the VanillaDPO training data, each having 121 frames (5s). For each video, construct static version of it by duplicating one frame of it, and we compare this static video with the original video.
Tab. 8 presents the winning rate of static vs. original video. Surprisingly. VisionReward favors static videos over original videos with sizable gap, indicating clear motion bias. In contrast, VideoReward prefers original videos in around 80% of cases. We note that both VisionReward and VedeoReward output multi-dimeensional scores (e.g.. visual quality, dynamic degree), and aggregate them to predict the binary human preference. VideoReward simply averages all dimensions, and thus not biased to any dimension. VisionReward instead first labels human preferences on video pairs, and then learns per-dimension weights via logistic regression. This inevitably inherits the motion
32

---

Table Winning rate of static video vs. original video measured by video reward models. The static video is constructed by duplicating frame to the video length, where we tested frame 0, 24, 48, and 96 here. A lower winning rate means the video reward model more sensitive to motion.
Frame 0 vs. Original video
Frame 24 vs. Original video
Frame 48 vs. Original video
Frame 96 vs. Original video
Method
Rewidn [80]
70.63%
68.28%
69.84%
69.06%
VideoReward [46]
20.33%
17.76%
17.72%
17.91 %
bias in human labels. Indeed, Tab. 25 in the VisionReward paper [80] reveals that human preference is negatively correlated with object dynamics, while positively correlated with temporal smoothness.
These results suggest that it is better to label per-dimension scores and predict them, instead of predicting an overall score. The bias in human preferences will be leaked into the reward model we train the model to regress it. We note that our analysis is still preliminary. Further investigations similar to [20] are required to fully understand the bias in recent VLMM-bssed video reward models.
GPT dense preference label. We visualize some dense preferences predicted by GPT o3 Segment in Fig. 11. Overall, GPT is able to identify obvious artifacts such as distorted faces and deformed limbs. With our carefully designed prompt and self-consistency check, it only predicts a preference when there is a clear difference between two segments. However. still does not understand complex motion, such as playing tennis and cartwheels. This is partially because GPT o3 Segment only has access to video clips, which is too short to finish these actions.
C.4
Failed Attempts
Here, we record some failed approaches we tried.
Real videos as winning data. Since our goal is to improve the motion quality of video models, real-world videos can naturally serve as winning samples as they follow physical laws perfectly. We tried DPO using our 55k high-quality training videos winning data, and videos generated by model as losing data. The implicit accuracy quickly converges to almost 100% within 500 steps, yet the model generation does not improve. This is likely because real videos are significantly better than the generated ones, which fails to provide useful signals to improve the video model.
Video pairs with different guidance 1). Intuitively, videos generated with more guidance are often better than those with lower guidance. We tried generating video pairs with different 1) and assign winning-losing labels based on it. This gives us preference labels "for-free". However, DPO on this data does not improve model generation. Our hypothesis that the model may infer the noise level from the generated samples instead of measuring its quality.
D Limitations and Future Works
Similar to prior works [46, 80), we also observed unstable training and reward hacking when fine- tuning the entire model. As a result, have rely on LoRA training and early stopping. This is in stark contrast to DPO large language models, where DPO training is relatively stable. More investigation on diffusion DPO basics is needed to resolve this issue.
To mitigate the motion bias in VanillaDPO, we propose guided sampling [51] to generate structurally- similar pairs. However, this reduces the variations in comparison pairs, degrading the DPO perfor- mance. We note that image-to-vidde generation with the same conditioning image is another way to retain similar structure between video pairs. In addition, it allows more variations in the generated videos, which may improve StructuralDPO performance.
Finally, our segment-level preference optimization method can be useful beyond the DPO framework. Our experiments show that even SOTA video reward models are bad at assessing all aspects of generated video, yet effective online RL training relies good reward feedback [45, 82]. An interesting direction is to leverage VLM to provide higher-quality dense reward feedback, and adapt RL methods to optimize such dense reward signals.
33

---

"A womaa dancing in gym. The woman is spinning around repeatedly'
Pre-trained
VanillaDPO
DenseDPO StructuraiDPe
'Fingers press into a shimmering slime bail
DenseDPO StructuralDPO VanillaDPO Pre-trained
skateboarder performs jumps"
DenseDPO StrudturalDPS IPP VanillaDPO Pre- traiiindd
Figure 10: Qualitative comparison with baselines. Pre-trained model often generates deformed human body or unnatural object composition. VanillaDPO fixes these artifacts, but with significantly reduced dynamics. StructuralDPO retains the dynamics. but generates oversaturated frames some artifacts. DenseDPO strikes the best balance over these dimensions. Please check out our project page for video results of baselines and methods.
34

---

Uncurated samples of GPT o3 predicted dense preference labels. Each sample consists of pair of structurally similar videos generated via guided sampling. Vidcos are sampled at 2 FPS. top bar means GPT prefers the first example, and a bottom green bar means GPT prefers the second example, otherwise it is a tie. We highlight some obvious artifacts with blue rectangles.
Figure 11:
35

---

E
Pytorch-style Pseudo Code for StructuralDPO and DenseDPO
StructuralDPO applies the same Flow-DPO objective as VanillaDPO, which is adopted from [46]:
def flow_dpo__ loss (model, ref_model, x_0, x_1, c, 1, beta):
model: Flow model that takes text embeddings and timestep as inputs and predicts velocity с
# ref red_mo Pre-trained model that is frozen
x_0: The first video in the pair, shape [B, т, C, H, W] the pair, shape [B, T, C, H, c: Text embedding of the prompt
x_1: The second video
# 1: Preference label, shape [B], each item is either +1 or -1 +1 means x_0 is better than x,1, -1 means the other way beta: DPO regularization hyper parn---aaameeee
returns: FPo loss value
# Add noise to videos Dogit_normal_sapple. shape (0])
noise torch. randnanddnn..mman 1ike (x_0) noise
noisy_x_0 (1- t) x_0
noisy_x_1 = (1 t) x_1 + t noise
Compute velocity prediction loss v_0_pred model (noisy_xx____0,, c, t)
v_i-pred model (nois_yoiy___11. c, t)
v_ref -_ref_0_pred ref_model mni_m 0, c, t) v_ref __pred ref_mode (noisy x_1, c, t) v_0_gt noise x_0
v_1.gt noise x_1
model_0_err ((v_0_pred v_0-gt) ** 2) mean (dim=[1, 2, 3, 4])
model_1_err ((v_1_pred v_1_gt) ** 2) mean (dim=(1, 2, 3, 4])
ref_0_err ((v_ref_0_pred v_0_gt) ** m)mmenndimm 2, 3, 4))
ref_i_err ((v_ref_1_pred v_1.gt) mean (dim=(1, 3, 41)
# Compute DPO loss diff_0 sodel_0_ers ref_0_en # Shape [B] diff_1 model_
ref_l_err Shape [B]
inside_tern -0.5 beta (diff_0 diff_1) mean ()
loss -1 * log
return loss
36

---

DPO extends the Flow-DPO loss to frame-level (or token-level for latent models):
def bttowddenee_depee loss (model, ref model,, x_0, x_1, c, 1, beta):
model: Flow model that takes text embeddings and timestep inputs and predicts velocity
ref_model: Pre-trained model that is frozen
x_0: The first video in the pair, shape [B, т, c, H, W]
# x_1: The second video in the pair, shape [B, T, C, H, W]
c: Text embedding of the prompt
1: Dense preference label, shape [B, T], can +1, 0, means x_0 is better than x_1, -1 means the other way 0 means a tie
# beta: DPO regularization parametrr
returns: FLow-DPO loss value a a*n
Add noise to videos t "irr_normal__aaaee
(x_0. (x_0.shapp) shape [0])
noise torch.randn randn like (x_0)
noisy_x_0 (1 noisy_x_1
x_0 noise x_
noise
# Compute velocity prediction loss v_0_pred - model (noiss_x__, t)
v_1-pred model (noisy_x_y, c, t)
ref (nn_modellnolsyy___ x__0,____,,00 c, t)
v_ref_1_pre ref_model (noisy_x_1) c, t)
v_0_gt noise x_0
v_1-gt noise x_1 model_0_err ((v_0_ v_0_gt) 2) mean (dim= [2, 3, 41)
model_1_err ((v_1_pred v_1_gt) ** 2) mean (dim [2, 3, 4))
ref_0_err = ((v_ref_O_pree v_0_gt) ** 2) mean (dim= [2, 3, 4))
ref_i_err ((v_ref_i_pred v_1-gt) 2) mean [2, 4])
Compute DPO loss
diff_0 model_0_err ref 2ef_0_err # Shape [B, T]
diff_1 model_1_ers ref_i_err # Shape [B, T] inside_tern -0.5 beta * 1 (diff_0 diff_1)
inside_tera. inside_term[1 != 0] Only take frames loss -1 log (igii
mean ()
return loss
37


Reviews:
[{"review_id": "5rSx4XV4lo", "reviewer": "Reviewer_RyCj", "summary": "The paper proposes DenseDPO, a method to improve video diffusion models by using segment-level preference labels instead of whole-video comparisons. It generates structurally similar video pairs and collects fine-grained human or VLM-based preferences, avoiding bias toward slow-motion clips. DenseDPO achieves better motion quality, matches other methods in visual fidelity and text alignment, and is more data-efficient. Therefore, it needs fewer labeled videos to perform well.", "questions": "1. Can you provide comparisons with more methods? The experiments can be done using a small data scale.\n2. Can you provide the results that directly utilize the VLM rewards to train the generation model?", "limitations": "1. The upper bound of the method is limited by the data and the quality of annotations, which means the method can’t be easily scaled.\n2. Lack of comparisons with many methods", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Segment-level preferences capture more accurate feedback than global labels.\n2. Using structurally aligned video pairs avoids overfitting to static, artifact-free clips.\n3. It achieves strong results with only one-third of the labeled data.\n\nWeaknesses：\n\n1. The method is more likely to find different reward aspects for DPO. If so, why not directly use the reward model for online RL?\n2. Missing comparisons with many DPO-like methods, such as D3PO, SePPO, DPOK, etc.", "was_revised": true, "final_justification": "The explanation solve the problem I mentioned in the weaknesses.", "rebuttal": "We thank the reviewer for acknowledging the benefit of using structurally similar videos, segment-level preference labels, and the data-efficiency of DenseDPO. We address the concerns below:\n\n### **W1. Directly use the reward model for online RL.**\n\nA: We kindly remind the reviewer that the major goal of this work is to improve the original DPO framework, which relies on pre-collected human preference pairs [46, 53, 62]. Our method demonstrates great performance gain by improving the offline human labeling process. Moreover, in the next question, we will show that DenseDPO outperforms DPOK [15], an online RL method.\n\nCompared to online RL training, DPO bypasses the need for a reward model thus is simple to implement. It also achieves stable improvements and has been applied in several industry lab developed video diffusion models [1, 2]. In addition, existing video reward models are only trained in pixel space. This requires decoding the latent to videos to compute reward, which is both computation and memory intensive. Overall, while RL holds strong potential, the topic of RL for diffusion has only recently been explored [3, 4] and is concurrent to our work.\n\nWe also want to point out that effective online RL training relies on good reward models. Yet, even SOTA video reward models are bad at assessing all aspects of a generated video (please see more discussions in the response to Q2). We believe applying online RL for video diffusion models requires developing better reward models [4], which is beyond the scope of this work.\n\n[1] Chen, Guibin, et al. \"SkyReels-V2: Infinite-length film generative model.\" arXiv 2025.\n\n[2] Ma, Guoqing, et al. \"Step-Video-T2V Technical Report: The practice, challenges, and future of video foundation model.\" arXiv 2025.\n\n[3] Liu, Jie, et al. \"Flow-GRPO: Training flow matching models via online rl.\" arXiv 2025.\n\n[4] Xue, Zeyue, et al. \"DanceGRPO: Unleashing GRPO on Visual Generation.\" arXiv 2025.\n\n\n### **W2 & Q1 & L2. Comparison with more baselines such as D3PO, SePPO, DPOK.**\n\nA: Thank you for bringing up these methods. To provide comparisons, we adapted D3PO’s public code to our video setting. For DPOK, we also modified their open-source code and leveraged VisionReward [61] to provide online feedback. Since no training code is available for SePPO, we re-implement it based on its paper. All methods are trained on the same data as VanillaDPO (the human preference pairs and text prompts) with LoRA and the hyper-parameters listed in their original papers.\n\nWe report the results on VideoJAM-bench in the table and summarize them below:\n- D3PO achieves similar results as VanillaDPO which is based on Diffusion-DPO. This is in line with prior works [1, 2] that also report similar performances between the two methods.\n- DPOK, an online RL method, slightly improves the visual quality and temporal consistency compared to the pre-trained model, yet significantly degrades the dynamic degree. This is because VisionReward is biased towards per-frame visual quality instead of temporal motion. Thus, tuning with its noisy reward leads to static motion. Please see more discussions about the reward model in the next question.\n- SePPO [1] achieves a better dynamic degree than VanillaDPO, yet performs worse in visual quality. Instead of using offline generated losing samples, SePPO generates online samples with a reference model, and designs a filtering method (AAF) to assess its quality. This can be viewed as a combination of DPO and SFT. However, we noticed that AFF is based on model denoising loss, which is sometimes unreliable. In those cases, the model is optimized on low-quality videos, leading to worse visual quality.\n\nOverall, DenseDPO outperforms D3PO, DPOK, and SePPO across all metrics, showing the effectiveness of our approach. We will add these results and discussions to the paper.\n\n| Method | Aesthetic Quality | Imaging Quality | Subject Consistency | Background Consistency | Motion Smoothness | Dynamic Degree | Text Alignment | Visual Quality | Temporal Consistency | Dynamic Degree |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Pretrained | 54.65 | 55.85 | 88.29 | 91.50 | 92.40 | 84.16 | 0.770 | 0.192 | 0.354 | **0.680** |\n| D3PO | 56.15 | 58.03 | 88.93 | 92.23 | 92.78 | 82.53 | 0.833 | 0.322 | 0.482 | 0.602 |\n| DPOK | 54.99 | 56.28 | 89.14 | 92.36 | 92.95 | 78.65 | 0.795 | 0.337 | 0.518 | 0.457 |\n| SePPO | 55.83 | 57.42 | 89.93 | 92.65 | 92.93 | *82.85* | 0.841 | 0.326 | 0.554 | 0.587 |\n| VanillaDPO | **57.25** | *60.38* | *91.21* | **93.94** | *93.43* | 80.25 | **0.867** | *0.371* | **0.636** | 0.535 |\n|**DenseDPO**| *56.99* | **60.92** | **91.54** | *93.84* | **93.56** | **85.38** | *0.863* | **0.376** | *0.632* | *0.680* |\n\n[1] Zhang, Daoan, et al. \"SePPO: Semi-policy preference optimization for diffusion alignment.\" arXiv 2024.\n\n[2] Liang, Zhanhao, et al. \"Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization.\" CVPR 2025.\n\n\n### **Q2. Results on directly utilizing VLM rewards to train the generation model.**\n\nA: VLM rewards have been used in the literature to improve generation models in different ways: 1) online RL training with VLM rewards, 2) DPO training with preferences derived from VLM rewards, and 3) REFL-style [60] direct gradient backpropagation to maximize the VLM reward. Below, we discuss each direction:\n\n**Online RL training with VLM rewards**: as discussed above, we implemented an online RL algorithm DPOK [15] based on VisionReward [61], and DenseDPO outperforms this baseline.\n\n**DPO with VLM rewards**: we refer the reviewer to Sec.4.3 of the main paper and Appendix C.2 for detailed results. We test SOTA video rewards models fine-tuned from VLMs such as VideoReward [34] and VisionReward [61], as well as advanced commercial VLMs such as GPT o3. We summarize some key findings below:\n- Tab.3 (a) of the main paper shows that even SOTA VLMs achieve low accuracy in predicting human preferences on 5s videos. In contrast, predicting our proposed segment-level preferences leads to higher accuracy.\n- Appendix Tab.1 reveals that VLM-based reward models might be biased towards video content rather than temporal motion, which may explain their low preference accuracy.\n- Tab.3 (b) of the main paper shows that StructuralDPO with VLM rewards achieves moderate improvements compared to the pre-trained model. Applying segment-level VLM rewards further improves the results, which only slightly underperforms DenseDPO with human labels.\n\n**Direct VLM reward maximization**: we tried REFL in our preliminary experiments with HPSv2 [57] and VisionReward [61] as the target reward model. However, we observed severe reward hacking, e.g., the video frames become oversaturated and the temporal motion gets static, which is clearly worse than applying DPO. It also requires huge GPU memory and computation as we need to decode latent tokens to raw pixels to apply these VLMs and backpropagate gradients through them. We will include these results in the paper.\n\n\n### **L1. The scalability of the proposed method.**\n\nA: While DPO requires pre-collected human preferences, training a reward model for online RL also requires human labels (e.g., VideoReward uses 182k video annotations, and VisionReward uses around 90k video labels). In fact, a post-training stage with high-quality human-labeled data has been a common practice in both diffusion models [1, 2] and LLMs [3, 4]. In addition, we kindly note that DenseDPO is already more data-efficient than DPO baselines.\n\nIn the paper, we also explored VLM labels for scalable DPO training. We conducted additional experiments comparing DenseDPO trained with different amounts of GPT o3 labels on VideoJAM-bench and show the results below.\n\n| Method | Aesthetic Quality | Imaging Quality | Subject Consistency | Background Consistency | Motion Smoothness | Dynamic Degree | Text Alignment | Visual Quality | Temporal Consistency | Dynamic Degree |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Pretrained | 54.65 | 55.85 | 88.29 | 91.50 | 92.40 | 84.16 | 0.770 | 0.192 | 0.354 | 0.680 |\n| 10k GPT labels | 55.21 | 56.83 | 88.31 | 91.45 | 92.38 | 84.20 | 0.782 | 0.247 | 0.360 | 0.668 |\n| 35k GPT labels | 55.89 | 58.75 | 89.68 | 92.25 | 92.70 | 84.89 | 0.817 | 0.316 | 0.498 | 0.670 |\n| 55k GPT labels | 56.23 | 60.15 | 90.75 | 93.01 | 92.99 | 85.21 | 0.842 | 0.368 | 0.598 | 0.672 |\n\nAs the amount of VLM labels grows, we see consistent improvement in all metrics. This trend clearly shows that our method scales well with the quantity of VLM labels. Moreover, VLM itself is an actively evolving field, and our method will benefit from its progress. We would also like to note that reviewer jRmA recognized these results as “removes reliance on costly human labels and lays a solid foundation for scaling post-training temporal dense alignment”. We will add the discussion on the data and the quality of annotations to the Limitation section of the paper.\n\n[1] ByteDance. \"Seaweed-7B: Cost-effective training of video generation foundation model.\" arXiv 2025.\n\n[2] StepFun. \"Step-Video-T2V Technical Report: The practice, challenges, and future of video foundation model.\" arXiv 2025.\n\n[3] DeepSeek. \"DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.\" arXiv 2025.\n\n[4] Meta. \"The Llama 4 herd\". Meta 2025."}, {"review_id": "mBQ5EZvDoz", "reviewer": "Reviewer_BRHs", "summary": "The paper proposes DenseDPO, which provides fine-grained segment-level annotations for DPO.\nGround-truth videos are used to construct video pairs for training.\nThe motion generation results are improved, while text alignment, visual quality, and temporal consistency are maintained.\nThe authors leverage a VLM to automatically score the video pairs and show that the results are comparable to human annotations.", "questions": "see weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths   \nThe writing is generally good, and the experiments are comprehensive.   \nThe visual results demonstrate large motion.   \n\nWeaknesses  \nThe proposed method, DenseDPO, has limited technical novelty. Each component offers only minor innovation.\nThe results are not strong: in some cases, the performance is worse than vanilla DPO, and the degree of motion dynamics is lower than that of the pretrained baseline.   \nThere is a lack of ablation studies on the use of ground-truth data.    \nThe results on motion smoothness and motion dynamics in MotionBench are worse than those of vanilla DPO and SFT, making it difficult to support the paper’s claim of improved motion generation.", "was_revised": true, "final_justification": "I have gone through the author's rebuttal and other reviewers' comments. The rebuttal and explainations have almost address my concerns. However, I still have concerns on the method's novelty. Considering other reviewers' comments I decide to raise my score.", "rebuttal": "We are glad that the reviewer found our writing generally good, our experiments comprehensive, and the visual results showing large motion. We address the concerns below:\n\n### **W1 & W3. Results on motion smoothness and motion dynamics compared to baselines.**\n\nA: We would like to point out that motion smoothness or motion dynamics alone does not represent the video quality. For example, an entirely static video will have perfect motion smoothness, while a video of completely random noise will have high motion dynamics. For video generators, there is an inherent trade-off between the two [35, 61]. **Please check out the video results in our supplementary material for a clear comparison**. Specifically,\n- VanillaDPO achieves high motion smoothness because it generates **almost static videos**. This can be verified by its low motion dynamics score. In addition, DenseDPO actually achieves a higher motion smoothness than VanillaDPO on VideoJAM-bench, and only slightly underperforms on MotionBench.\n- Pre-trained and SFT models achieve high motion dynamics but generate videos with **severe temporal inconsistency**, such as deformed limbs and objects. This can be verified by their low motion smoothness score. Besides, DenseDPO also outperforms baselines on VideoJAM-bench, and the gap of motion dynamics is marginal on MotionBench.\n- Overall, DenseDPO is the only method that maintains the motion dynamics of the pre-trained model, while significantly improving the motion smoothness. We would also like to note that other reviewers find our results “strong” while using “fewer labeled videos” (RyCj), “validate the effectiveness of each component” (jRmA), and “compelling across multiple challenging benchmarks” (PkkZ).\n\nWe also note that all automatic metrics in video generation have limitations [1, 2], and **human evaluations are considered more reliable**:\n- Fig.5 right of the main paper shows that DenseDPO achieves slightly higher motion smoothness (TC) with VanillaDPO, with significantly better motion dynamics (DD).\n- Fig.3 right of the Appendix shows that DenseDPO achieves slightly better motion dynamics (DD) as the pre-trained model, while improving motion smoothness (TC) by a sizable margin.\n\n[1] ByteDance. \"Seaweed-7B: Cost-effective training of video generation foundation model.\" arXiv 2025.\n\n[2] Zeng, Ailing, et al. \"The dawn of video generation: Preliminary explorations with sora-like models.\" arXiv 2024.\n\n\n### **W1. Novelty of DenseDPO.**\n\nA: We agree that both DPO and guided generation were known individually in the diffusion literature. However, this paper offers novel insights in **designing a better DPO method for video diffusion models**, which are not obvious from previous work:\n- We are the first work that explicitly points out the **motion bias** in VanillaDPO, and adapts guided generation to solve this issue with StructuralDPO.\n- Instead of naively borrowing the technique, we provide **mathematical derivation** to show why StructuralDPO is sub-optimal (see Appendix B). This aligns with previous observations in LLM DPO papers [40, 47] and serves as a missing piece in diffusion DPO study.\n- With this insight, we further propose to use **segment-level preference labels** to provide dense learning signals, achieving better generation results and data efficiency compared to baselines. The use of dense preferences is never explored in prior video DPO papers.\n- Besides the DenseDPO framework, we also want to highlight our results using **VLM labels**. It shows a promising way to scale up automatic preference learning with off-the-shelf VLMs.\n\nOverall, these design choices and results were neither known nor obvious for video DPO before our work, and we view them as our major contributions driving the performance gains. We believe our work has the potential to impact both the video generation and diffusion post-training communities.\n\n\n### **W2. Ablation on the use of ground-truth data.**\n\nA: We are not entirely sure what the reviewer means by “use of ground-truth data”. Based on context, we assume the reviewer is asking about ablating the ground-truth videos used in guided video generation.\n\nIn DenseDPO, we randomly sample 10k videos from our 55k high-quality dataset to serve as guidance videos. Here, we randomly sample another 10k videos without overlap with the previously selected ones, use them to generate paired videos, and conduct a DenseDPO training. We report the result on VideoJAM-bench below. Both runs achieve similar results, proving that our method is robust to the selection of ground-truth data as long as they are high-quality videos.\n\n| Method | Aesthetic Quality | Imaging Quality | Subject Consistency | Background Consistency | Motion Smoothness | Dynamic Degree | Text Alignment | Visual Quality | Temporal Consistency | Dynamic Degree |\n|-|-|-|-|-|-|-|-|-|-|-|\n| DenseDPO   (old GT data) | 56.99 | 60.92 | 91.54 | 93.84 | 93.56 | 85.38 | 0.863 | 0.376 | 0.632 | 0.680 |\n| DenseDPO   (new GT data) | 56.94 | 60.91 | 91.48 | 93.76 | 93.57 | 85.39 | 0.868 | 0.377 | 0.636 | 0.678 |\n\nPlease let us know if this is the correct interpretation. We are happy to provide extra ablations or results if needed. We will add more discussions and the results above to the paper."}, {"review_id": "LXhuDgoFSp", "reviewer": "Reviewer_jRmA", "summary": "This work proposes DenseDPO, which first creates aligned video pairs with different local motion details and then labels segment-level preferences to yield a dense learning signal for physics-realistic motion reinforcement learning. It also shows that DenseDPO trained with VLMs for segment-level preferences achieves performance comparable to using human labels. The experiments are thorough and validate the effectiveness of each component.", "questions": "See Weaknes", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. This work first proposes StructuralDPO, which generates each pair by denoising partially noised real videos under different guidance strengths. This produces video pairs with shared structural similarity but differing in local details—unlike vanilla DPO’s independent noise samples.\n\n2. By collecting and optimizing with segment-level preference data, DenseDPO achieves clear gains: qualitative examples show markedly improved motion fidelity, and quantitative results on VBench metrics outperform both SFT and vanilla DPO. The appendix provides a mathematical derivation of the guided sampling scheme, explaining why StructuralDPO alone falls short and why DenseDPO prevails.\n\n3. The paper demonstrates that off-the-shelf VLMs (notably GPT) can predict segment-level preferences nearly as well as humans. This removes reliance on costly human labels and lays a solid foundation for scaling post-training temporal dense alignment.\n\nWeaknesses\n\n1. Unclear base model and high post-training cost. The paper omits details on the base diffusion model’s architecture and size. Post-training on 50K videos requires approximately 16 hours on 64 GPUs, imposing substantial computational overhead. Moreover, with code, data, and pretrained checkpoints unreleased, reproducing or evaluating these results is very challenging.\n\n2. Limited novelty in the RL formulation.\nMost gains stem from segment-level preference labels rather than any new reinforcement-learning algorithm. The algorithmic contribution over existing DPO variants is therefore modest.\n\n3. Unexplained VLM label noise compensation.\nGPT-3.5 labels achieve only ~70 % segment-level accuracy yet match human-label performance in DPO fine-tuning. The manuscript does not analyze why noisy but abundant VLM labels suffice—for example, whether sheer data volume outweighs reduced label quality.", "was_revised": true, "final_justification": "My concerns have been well resolved, so I intend to increase my score to Accept. Also, I think the authors have done enough experiments during the rebuttal period, especially comparisons to other RL methods and scaling the VLM reward. I appreciate this feedback.", "rebuttal": "We thank the reviewer for the detailed feedback and the positive assessment of our work. We are glad that the reviewer accurately identifies the advantages of StructuralDPO and DenseDPO, as well as highlighting our mathematical derivation and VLM results. We address the concerns below:\n\n### **W1. Details on models & Training cost & Reproducibility of the work.**\n\nA: We kindly refer the reviewer to Appendix A.2 (the “Pre-trained model” paragraph) for details on the base diffusion model’s architecture. Overall, it is a DiT model similar to Open-Sora [19]. The size of the model is around 10B.\n\nRegarding the training cost, one can easily adapt our method to train on 8 GPUs with gradient accumulation. We also note that all baselines are trained with a similar amount of compute, ensuring a fair comparison.\n\nRegarding code release, we are working on releasing a reference implementation of StructuralDPO and DenseDPO. We also note that we have provided all implementation details in Appendix A such as the GPT prompts to select high-quality data, and the pseudo code in Appendix E, which is only a few lines different from Diffusion-DPO [53] and should be easy to re-implement and try out.\n\n\n### **W2. Novelty of our algorithm.**\n\nA: We agree that the DPO loss we use is similar to the one proposed by Diffusion-DPO [53]. Yet, this paper focuses on another important component in the DPO framework: how to create better human preference labels. We believe it offers novel insights in designing a better DPO method for video diffusion models, which are not obvious from previous work:\n- We are the first work that explicitly points out the **motion bias** in VanillaDPO, and adapts guided paired video generation to solve this issue in StructuralDPO.\n- As the reviewer correctly pointed out, we provide **mathematical derivation** to show why StructuralDPO is sub-optimal. This aligns with previous observations in LLM DPO papers [40, 47] and serves as a missing piece in diffusion DPO studies.\n- With this insight, we further propose to use **segment-level preference labels** to provide dense learning signals, achieving better generation results and data efficiency compared to baselines. The use of segment-level preference optimization has never been explored in prior diffusion DPO papers.\n- Please note that labeling dense preferences is only possible with structurally similar videos, as it requires the video pairs to be temporally aligned. It also opens up new possibilities to leverage off-the-shelf VLMs for preference learning.\n\nOverall, these design choices and results were neither known nor obvious for video DPO before our work, and we view them as our major contributions driving the performance gains. We believe our work has the potential to impact both the video generation and diffusion post-training communities.\n\n\n### **W3. VLM label noise compensation.**\n\nA: Thank you for this great question. We conducted additional experiments to compare DenseDPO trained with different amounts of GPT o3 labels on VideoJAM-bench. The results are shown in the table below.\n\n| Method | Aesthetic Quality | Imaging Quality | Subject Consistency | Background Consistency | Motion Smoothness | Dynamic Degree | Text Alignment | Visual Quality | Temporal Consistency | Dynamic Degree |\n|-|-|-|-|-|-|-|-|-|-|-|\n| Pretrained | 54.65 | 55.85 | 88.29 | 91.50 | 92.40 | 84.16 | 0.770 | 0.192 | 0.354 | 0.680 |\n| 10k GPT labels | 55.21 | 56.83 | 88.31 | 91.45 | 92.38 | 84.20 | 0.782 | 0.247 | 0.360 | 0.668 |\n| 35k GPT labels | 55.89 | 58.75 | 89.68 | 92.25 | 92.70 | 84.89 | 0.817 | 0.316 | 0.498 | 0.670 |\n| 55k GPT labels | 56.23 | 60.15 | 90.75 | 93.01 | 92.99 | 85.21 | 0.842 | 0.368 | 0.598 | 0.672 |\n\nSince VLM labels are noisy, DenseDPO with 10k GPT labels achieves only marginal improvement compared to the pre-trained model. Nevertheless, **as the amount of VLM labels grows, we see consistent improvement in all metrics**, closing the gap with human labels. This trend reveals that the quantity of VLM labels indeed compensate for their quality. We view this as a promising future direction to further scale up automatic preference learning with more data and better off-the-shelf VLMs. We will add these results to the paper."}, {"review_id": "EAtQnAyEFL", "reviewer": "Reviewer_PkkZ", "summary": "This paper presents DenseDPO, a framework designed to improve preference optimization in text-to-video diffusion models. Building upon prior work on Direct Preference Optimization (DPO), the authors identify two key limitations in existing video DPO pipelines: (1) a bias toward low-motion video clips due to independent noise sampling, and (2) coarse, whole-video preference labels. DenseDPO addresses these by: (1) Using partially noised real videos to generate structurally similar pairs (StructuralDPO), mitigating motion bias. (2) Introducing segment-level (fine-grained) preference annotations to create richer learning signals. Extensive quantitative evaluations (VideoJAM-bench, MotionBench) and human studies show DenseDPO outperforms VanillaDPO and StructuralDPO in generating high-quality, dynamic, and text-aligned videos.", "questions": "See weakness.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1) Clear problem formulation: The paper clearly identifies and investigates the motion bias issue inherent in vanilla DPO setups for post-training video generative models. This is a well-motivated and important problem for post-train the video generative model.\n2) The method demonstrates compelling results across multiple challenging benchmarks, complemented by qualitative examples.\n\nWeakness:\n\n1) Reliance on real-video-guided generation: DenseDPO uses partially noised real videos to create structurally similar pairs, which may reduce motion diversity. This raises the question: could this approach limit the ability to select or generate more diverse and high-quality motion?\n2) Terminological ambiguity: The introduction of both StructuralDPO and DenseDPO could be confusing. Since DenseDPO builds upon StructuralDPO by adding segment-level labels, the distinction between the two might be better integrated or clarified to reduce redundancy and improve conceptual coherence.\n3) Limitations of image-based VLMs: The use of Vision-Language Models (GPT-o3), which are primarily trained on static images, raises concerns about their effectiveness in evaluating motion. It’s unclear whether their judgments reflect true motion quality or simply frame-level visual quality.", "was_revised": true, "final_justification": "The authors' response solved my concern. I tend to increase my score and lean to borderline acceptance.", "rebuttal": "We thank the reviewer for the constructive feedback. We are glad that the reviewer finds our paper well-motivated and our results compelling across multiple challenging benchmarks. We address the concerns below:\n\n### **W1. Reliance on real-video-guided generation and its effect on learning high-quality motion.**\n\nA: This is an interesting question. We agree that in theory, VanillaDPO independently samples two videos with distinct motion, which should ideally give more diverse learning signals. However, on one hand, human labeling suffers from a motion bias that favors high-quality yet slow-motion videos. As a result, VanillaDPO often biases the model towards generating static motion, further reducing the motion diversity. On the other hand, our training videos are filtered to contain high-quality motion. It is enough to just learn from the distribution of motion in them.\n\nIn DenseDPO, we intentionally choose to hold the motion dynamics while improving visual quality and temporal consistency. We believe the significantly improved motion smoothness of videos generated by our model proves that it learns high-quality motion in training videos.\n\nAt a higher level, recently there is still debate about whether post-training teaches models new knowledge, or just amplifies the behaviors learned in pre-training [1, 2]. For DPO methods, we can view it as re-allocating the model’s distribution density to the modes we, as humans, want to see. For example, if before DPO, the model can generate high-quality motion only once in 100 samples, then after DPO, high-quality motion will appear much more frequently. The main goal of DPO is to incentivize high-quality motion presented in training videos. If a type of motion is not in training data and thus not learned by the model in pre-training, it is likely impossible to learn it in post-training either. Therefore, relying on high-quality real videos does not hurt the general DPO objective.\n\n[1] Yue, Yang, et al. \"Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?.\" arXiv 2025.\n\n[2] Zhao, Rosie, et al. \"Echo chamber: RL post-training amplifies behaviors learned in pretraining.\" arXiv 2025.\n\n\n### **W2. Terminological ambiguity between StructuralDPO and DenseDPO.**\n\nA: Thanks for this suggestion. As the reviewer correctly pointed out, DenseDPO builds upon StructuralDPO by adding segment-level labels, which is only possible with structurally similar videos since they are temporally aligned. We can also rename StructuralDPO to a variant of DenseDPO whose segment length $s$ equals to the video length. We will add more discussions and clarifications to the main text of the paper. Please let us know if you have better suggestions on how to make their connections and distinctions clearer!\n\n\n### **W3. Limitations of GPT-o3 in evaluating videos.**\n\nA: We kindly note that **we did not use GPT-o3 to measure quantitative results**. Instead, we use VBench [22], a benchmark designed specifically for videos, and VisionReward [61], a video language model fine-tuned to assess video quality, for automatic metrics. They are widely adopted in the literature of video generation and reflect both visual and motion quality of videos. In addition, we also provide user studies that show highly consistent results with the quantitative metrics. This validates the performance gain of DenseDPO.\n\nRegarding GPT-o3’s ability to evaluate videos, we believe our VLM label experiments in Sec.4.3 provide some insights. For ease of discussion, we summarize some key results in the table below. First of all, we agree with the reviewer that since GPT-o3 is primarily trained on static images, it fails to assess long-range motion in 5s videos. As shown in the “Preference Accuracy” column, GPT-o3 only achieves 53.45\\% preference prediction accuracy when evaluating 5s videos. Yet, we observe that GPT-o3 works fairly well in assessing short 1s video segments since they only contain modest motion. When tasked with our proposed segment-level preference labeling, GPT-o3 achieves 70.59\\% accuracy, even surpassing VisionReward, a video reward model specifically fine-tuned for this task. In addition, we also tried running DPO with VLM predicted preference labels. DenseDPO with GPT-o3 segment-level labels clearly improves several metrics on our video benchmark, and outperforms StructuralDPO with VisionReward. Overall, this proves the effectiveness of our method in adapting off-the-shelf VLMs primarily trained on images to improve video assessment and generation. This is one of the advantages of DenseDPO compared to previous DPO methods.\n\n\n| VLM | Preference Accuracy | Text Alignment | Visual Quality | Temporal Consistency | Dynamic Degree |\n|-|-|-|-|-|-|\n| Pre-trained | - | 0.770 | 0.192 | 0.354 | **0.680** |\n| VisionReward + StructuralDPO | *62.11\\%* | *0.785* | *0.347* | *0.521* | 0.622 |\n| GPT-o3 + StructuralDPO | 53.45\\% | 0.759 | 0.284 | 0.506 | 0.621 |\n| **GPT-o3 Segment + DenseDPO** | **70.59\\%** | **0.842** | **0.368** | **0.598** | *0.672* |"}]

Author rebuttal:

Meta-review: This draft introduces DenseDPO, which uses aligned video pairs and segment-level preferences to improve video diffusion models. The method reduces motion bias, yields stronger motion quality, and remains data-efficient, with promising results even using VLM-based labels. While questions on novelty and reliance on guided videos were raised, AC and Rs agree that the rebuttal provided solid comparisons and clarifications. Overall, the work is well-motivated, convincingly validated, and has a clear impact. AC recommend accept.
Decision: Accept (spotlight)