T2V-Turbo-v2: Enhancing Video Model Post-Training through Data, Reward, and Conditional Guidance Design
Authors: Jiachen Li
Qian Long
Jian Zheng
Xiaofeng Gao
Robinson Piramuthu
Wenhu Chen
William Yang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-video generation
diffusion model
consistency model
Primary area: generative models
TL;DR: We focus on enhancing a video generation model during the post-training phase by integrating supervision signals from high-quality training data, reward feedback, and conditional guidance into the consistency distillation process.

Abstract: In this paper, we focus on enhancing a diffusion-based text-to-video (T2V) model during the post-training phase by distilling a highly capable consistency model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2, introduces a significant advancement by integrating various supervision signals, including high-quality training data, reward model feedback, and conditional guidance, into the consistency distillation process. Through comprehensive ablation studies, we highlight the crucial importance of tailoring datasets to specific learning objectives and the effectiveness of learning from diverse reward models for enhancing both the visual quality and text-video alignment. Additionally, we highlight the vast design space of conditional guidance strategies, which centers on designing an effective energy function to augment the teacher ODE solver. We demonstrate the potential of this approach by extracting motion guidance from the training datasets and incorporating it into the ODE solver, showcasing its effectiveness in improving the motion quality of the generated videos with the improved motion-related metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2 establishes a new state-of-the-art result on VBench, **with a Total score of 85.13**, surpassing proprietary systems such as Gen-3 and Kling.

Published as a conference paper at ICLR 2025
T2V-TURBO-V2. ENHANCING VIDEO GENERATION MODEL POST-TRAINING THROUGH DATA, REWARD, AND CONDITIONAL GUIDANCE DESIGN
Jiachen Li', Qian Long", Jian Zheng"; Xiaofeng Gao"; Robinson Piramuthu", Wenhu Chen William Yang Wang
UC Santa Barbara, PUC Los Angeles, Amazon AGI, University of Waterloo
(jiachen.li, william]@ce @ccs ucsb. edu, "LongqianBucle" edu,
[[nzhengji, gxiaofen, robinpir]eamazonm com *nerhuchhnngwaterrloo.c
Project Page: https:
Videos: click to play
Pikachu snowboarding
Adobe Acrobat
T2V-Turbo
#2V Poy-turboooooo -22
A feddy bear is swimming in the ocean
T2V-Turbo
12V D2V-Toorbo--g v2
Figure 1: By meticulously designing and selecting the training datasets, reward models, and guid- ance, T2V-Turbo-v/ generates videos that visually appealing, semantically better aligned, and more dynamic compared to T2V-Turbo (Li et al. 2024b).
ABSTRACT
In this paper, we focus on enhancing diffusion-based text-to-video (T2V) model during the post-training phase by distilling highly capable consistency model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2, intro- duces significant advancement by integrating various supervision signals, in- cluding high-quality training data, reward model feedback, and conditional guid- ance, into the consistency distillation process. Through comprehensive ablation studies, we highlight the crucial importance of tailoring datasets to specific learn- ing objectives and the effectiveness of learning from diverse reward models for enhancing both the visual quality and text-video alignment. Additionally, we highlight the vast design space of conditional guidance strategies, which centers designing effective energy function to augment the teacher ODE solver. We demonstrate the potential of this approach by extracting motion guidance from
"This work does relate the author's position Amazon.

---

Published as a conference paper at ICLR 2025
the training datasets and incorporating it into the ODE solver, showcasing its ef- fectiveness in improving the motion quality of the generated videos with the im- proved motion-related metrics from VBench and T2V-CompBench. Empirically, our
establishes a new state-of-the-ar result on VBench, with a Total score of 85.13, surpassing proprietary systems such Gen-3 and Kling.
INTRODUCTION
cnffusion-basede (Sohl-Dickstein et al. 2015 Ho et al. 2020) neural video synthesis has been ad- vancing at an unprecedented pace, giving rise to cutting-egge text-to-video (T2V) systems like Sora (Brooks et al. 2024), Kling (Kuaishou 2024). DreamMachine (Lumal 20244 and Gen-3 (Gen- 3,20224 These models are capable of generating high-quality videos with detailed motion dynam- Ics. However, the majority of these systems are proprietary, pretraining them from scratch requires significant computational resources and access to extensive, human-curated video datasets, which not readily accessible to the academic community. Consequently, notable performance dispar- ity has emerged between proprietary models and currently available open-source alternatives (Chen) al. 2024a Guo al., 2023 Wang al. 2023a c -penn-Sort 2024 Lab & etc., 2024). which are trained on datasets with varying video quality, c.g., Web WebVid-IOM [Bann et al. (021).
Efforts have been directed toward bridging the performance gap. On the one hand, various video datasets (Tan et al. 2024 Nan et al. 20244 Chen et al., [024b] Ju et al. (024) with higher visual quality and detailed captions are released the public domain. On the other hand, several methods have been proposed to enhance the sample quality of pretrained T2V models during the post-training phase. For example, InstructVideo (Yuan al. 2023) and Vader (Prabhudesai et al. 2024) pro- pose aligning pretrained T2V models by backpropagating gradients from image-text reward models (RMs). More recently, Li et al. 2024b) proposes T2V-Turbo, achieving fast and high-quality video generation by incorporating feedback from mixture of RMs into the consistency distillation (CD) process (Song et al. 2023 Luo et al. 2023). ). However, these methods still employ the Web Vid data to train the model and thus solely rely the reward feedback improve the generation quality. In addition, training-free methods (Feng el al.] [2023] Mo et al.) 2024 Xie et al., 2023) have emerged that seek to enhance generation quality by introducing conditional guidance during inference. For example, MotionClone (Ling et al. 2024 extracts the temporal attention matrix from a reference video as the motion prior and uses formulate an energy function that guides the T2V model's sampling process, achieving impressive results. However, this approach requires calculating the additional guidance term at each sampling step, imposing significant computational overhead, and requiring access to a high-quality reference video.
In this paper, we aim advance the post-training of the T2V model by incorporating supervision signals from various sources. To this end, we introduce r2V-Turbo-vee integrating various su- pervision signals from high-quality video datasets, reward feedback, and conditional guidance into the CD process. Additionally, which can be boiled down to designing an energy function to augment the teacher ODE solver to- gether with the classifier-free guidance (CFG) (Ho & Salimans| 2021). In this paper, we empirically showcase the potential of this approach by leveraging the motion guidance from MotionClone to formulate our energy function. Regarding the reference video. our key insight is that, given pair of video and caption, the video itself naturally serves as the ideal reference when the generation prompt is its own caption. Moreover, we remedy the substantial computational cost of calculating the conditional guidance by dedicating a data preprocessing phase before training the consistency model. implementation-wiae wise, we improve upon T2V-Turbo by eliminating the target network used to calculate the distillation target without experiencing training instability. The saved memory from the preprocessing phase and removing the target network allows us to perform full model training. whereas T2r-o can only train its model with LoRA (Hu al. 2021).
highlight the vast design space of conditional guidance strategy,
We design comprehensive experiments to investigate how different combinations of training datasets, RMs, and conditional guidance impact the performance of our 22V-Turbo-v2 Our design of RMs is more rigorous compared to T2V-Turbo, leveraging feedback from vision-language foundation models, e.g., CLIP (Radford et al. 2021 and the second stage model of Intern Video2 (Intern /Wang et al. 2024 2023d). after observing that the baseline VideoLCM (Wang et al.) 2023b) (VCM) fails to align the text and its generated every dataset variants. We also empirically
2

---

Published as a conference paper at ICLR 2025
show the benefits of learning from diverse RMs. Moreover, identified that while training datasets with higher visual quality and dense video captions benefits visual quality, the dense cap- tions prevent the model from fully optimizing the benefits from the reward feedback, particularly due to the limited context length of existing RMs. To combat this issue, we carefully curate the training data for different learning objectives, leading substantial performance gain. These experiments provide solid empirical evidence to motivate future research to develop long-context RMs. Fi- nally, verify the motion quality improvement made by the incorporation of motion guidance, leverage various metrics from both VBench (Huang et al. 2024) and T2V- I2V-CompBench (Sun et al., 2024) to access the motion quality of different methods.
We distill our D2V-Turbo-v2 from VideoCrafter2 (Chen et al. 2024a). Empirically, we evaluate the performance its 16-step generation on the VBench (Huang et al.] 2024), both with and without augmenting the ODE solver with the motion guidance. Remarkably, both variants outperform all existing baselines on the VBench leaderboard. The variant that incorporates motion guidance sets new SOTA on VBench, achieving Total Score of 85.13, surpassing even proprictary systems such as Gen-3 (Gen-3 2024) and Kling (Kuaishou 2024).
Our contributions are threefold.
A rigorous and thorough empirical investigation into the effects of training data, RMs, and conditional guidance design on the post-training phase of the T2V model, shedding light future T2V post-training research.
Establish new SOTA Total Score VBench, outperforming proprictary systems, includ- ing Gen-3 and Kling.
Highlight the vast design space energy functiinn to augment the ODE solver, demon- strating its potential by extracting motion priors from training videos to enhance T2V model training. To the best of our knowledge, this the first work to introduce this approach.
2 PRELIMINARY
Diffusion Sampling The sampling process from latent video diffusion model (Ho et al. 2022b) can be treated as solving an empirical probability flow ordinary differential equation (PF-ODE) (Song et al. 2020b). The sampling process executes in reversed-time order, starts from standard Gaus- sian noise žy, and returns a clean latent Zg.
dz: = [ (t) z (t dt, 2τ ~ ~
1
(1)
where pz(z)) is the noisy latent at timestep f, c is the text prompt, and pa(-) and (-)) are the drift and diffusion coefficients, respectively. The denoising model ( c, t) is trained to approximate the score function - log Pt (zz) via score matching, and is used to construct an ODE solver V.
To improve the quality of conditional sampling, we can augment the denoising model Go with classifier- free guidance (CFG) (Ho & Salimans 2021) and the gradient of an energy function g
(Z), t) Go (Zz, 1) w (Fi c,t) cs (ze, ()) + 2V (z,,t),
where w and À are parameters controlling the guidance strength.
Consistency Model (CM) (Song et al. 2023 Luo et al. (023) is proposed accelerate the sampling from PF-ODE. Specifically, it learns consistency function f (Et, c, t) - to directly map any 2, on the PF-ODE trajectory model with CM fo and distill it from pretrained diffusion model, e.g., denoising model e by minimizing the consistency distillation (CD) loss.
its origin, where is a fixed small positive number. We can
Consider discretizing the time horizon into N - sub-intervals with T. we can sample any zt.-x (k the skipping interval of the ODE solver v from the PF-ODE trajectory and obtain its solution with V. Conventional methods (Luo et al. 2023 Wang et al. 20236 Li et al. 2024a augment & with CFG and the solution is given as below
< l2 <
IN
V.w 2t (1 + w) St butky & bay C; 1b) WV +&, Antk, 03
3

---

Published as a conference paper at ICLR 2025
Single-Stop
Zo
Zi,
Generation
J (0)
Ring
Forward Diffusion
fo
R,
Teacher ODE Solver
A Att
DDIM
A(z
LCD (0; v
Consistency Distillation
g
2
A majestic, white horse gallops gracefully across a mooniit beach....
fo
from RMs Gradient from Distillation
Temporal Attention
Training pipeline of our REV-Turbo-v2 When augmenting the teacher PF-ODE solver with CPG and motion guidance, we extract motion prior A(2 from the training videos and distill into the student CM fo along with the CFG.
Figure 2:
One can condition fa on w. and formulate the CD loss by enforcing CM's self-consisteeccy property:
LCD (0,0-:1) E Ex,e,.,n (d (fo (Zintz W. c, tn+k) fo-
( w. c, tn
Luo al. 2023 Wang et al. 2023b Li et al. 2024b) update 0= as the exponential moving average (EMA) of 0, i.e.. 0- stop.ttope....... grad (n) + (1 - ())0)) In this paper, we find that 0- is actually removable.
where d(.,-) is a distance function. Conventional methods (Song et al., 2023
Temporal Attention is introduced to model the temporal dynamic of video. Given batch of with batch size B, frames, channels, and spatial dimen- sions H W, the temporal attention operator first reshapes the features by merging the spatial dimensions into the batch size dimension, leading to z E R XxX)xxxxx Then, it performs self-attention (Vaswani et al. 2017) along the frame axis to derive the temporal attention as below
video latents z RBxFFXXXH
A(z) = Wtwettiniiio on (Q (z), K (=),K V (#)) E R(BxHxW)xFXF
(5)
where Q, K, and V are the Query, Key, and Value heads. And A(z) satisfies E A(=) 1.
MotionClone (Ling al. 2024) represents the video motion the temporal attention and enables the cloning of the motion from reference video to control the video generation process. At its core, MotionClone augments the conventional classifier-fiee guidance (Ho & Salimans (021) with Primary temporal-attenie attention guidance (PTA guidance) and Location-awar semantic guidance (LAS guidance). In this paper, focus the PTTA guidance. Given reference video, we first obtain its latent z; at the timestep via DDIM inversion (Song et al. 2020a). Then, we can derive the energy function Gm associated with PTA guidance at timestep as below
zof
nTh =2),)(")1) ||M (A(z)") A(=+))
where o denotes the Hadamard product and M is the temporal mask set to mask everything except for the highest activation along the temporal axis of A(z i.e.
[1, M(p.s.) = ( otherwise
if ((x)))))0.))) = max, A(z) (p...))
INTEGRATE REWARDS AND CONDITIONAL GUIDANCE INTO CONSISTENCY DISTILLATION
Conventional CD methods Song et al. 2023 Luo et al. 2023 Li et al.
2024a(b) typically neglect
the design space of conditional guidance methods and do not employ energy function G aug-
4

---

Published as a conference paper at ICLR 2025
derive the solution of the w. ب, + 2V (zz, with À controlling the conditional guid- ance strength. And similarly, we further condition our CM fe on X, i..., fo (Zt, W, X, C, t) + Zo-
ment the ODE solver V. With the conditional guidance from G, augmented ODE solver ",
In this paper, we are particularly interested in leveraging the PTA guidance our motion guidance gmm. With CFG, the solution of the augmented solver can be given by
Blatk (1 + ( tn+ki tns C; v)) w thi bntks tn 2i
A V Rense 9 BBn44e Anote bn+k)
The core insight of our method is that given video datasets with decent motion quality, the training video itself naturally serves as the ideal reference when the generation prompt its own caption. Therefore, we can always extract the motion information from the training video and employ it guide the video generation. Following MotionClone (Ling et al. 2024), we only apply motion guidance to the first percent of the sampling steps, i.e., we explicitly set X 0 for < N (). Note that for 2 N(1 - T), we can still set À = 0 for video without good motion quality.
3.1 LEARNING OBJECTIVES
Our preliminary experiment shows that the target network fo- can be removed without affecting the training stability. This is empirically significant, as it frees us from maintaining the EMA 0=, saving a substantial amount of GPU memory. With fo conditioned X, our new CD loss can be derived by slightly modifying Eq. 4
LCD (0: V = Ex,c.., (dl fo (Zinte w. X, c, tx+k) stop grad ( fo (= w. A, C, c,,,,,,)))))
Note that we do not distill the LAS guidance term from the original MotionClone, which has been demonstrated to enhance generation quality. Instead, we mitigate the potential quality loss by fol- lowing T2V-Turbo (Li et al., 2024b). augmenting the conventional consistency distillation with an objective to maximize a mixture of RMs, including an image-text RM Ring and video-text RM Ry,. The reward optimization objective J can be formulated as below
M
J (0) Ez,en
Fimg
Ring ( c) + B,R. (#o, c) - Dfo (Zemles X, e, bm+x)),
(10)
where Bung and By are the weighting parameters. Note that with minimal change to Eq. [0] In Sec. [4.3 we investigate the effects of different choices of RMs. Our total training loss combines the CD loss and the reward optimization objective as follows:
optimize multiple Rang and R.
L (0; v) LCD (0; v) (().
(11)
3.2 DATA PROCESSING AND TRAINING PROCEDURES
important to note that calculating the gradient of energy function 9 can be computationally expensive, consuming substantial memory. For example, using the motion guidance Gm as the energy function further requires performing DDIM inversion to obtain v which is too expensive to be done during each training iteration. For example, MotionClone's original implementation can consume over 40GB of GPU memory and require 3 minutes to perform the DDIM inversion.
Fortunately, identify that the solution A.w. be pre-caleulated before training the CM fo- Appendix A describs the detailed procedures our preprocessing procedures in Algorithm I and includes the useudo-codes for training in Algorithm 2 Our training pipeline is depicted in Fig.
EXPERIMENTAL RESULTS
Our experiments aim demonstrate T2V-Turbo-v-22 ability to generate high-quality videos and unveil the key design choices contributing to our superior performance. Sec. [.1 evaluate our method V Vench (Huang et al.) 2024) from various dimensions against broad array of baseline
5

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results on VBench (Huang et al. 2024). We present the performance of our T2V-Turbo-v2 with 16 inference steps, both with and without the application of Motion Guid- ance, and benchmark it against a wide range of baseline models. Quality Score, Semantic Score, and Total Score respectively reflect the visual quality, text-video alignment, and overall human pref- erence of the generated videos. [Table 11 in Appendix [ provides a detailed breakdown for each evaluation dimension. The best result for each score is highlighted in bold, and the second-best result is underlined. Our TZV-Turbo-vz achieves SOTA results on VBench, outperforming all baseline methods, including proprictary systems such Gen-3 and Kling, in terms of Total Score.
T2V-Turbo
T2V Turbo v2
Pika
Gen-2
Gen-3
Kling
VidenCrafter2
4-step
16-step
w/o MG
MG
Quality Score
82.92
82.47
84.11
83.39
82.20
82.57
82.27
84.08
85.13
Semantic Score
71.77
73.03
75.17
75.68
73.42
74.76
73.43
78.33
77.12
Total Score
80.69
80.58
82.32
81.85
80.44
81.01
80.51
82,93
83,52
methods. We then perform thorough ablation studies to demonstrate the importance of carefully selecting training datasets (See. 4.2 reward models (Sec. 4.3 and guidance methods (Sec. 4.4b
Settings. We distill our Tervo-uro--2 from VideoCrafter2 (Chen et al. 2024a). All our models trained on 8 NVIDIA A100 GPUs for 8K gradient steps without gradient accumulation. We use a batch size of 3 to calculate the CD loss and to optimize the reward objective on each GPU device. During optimization of the image- reward model Ring. we randomly sample 2 frames from each video by setting M = 2. The learning rate is set to le le- 5, and the guidance scale is defined within the range [amin, Wmax]
[5, 15]. We use DDIM (Song et al., 2020a)) as our ODE solver Vr with a skipping step parameter of k = 5. For motion guidance (MG), we set the motion
guidance percentage 0.5 and strength À 500.
4.1 COMPARISON WITH SOTA METHODS ON VBENCH
We train two variants of our T2V-Turbo-v2 Specifically, T2V-Turbo-v2 w/o MG is trained
using the CFG-augmented solver (Eq. 3 without motion guidance, whereas T2V- Turbo-yo v2 w/ MG includes motion guidance by using solver in Eq. 8 We train on mixed dataset VG + WV.
which consists of equal portions of VidGen- 1M (Tan et al. 2024) and Web 10M (Bain et al, 2021). While the CD loss is optimized across the entire dataset, the reward objective Eq. 10 optimized using only WebVid data. We utilize a combination of HPSv2.1 (Wu et al., 2023a) and ClipScore (Radford et al. 202T) as our Rimg. applying the same weight of Bang 0.2. Additionally, we employ the second-stage model of Intern Video2 (InternV2) (Wang et al. 2024) as the our R, with By 0.5. The rationale behind these design choices is elaborated in the ablation sections.
Table compares the 16-step generation of our methods with selective baselines from the VBench leaderboard including Gen-2 (Esser et al. 2023), Gen-3 (Esser et al. 2023). Pika (Pika Labs
2023). VideoCrafterl (Chen et al.) 2023). VideoCrafter2 (Chen et al.] 2024a). Kling (Kuaishou
2024.. 4 and the 4-step and 16-step generations of T2V-Turbo (Li et al. 20246). Except for the 16- step generations from T2V-Turbo, the performance of the other baseline method is directly reported from the VBench leaderboard. To evaluate the 16-step generation of our method and T2V-Turbo, we carefully follow VBench's evaluation protocols by generating 5 videos for each prompt. The Quality Score assesses the visual quality of the generated videos across 7 dimensions, while the Semantic Score measures the alignment between text prompts and generated videos across 9 dimensions. The Total Score is a weighted sum of the Quality Score and Semantic Score. Appendix B provides further details, including explanations for each dimension of VBench.
Both variants of our T2V-Turbo-v2 consistently surpass all baseline methods on VBench in terms of Total Score, outperforming even proprietary systems such as Gen-3 and Kling. Our mod- els establish a SOTA on VBench as of the submission date. The superior performance of our T2V-Turbo-v? without motion guidance (w/o MG) compared to T2V-Turbo underscores the im- portance of carefully selecting training datasets and reward models (RMs). While the 16-step results of T2V-Turoo underperform its 4-step counterpart, Appendix C.I shows that our T2V-Turbo-v2 effectively scales with increased inference steps and still outperforms Tur-o with 4 steps.

---

Published as a conference paper at ICLR 2025
Table 2: Ablation studies on the design of training datasets. While VCM performed best on OV, T2V-Turbo-v2 w/o MG only achieves modest improvements on OV but excels on VG+WV, highlighting the importance curating specialized datasets fully enhance model performance.
VCM
F2V-Turbo-T v2 w/o MG
OV
VG
WV
OV WV
VG WV
OV
VG
WV
OV WV
VG WV
Quality Score
83,62
82.24
81.31
80.00
82.95
84,04
82.28
83.41
82.32
84.08
Semantic Score
61.93
58.06
55.51
46.53
60.65
68.73
72.22
73.04
75.74
78,33
Total Score
78.52
77.41
76.15
73.30
78.49
80.97
80.26
81.34
81.00
82.93
A confused panda in calculus class
Videos: click to play
WV
W
An astronaut flying space
WV
vG WV
Figure 3: Trained the VG WV data, which tailored for specific learning objectives, D2V-Turbo-ves generates videos with improved visual quality and enhanced text-video alignment compared to training solely on the WV data of varying quality. Play videos in Adobe Acrobat.
4.2 ABLATION STUDIES ON THE DESIGN OF TRAINING DATASETS
To demonstrate the importance of training data selection, experiment with VidGen- 1M (Tan et al. (024) (VG), OpenVid-11 (Nan et al. 2024) (OV), Web (Bain el. (021) (WV), and their combinations. Specifically, OV and VG contain videos with high visual quality and detailed captions, whereas the conventional WV contains videos with mixed quality and short captions. VG
WV combines equal portions of VG and WV. OV WV combines equal portions of OV and WV. We train VCM and our -12V-uuro--o 222 w/o MG using the same set of RMs as in Sec. 4. for each dataset and report the evaluation results in Table 2 (per-dimension scores are in Table 8).
Surprisingly, T2V-Turbo-v2's performance does not scale in parallel with VCM's perfor- mance. While VCM attains its highest performance on OV, incorporating reward feedback only leads to modest performance gain: gain of 0.41 (83.63 84.04) on Quality Score, 5.80 (61.9 - 68.73) on Semantic Score, and 2.45 (78.52 - 80.97) on Total Score. similar phenomenon WV is relatively low, but integrating reward feedback yields substantial gain, boosting Quality Score by 2010 (81.31 + 83.41), Semantic Score by 17.53 (55.5 + 73.04), and Total Score by 5.19 (76.15 + 81.34).
observed with VG. In contrast, VCM's performance
Based on these results, we hypothesize that the modest performance gains on the higher-quality datasets, OV and VG, due to their excessively long video captions, which exceed the maximum context length of our RMs. For instance, the maximum context length of HPSv2.1 and CLIP is 77 tokens, while Intern V2 has maximum context length of only 40 tokens. As result, these RMs

---

Published as a conference paper at ICLR 2025
Table Ablation studies on the design of reward models. We train our TZV-Turbo w/ MG on VG WV with different combinations of RMs. While HPSv2. contributes the most to T2V-Turbo s performance as reported by [ii et al. (2024b), incorporating feedback from a diverse set of RMs crucial for our good performance, highlighting that RM selection is dataset dependent.
HPSv2.1
HPSv2.1
CLIP
HPSv2.1 + CLIP + Intem
HPSv2.1
CLIP
Inter
CLIP
IntemVV
+ InternV2
Quality Score
82.78
82.76
83.11
83.02
82.13
84.17
84.05
85.
Semantic Score
64.01
64.28
70.80
74.75
77.66
73.40
74.01
77.12
Total Score
79.02
79.07
80.65
81.37
81.24
82.02
82.04
83,52
Vincent van Gogh painting the room
Videos: click to play
HPSV2.1
HPSV2.1
HPSV2.1
HPSv2.1 CLIP
InterV2
Figure 4: Qualitative comparison when learning with different combinations of RMs. Incorporating feedback from HPSv2. is not enough to achieve satisfactory text-video alignment. Learning from diverse RMs enhances both visual and semantic quality. Play videos in Adobe Acrobat.
only operate optimally when trained datasets with shorter captions. To fully leverage high-quality training videos and maximize the impact of reward feedback, a plausible approach would be using visually appealing data for CD loss and short-captioned data for reward optimization. However, as demonstrated in Appendix C.2 this decoupling can result in undesired color distortion in the gener- ated videos, potentially due the substantial domain shift between the two datasets regarding the prompt space. This finding suggests that the CD loss acts as an essential regularizer that prevents reward over-optimization. Hence, we minimize CD loss using entire datasets while restricting re- ward optimization to short-captioned datasets. In Appendix C.2 we empirically demonstrate that this approach yields superior results compared to using the entire dataset reward optimization.
As shown in Table 2, R2V-Turbo-v? achieves the highest scores the VG WV datasets. Fig. 3 compares vidcos generated from the model trained on VG + WV and the model trained on WV. However, while T2V-Turbo v2 shows notable improvements over VCM on OV WV, its overall performance remains moderate, possibly due to the substantial domain gap between OV and WV.
4.3 ABLATION STUDIES ON THE DESIGN OF REWARD MODELS
From Table 2 we can conclude that VCM falls short aligning text and video from the results. None of the VCM variants achieve satisfactory Semantic Score, which motivates to incorporate lasiun-angguaee foundation models (Bommasani et al. 2021). such as CLIP and Intern to en- hance text-video alignment in addition to feedback from 1. In this section, we perform com- prehensive ablation studies to assess the feedback from each model by training T2V-Tuzbo-v2 w/ MG with different combinations of 1,Sv2.1, CLIPScore, and Intern V2 on VG MV.
8

---

Published as a conference paper at ICLR 2025
A soophistiated chimporzee in a tweed jacket and glasses ponders over a chessboard
Videos: click to play
wio MG
MG
A Corgi dog ricing tike in Times Square. à wearing sur cuseessses and a beach hal
wio MG
wi MG
Figure 5: Comparison between the generations from (Bottom). Integrating the Motion guidance leads to richer video motion that aligns with the prompt.
-2 w/o MG (Top) and w/ MG
Table 4:
Ablation studies on the effectiveness of the motion guidance. Augmenting the PF-ODE solver with motion guidance improves performance in the Human Action, Dynamic Degree, Motion Binding, Action Binding and Dynamic Attribute without decreasing Motion Smoothness.
Human
Dynamic Degree
Motion Smooth.
Motion Binding
Action Binding
Dynamic
T2V- TVy-Turb--2
w/o MG
97.17
61.39
97.00
22.15
72.73
20.05
97.35
90.00
97.07
24.38
(per-dimensiin scores are in Table 9) and Fig. learning from a more diverse set of RMs results in better performance. Relying solely feedback from HPSv2. results in only minimal enhancements in video quality compared to the baseline VCM trained without reward feedback. This contrasts with the findings [i al. (2024b), where HPSv2. significantly contributed to T2V-Turbo's performance when purely trained on the WV datasets. This discrepancy underscores that the impact of reward feedback is highly dependent on the dataset, emphasizing the importance of carefully designing the RM sets to achieve optimal performance.
As illustrated in Table 3
4.4 ABLATION STUDIES ON THE EFFECT OF MOTION GUIDANCE
Table 1 demonstrates that augmenting V with motion guidance, G. improves performance on VBench. further evaluate the improvements in terms of video motion quality, we provide scores for Human Action, Dynamic Degree, and Motion Smoothness across different methods in Table 4 For more comprehensive assessment of motion quality, we further report the Motion Binding, Ac- tion Binding and Dynamic Atribute scores from T2V-CompBench (Sun et al.) (022).. Experimental results indicate that incorporating motion guidance enhances performance and improves all metrics listed in Table 4 Fig. further includes qualitative comparison between different the two variants. Additional qualitative results are included in Fig. 7 and Fig. g
5 RELATED WORK
Diffusion-basel T2V Models. Conventional T2V models often rely large-scale image datasets
for training (Ho et al., 20226 Wang et al., [023a Chen et al., 2023 2024a] Ho et al.) 2022a) or adopt weights from pre-trained text-to-image (T2)) models (Zhang al. 2023 Blatmaan et al.

---

Published as a conference paper at ICLR 2025
2023
Khachatryan et al. 2023). For example, Vie (Wang al. 2023c) begins training with
Web Vid- 10M and LAION -BB before fine-tuning on a curated internal dataset of 23 million videos. Text-image datasets, such LAION (Schuhmann et al.) 2022), tend to be more than ten times larger than open-source video-text datasets like Web
BBann et al., 2021), offering higher spatial resolution and greater diversity (Wang al. 2023a). Recently. high-quality video datasets (Tan) itt al. 2024: Nan et al., 2024] Chen et al. 2024b| Wang et al. 2023d; Yang et al. (024) with dense captions are collected and released to the public. In this paper, we aim to leverage the supervision signals from high-quality video to improve pretrained T2V model during the post-training phase.
Vision-and-langure Reward Models. Several open- image-text reward models (RMs) have been developed to mirror human preferences for given image-text pair, including HPS (Wu et al.) 2023b ImageReward (Xu et al. 2024), and PickScore (Kirstain et al.. 2024). These models are typically fine-tuned from image-text foundation models like CLIP (Radford et al., (021) and BLIP (Li et al.] 2022) with human preference data. Recently, VideoScore (He et al. (024) is re- leased to reflect human preference on video-text pair. In this paper, we choose our RMs based on the fact that VCM struggles to achieve satisfactory text-video alignment on all of our training dataset variants. Thus, we leverage vision- vision-language foundation models CLIP and Intern V2, along with HPSv2.1, to improve the semantic quality of the generated videos. We perform thorough ex- periments to show that our model benefits from a diverse set of RMs.
Learning from Human/Al Feedback is an effective method for aligning generative models with human preferences (Leike et al.] 2018 ZZegler al., 2019 Ouyang al., 2022 Stiennon et al. 2020 Rafailov et al. 2024). In the domain of image generation, various approaches have been introduced align T2I models with human preferences, including reinforcement learning (RL)- based methods (Fan et al. 2024] Prabhudesai et al.) 2023; [Zhang et al.) (0024) and backpropagation.
based reward fine-tuning techniques (Clark et al.] 2023; Xu et al. [2024] Prabhudesai et al. 2023).
Recently, Instruct Video (Yuan et al.] 2023) and Vader (Prabhudesai et al. (2024) extended reward fine-tuning to optimize T2V models. Our method extends T2V-Turbo (Li et al. 2024b). integrating supervision from high-quality datasets, diverse reward feedback, and conditional guidance.
Training-Free Conditional Guidance has been widely adopted in controlling image genera-
tions (Feng et al., [023] Mo et al. 2024| Cao et al. 2023 Epstein et al., [2023] Ge et al., 2023]
Patashnik et al.] 2023] Xie et al., 2023] Zhao et al., 2022) and achieves great success. Motion- Clone (Ling et al. 2024) tackles T2V generation by leveraging temporal attention from reference video to guide the generation process. In this paper, we leverage the same motion guidance from
training videos to augment the ODE solver, enhancing motion quality of the generated videos.
CONCLUSION AND LIMITATIONS
In this paper, we present 12V-Turbo-v2. which integrates additional supervision signals from high-quality data, diverse reward feedback, and conditional guidance into the consistency distillation process. Notably, the 16-step generations from r2V-Turbo-v2 establish new state-of-the-ar result on VBench, surpassing both its teacher VideoCrafter2 and proprietary systems such as Gen- 3 and Kling. Through comprehensive ablation studies, we demonstrate the critical importance of tailoring data to specific training objectives. Additionally, we observe that VCM, without reward feedback, struggles to align text with the generated videos, highlighting the need to align with vision- language foundation models to improve text-video alignment. Furthermore, we identify and explore the vast design space of energy function, which can be used to augment the teacher ODE solver. We empirically validate the potential of this approach by showing that incorporating motion priors extracted from training data enhances the motion quality of the generated videos.
While our T2V-Turbo- demonstrates impressive empirical results, it is important to acknowl- edge certain limitations. One key constraint is that our approach cannot fully capitalize on high- quality datasets, such as Open VM, due to the limited context length of existing reward mod- els. Additionally, the teacher model used in this work, VideoCrafter2, also relies on the CLIP text encoder for processing the generation text prompt, which may limit its ability to serve as the teacher ODE solver to handle dense video captions. Therefore, developing long-context reward models would benefit future post-training research in video generation models. Furthermore, future T2V models should incorporate text encoders capable of comprehending longer and more detailed prompts, thereby enhancing their capacity generate richer and more aligned video content.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
In conducting this research, we have adhered to ethical guidelines to ensure integrity and trans- parency throughout the study. Our work does not involve human subjects, and therefore, no IRB ap- proval was required. The datasets and models used in this study, such as OpenVid- MM, VigGen- IM, WebVid-10M, and VideoCrafter2, are publicly available and adhere to relevant data usage policies. We have taken care to datasets that respect individual privacy, and personally identifiable information has been included our research.
Since we plan to release the code and models in the future, are committed to formulating detailed guidelines to prevent the misuse of our models. These guidelines will emphasize ethical usage, dis- courage harmful applications, and promote responsible AI practices. Additionally, we will explore and incorporate safeguard mechanisms into our generation pipeline to mitigate risks associated with misuse, ensuring that the released models and code aligned with ethical standards and safety considerations. This proactive approach is part of our ongoing responsibility to contribute to the safe and positive development AI technologies.
REPRODUCIBILITY STATEMENT
Our experiments are conducted with all open-sourced codes and training data. Our implementation codes have been included in the supplementary material and will be released to the public in GitHub repository without breaking the double-blind rules.
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bobg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 07258, 2021.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL ttttts::/oppnatt com/research/ s- was-worr dd- wwwrrdddssmmaattai simulators
viddeo-generationnnmee
Mingdeng Cao, Xintao Wang, Zhongang Qi. Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22560-22570, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia. Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint -xXx:22411009047 2024a.
11

---

Published as a conference paper at ICLR 2025
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wer Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple mross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13320-13331, 2024b.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2023.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-16239, 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7346-7356, 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fiee ffree structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Confer- ence Learning Representations, 2023. URL https:/ http///pppeee..eeeeeeeee net forum?id PUIc)T44zQQ7
songwei Ge, Taesung Park, Jun Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text, In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pp. 77545-556,, 2023.
Gen-3. Gen-3.
2024.
URL
https://rrovaaml.ccom/l..
introducing gen 3 a alpha/
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning, 2023.
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, Kai Wang, Quy Duc Do, Yuansheng Ni, Bohan Lyu, Yaswanth Narsupalli, Ronggi Fan, Zhiheng Lyu, Yuchen Lin, and Wenhu Chen. Videoscore: Building automatic metries simulate fine grained human feedback for video generation. ArXiv, abs/2406.1523 15552, 2024. URL https: pattts://arxiqe www.voorg//aaae abs 2406. 15252
Jonathan Ho and Tim Salimans. Classifier-fres diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1166840-851.. 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2rXiv:2210.02300: 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Edward J Hu, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2025
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open- compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747 78747, 2023a.
Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. In The Twelfth International Conference on Learning Representations, 2023b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06:3 06358, 2024.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale im- age quality transformer. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 5148- 2021.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Kuaishou, Kling. 2024. URL https:ttt/tkiiiioe kuai shou. www..coo/eeee en
PKU-Yuan Lab and Tuzhan Al etc. Open-sora-plan, April 2024. URLittps:://oi..rg///
5281/zenodo. 10948109999
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv: 1..0077, 2018.
Jiachen Li, Weixi Feng, Wenhu Chen, and William Yang Wang. Reward guided latent consistency distillation. arXiv preprint arXiv:2403. 1027, 2024a.
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2y-turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arXiv preprint arXiv:2405. 18750, 2024b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified languaee understanding and generation. In ICML, 2022.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efffcient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19960, 2023a.
Zhen Li, Zuo Zuo-Liang Zhu, Ling-Har Han, Qibin Hou, Chun-Le Guo, and Ming- Ming-Ming Cheng, Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9801-98118 10, 2023b.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang. Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05338g 2024.
Luma. Luma ai. 2024. URL https::t/////t laas www.rr/mm
13

---

Published as a conference paper at ICLR 2025
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condi- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 7465-7475, 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv2247.003771, 2024.
Open-Sora. Open-sora:
Democratizing efficient video production for all, 2024. URL https:
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localiz- ing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23051-23061, 2023.
Pika Labs. Accessed september 25, 2023, 2023. URL https: www. wttp:://wwwwpi.a.ttttt..
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-
image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki. and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:240..08737 2024.
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 13325278-25294 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representattons, 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. International conference on machine learning, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
14

---

Published as a conference paper at ICLR 2025
Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v- compbench: A comprehensive benchmark for compositional videoo generation. arXiv preprint arXiv:2407.14500, 2024.
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-Im: A large-scale dataset for text-to-video generation. arXiv preprint arxiv:2408. 02629, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer EisionEECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint urXiv:2308.0633 2023a.
Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint arXiv:2312.09100 2023b.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv: 15103, 2023c.
Yi Wang. Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understand- ing and generation. arXiv preprint arXiv:2307.06942 2023d.
Yi Wang, Kunchang Li, Xinhao Li. Jiashuo Yu, Yinan He. Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint 1:Xiv:2403./5377 2024.
Jialian Wu, Jianfeng Wang, Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint 22i2:22112.00280. 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprini arXiv:2306.092:
2023a.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao and Hongsheng Li. Human preference score: Better aligning cox-tto-maaee models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096-2105, 2023b.
Jinheng Xie. Yuexiang Li. Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. arXiv preprint arXiv:2406.06044 06040, 2024.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. arXiv preprint arXiv:2312. /2490, 2023.
15

---

Published as a conference paper at ICLR 2025
David Junhao Zhang, Jay Zhangjie Wu, Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint arXiv:2309.15818 2023.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401. 12244, 2024.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation
via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xiv: /90.//9990000 1909008993, 2019.
16

---

Published as a conference paper at ICLR 2025
A PSEUDO-CODES OF OUR T2V-TURBO-v2's DATA PREPROCESSING AND TRAINING PIPELINE
Algorithm and Algorithm presents the pseudo-codes for data preprocessing and training, respec- tively.
Algorithm Data Preprocessing Pipeline
Require: text-video dataset D, ODE solver V, noise schedule a((t), (t),, guidance scale interval [Wiinn W. skipping interval k, guidance percentage т, VAE encoder E.
Initialize the processed dataset: DD + O
for (æc, c) E D do
Encode the video x into latent space: = L'xx)
Sample the time index: n U[1, k). w [Wminy Wmax]
# Obtain the solution with the PF-ODE solver
Sample Ztatk N (a (In+*) 2; (t++*) I)
2, H.w f- tt + (1 + w) (z th bu a/) - (iz, batke las 23; 1b)
Obtain the motion guidance
if n > N (1 -1-77 -)/k then
Obtain the temporal attention A(Z,, ai 1))
Perform DDIM inversion to obtain 2
Obtain the temporal attention A(2
Calculate attention mask M by following Eq.
Calculate the motion guidance:
Vz
Mo (A(z A(zeata))
else
g = end if
Update D. + D2 U2 Ie w
end for return D
Algorithm T2V- Turbo-v2 Training Pipeline
Require: processed latent dataset D,, initial model parameter 0, learning rate y,, distance metric d, decoder D, image-text RM Rang. video-text RM R,, reward scale Sing and Sy, motion guidance
scale X.
repeat
Sample (ZinER
Niw at G,w, c) D
Ztn V.Wdd +- 2tn G
N.w
æo D (fo (2, c,,x+t))
Jumg (0) Egnc EM Ring (2m7 c)
Jua () = Eisio [R, (2),,))
LCD = d (Fo (Zimro, W. C, En+k) stop stop-graad (fo (2th ee, c.tn)
L. (0; V) <- TCD - Tmng (vi)
0 0+ 0 VoL (0; 4v
until convergence
17

---

Published as a conference paper at ICLR 2025
B FURTHER DETAILS ABOUT VBNNC
In this section, we provide an overview of the metrics used in VBench (Huang et al. 2024). followed by a discussion of how the Quality Score, Semantic Score, and Total Score are derived. For further details, we encourage readers to consult the original VBench paper.
The Quality Score is determined using the following metrics:
Subject Consistency (Subject Consist.): This metric measures the similarity of DINO (Caron et al., 2021) features across frames.
Background Consistency (BG Consist.): This is calculated based on the CLIP (Radford et al. 2021) feature similarity across frames.
Temporal Flickering (Temporal Flicker.): The mean absolute difference between frames is used to quantify flickering.
Motion Smoothness (Motion Smooth.): Motion priors from the video frame interpolation model ([i et al. 2023b) assess the smoothness of motion.
Aesthetic Quality: This metric is based on the average aesthetic scores generated by the
LAION aesthetic predictor (Schuhmann al 2022).
Dynamic Degree: The RAFT model (Ted & Deng 2020 calculates the dynamic level of the video.
Imaging Quality: The MUSIQ (Ke al. 2021 predictor is used obtain the results.
The Quality Score is the weighted sum of these normalized metrics, with all metrics assigned weight of 1, except for Dynamic Degree, which weighted at 0.5.
The following metries contribute to the Semantic Score:
Object Class: The success rate of generating the intended object is assessed using GRiT (Wu al. 2022).
Multiple Object: GRiT (Wu al. 2022) also evaluates how well multiple objects generated as specified by the prompt.
Human Action: UMT (Li et al. 2023a) used to assess the depiction of human actions.
Color: This metric checks if the color in the output matches the expected color using
GRiT (Wu et al. 2022).
Spatial Relationship (Spatial Relation.): Calculated via a rule-based method similar (Huang et al., (2023)).
Scene: The video caption generated by Tag2Text (Huang et al. 2023b is compared to the
scene described in the prompt.
Appearance Style (Appear Style.): This metric uses VICLIP (Wagg et 2023d) match the video's appearance style to the prompt's 's style description.
Temporal Style: The similarity between the video feature and the temporal style descrip- tion from ViCLIP (Wang et al. 2023d) is evaluated.
Overall Consistency (Overall Consist.): The overall alignment between the video feature and the full text prompt is measured using ViCLIP (Wang et al. 2023d).
The Semantic Score the mean of the normalized values the above metrics. Finally, the Total Score is computed by taking the weighted sum of the Quality Score and the Semantic Score, using formula:
4 Quality Score + Semantic Score
Total Score
(12)
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL ABLATION RESULTS
C.I ABLATION STUDIES ON THE NUMBER OF INFERENCE STEPS.
We investigate the impact of varying the number inference steps Table 5 In general, increasing the number of inference steps leads to improved visual quality and better text-video alignment for our 12V-Turbo-v2. Our inference is performed using the BFloat data type. The 4-step sam- pling takes approximately I second, the 8-step sampling takes around 1.5 seconds, and the 16-step sampling takes about 3 seconds.
Table 5: Ablation studies on the number of inference steps. We collect the 4-step, 8-step, and 16- step generation from our T2V- TTV-uo--- and compare their performance VBench.
Total Score
Quality Score
Subject Consist.
BG
Temporal Flickering
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Steps
Consist.
4
82.34
83.93
94.30
94.80
96.82
97.17
61.52
84.72
72.77
8
83.05
84.74
95.03
95.86
97.23
97.14
62.30
88.06
72.32
16
83.52
85.13
95.50
96.71
97.35
97.07
62.61
90.00
71.78
Semantic Score
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall
Steps
Color
Scene
Consist
4
75.97
95.57
54.91
96.80
94.04
38,56
52.69
24.76
27.09
28.65
8
76.31
96.34
57.36
96.00
92.83
42.86
52.51
24.36
27.04
28.36
16
77.12
95.33
61.49
96.20
92.53
43.32
56.40
24.17
27.06
28.26
Note that the results T2V-Turbo (Li et al. 2024b) reported VBench (Huang et al. 2024)
leaderboard is obtained with 4 function evaluation steps. To ensure fair comparison between Toy-Turbo-vz and T2V-Turbo, we also evaluate the 4-step generation of our T2V-Tuzbo-vž and compare it with T2V-Turbo in Table 6. Our T2V-Turbo-v2 still outperforms T2r-ourb in terms of Quality Score, Semantic Score, and Total Score.
Table 6:
Comparison of 4-step generation between T2V-Turbo-v2 and T2V-Turbo. Our 12V-Turbo-v2 outperforms T2V-Turbo in Quality Score, Semantic Score, and Total Score.
Total Score
Quality Score
Subject Consist.
BG Consist.
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models
T2V-Turbo
81.01 82.34
82.57 83.93
96.28 94.30
97.02 94.80
97.48 96.82
97.34
63.04 61.52
49.17 84.72
72.49 72.77
12V-Turbo-va v2
97.17
Semantic Score
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall Consist.
Models
Color
Scene
Tur-Turbo
74.76 75.97
93.96
54.65
95.20
89.90
38.67 38.55
55.58
24.42 24.76
25.51 27.09
28.16
ReV-uurbo-o v2
95.57
54.91
96.80
94.04
52.69
28.65
19

---

Published as a conference paper at ICLR 2025
An astronaut riding a horse
Darth vader surfing in waves
Figure 6: Minimizing the CD loss on VG data and maximizing reward objectives on WV data leads
color distortion the generated videos.
C.2 ABLATION STUDIES ON THE DATASETS FOR REWARD OPTIMIZATION
When training on the mixed dataset VG + WV, we propose minimizing CD loss using both VG and WV data and only maximizing the reward objectives Eq. TO on the WV data with short captions. In this section, we further experiment with the other combination: 1) minimizing the CD loss on VG data and maximizing reward objectives on WV data. minimizing the CD loss and maximizing reward objectives using both VG and WV data.
Our experiments show that the first setting led reward over-optimization and color distortion in the generated videos, as shown in Fig. [ Table commares the second setting with the setting used the main paper, demonstrating that optimizing rewards using both VG and WV reduces the benefit of aligning with RMs.
Table 7: Ablation studies on the datasets for reward optimization. While both settings use both VG and WV data to minimize CD loss, only leveraging WV data with short captions for reward optimization leads to better performance.
Dataset for Optimizing Reward
Total Score
Quality Score
Subject Consist.
BG Consist.
Temporal Flickering
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
VG + WV WV
82.38
84.35 84.08
96.74 97.03
97.15 98.32
97.93
96.91 97.00
65.52
69.17
71.14
82.93
97.54
66.73
61.39
70.92
Dataset for Optimizing Reward
Semantic
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall Consist.
Color
Scene
Score
VG WV WV
74.49
94.59
49.21
95.00
93.98
42.59
51.31
23.87
26.09 26.85
28.18
78.33
96.42
64.76
94.40
94.85
48.08
56.41
24.29
28.76
20

---

Published as a conference paper at ICLR 2025
Table 8: Full results for the ablation studies on the design of training datasets, corresponding to Table 2 in the main paper. We bold the best results for each dimension and underline the second- best result.
Total Score
Quality Score
Subject Consist.
BG Consist.
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models (Datasets)
VCM
Vie
78.52
83.62
96.87
96.95
98.33
97.92
63.59
53.06
71.98
VidGen
77.41
82.24
94.39
94.57
97.95
96.82
59.65
78.33
65.29
WebVid
76.15
81.31
92.82
94.18
96.58
96.65
57.24
84.72
65.05
OV + WV
73.30
80.00
93.82
95.03
96.07
97.20
54.48
61.
67.88
VG + WV
78.49
82.95
94.83
95.84
97.42
97.17
60.92
75.56
67.98
T2V-Turbo-v2 w/o MG
VpenVid
80.97
84.04
97.50
98.02
98.28
97.82
64.43
55.83
70.78
VidGin
80.26
82.28
94.91
94.70
95.03
95.72
60.79
93.61
67.54
WebVid
81.34
83.41
97.32
98.29
98.15 13
97.16
65.36
48.89
71.75
OV + WV
81.00
82.32
97.27
97.22
97.60
97.47
62.76
45.28
70.93
VG + WV
82.93
84.08
97.03
98,32
97.54
97.00
66.73
61.39
70.92
Semantic Score
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall
Models (Datasets)
Color
Scene
Consist.
VCM
OpenVid
61.93
79.92
20.12
88.
26.59
28.21
23.22
26.24
VidGen
58.06
74.95
14.62
85.00
88.82
25.73
20.73
23.15
21.57
24.72
WebVid
55.51
62.52
10.15
82.40
86.22
20.98
18.39
23.78
23.11
24.86
OV + WV
46.53
34.35
3.99
90.60
19.67
22.97
21.34
21.72
VG WV
60.65
76.46
16.14
85.20
85.45
28.76
28.76
23.35
23.90
26.02
T2V Turbo v2
OpenVid
68.73
91.20
33.96
93.60
92.99
30.83
41.70
23.59
24.77
27.16
VidGen
72.22
92.50
43.63
94.40
91.30
40.34
47.89
24.22
25.31
27.40
WebVid
73.04
91.74
47.41
93.60
96.53
42.02
44.16
24.06
25.65
28. 28
OV WV
75.74
92.50
55.00
95.40
95.02
36.47
57.53
24.73
26.56
28.
VG + WV
78.33
96.42
64.76
94.40
94.85
48.08
56.41
24.29
26.85
28.76
21

---

Published as a conference paper at ICLR 2025
Table 9: Full results for ablation studies the RM design, corresponding Table in the main paper. We bold the best results for each dimension and underline the second-ber result.
Total Score
Quality Score
Subject Consist.
BG
Temporal Flicker.
Mation Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Reward Models
Consist.
VCM + (No RM)
79.02
82.78
95.28
95.38
97.00
97.35
60.12
76.39
67.84
HPSv2.1
79.07
82.76
95.45
95.77
95.22
95.01
63.61
81.39
73.85
+ CLIP
81.24
82.13
96.87
95,79
95.82
97.34
60.76
60.56
71.75
InternV2
82.02
84.17
97.18
99.07
97.33
97.03
67.67
58.61
71.26
CLIP InternV2
83.52
85.13
95.50
96.71
97.35
97.07
62.61
90.00
71.78
CLIP
80.65
83.11
95.72
94.89
96.49
97.87
61,86
76.94
67.73
+ I2tern
82.04
84.05
96.60
97.70
97.94
98.15
62.91
66.67
68.24
Intern
81.37
83.02
96.35
97.01
96.66
97.30
65.74
57.50
70.89
Semantic Score
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall
Reward Models
Color
Scene
Consist.
VCM + (No RM)
64.01
83.61
23.12
89.20
75.92
33.46
38.90
23.55
24.95
26.39
HPSv2.1
64.28
84.95
20.84
88.40
22.98
24.59
26.56
CLIIP
77.66
96.90
63.26
93.80
94.36
50.97
52.91
24.37
26.55
28.05
InternV2
73.40
94.19
47.70
93.40
90.69
40.19
53.30
23.59
25.70
27.80
CLIP IntermV2
77.12
95.33
61.49
96.20
92.53
43.32
56.40
24.17
27.06
28.26
CLIP
70.80
91.53
40.08
93.00
91.98
36.1
42.21
24.62
25.89
27.61
+ InternV2
74.01
94.51
47.47
96.40
94.25
36.47
50.71
24.23
26.43
28.36
InternV2
74.75
95.57
43.55
96.80
94.95
41.64
56.41
24.12
25.86
27.72
Table 10: Full results for ablation studies on the effectiveness of motion guidance, corresponding to Table in the main paper.
Total Score
Quality Score
Subject Consist.
BG
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models (Datasets)
Consist.
T2V-Turbo (OV+WV)
81.00
82.32
97.27
97.22
97.60
97.47
62.76
45.28
70.93
T2V-Turbo (VG+WV)
82.93
84.08
97.03
98.32
97.54
97.00
66.73
61.39
70.92
T2V-Durbo-v2 (OV+WV)
81.81
83.15
97.18
97.81
97.10
96.05
66.08
60.28
71.04
T2V-Zurbo v2 (VG+WV)
83.52
85.13
95.50
96.71
97.35
97.07
62.61
90.00
71.78
Models (Datasets)
Semantic Score
Object
Multiple Objects
Human Action
Color
Sputial Relation.
Scene
Appear. Style
Temporul Style
Overall
Class
Consist.
Turbo (OV-WV)
75.74
92.50
55.00
95.40
95.02
36.47
57.53
24.73
26.56
28.31
T2r-Turo (VG+WV)
78.33
96.42
64.76
94.40
94.85
48.08
56.41
24.29
26.85
28.76
T2V- Toy-Turb--22 v2 (OV+WV)
76.47
94.48
53.54
96.60
94.10
42.35
56.89
24.48
26.47
28.94
T2V-Tura--22 (VG-WV)
77.12
95.33
61.49
96.20
92.53
43.32
24.17
27.06
28.26
22

---

Published as a conference paper at ICLR 2025
Table 11: Automatic evaluation results VBench (Huang al. 2024 D2V-Turbo-wt with baseline methods across the 16 VBench dimensions. A higher score in- dicates better performance for a particular dimension. We bold the best results for each dimension and underline the second-best result. Quality Score is calculated with the 7 dimensions f rom the top table. Semantic Score is calculated with the 9 dimensions from the bottom table. Total Score weighted sum of Quality Score and Semantic Score. Our T2V-Turbo-v2 surpass all base- line methods with 8 inference steps in terms of Total Score, Quality Score, and Semantic Score, including the proprietary systems Gen-3 and Kling.
We compare
Total Score
Quality Score
Subject Consist.
BG Consist.
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models
VideoCrafter2
80.44
82.20
96.85
98.22
98.41
97.73
63.13
42.50
67.22
Pika
80.40
82.68
96.76
98.95
99.77
99.51
63.15
37.22
62.33
Gen-2
80.58
82.47
97.61
97.61
99.56
99.58
66.96
18.89
67.42
Gen-3
82.32
84.11
97.10
96.62
98.61
99.23
63.34
60.14
66.82
Kling
81.85
83.39
98.33
97.60
99.30
99.40
46.94
61.21
65.62
T2V-Turbo
81.01
82.57
96.28
97.02
97.48
97.34
63.04
49.17
72.49
r2V-Turbo-v2
w/o MG
82.93
84.08
97.03
98.32
97.54 97.35
97.00 97.07
66.73
61.39
70.92
w/ MG
83.52
85.13
95.50
96.71
62.61
90.00
71.78
Semantic Score
Object
Multiple Objects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall
Models
Color
Scene
Class
Consist.
VideoCrafter2
73.42
92.55
40.66
95.00
92.92
35.86
55.29
25.13
25
28.23
71.26
87.45
46.69
88.00
85.31
65.65
44.80
21.89
24.44
25.47
Gen-2
73.03
90.92
55.47
89.20
89.49
66.91
48.91
19.34
24.12
26.17
Gen-3
75.17
87.81
53.64
96.40
80.90
65.09 05.09
54.57
24.31
24.71
26.69
Kling
75.68
87.24
68.05
93.40
89.90
73.03
50.86
19.62
24.17
26.42
T2V-Turho
74.76
93.96
54.65
95.20
89.90
38.67
55.58
24.42
25.51
28.16
T2V-Turbo-v2
w/o MG
78.33
96.42
64.76
94.40
94.85
48.08
56.41
24.29
26.85
28.76
MG
77.12
95.33
61.49
96.2
92.53
43.32
56.4
24.17
27.06
28.26
23

---

Published as a conference paper at ICLR 2025
D ADDITIONAL QUALITATIVE RESULTS
shiba
Videos: click to play
wio MG
MG
a dog tunning hoppily
wio MG
ww MG
comero pon from left to right, man wenring sunglesses and business suif
wio
MG
waldng the
Wo MG
w MG
Figure 7: Additional qualitative comparison between T2V-Turbo-v2 w/o MG and w/ MG. Inte- grating the Motion guidance leads to richer video motion that aligns with the prompt. Play videos in Adobe Acrobat.
24

---

Published as a conference paper at ICLR 2025
Tur le swinming in
Videos: click to play
wo MG
WAY MG
gullar. serenacing moonii night
wio MB
WI MG
cute raccoon playing guttar in a boot on the ocean
MG
MG
a cow banding down to drink water from river
wio MG
we MG
Origamn dancers while poper. 3D render. on white bockgrourd. suddi shot. dancing modert donce
Wio MG
wi MG
Figure 8: Additional qualitative comparison between D2V-Turbo-vz w/o MG and w/ MG. Inte-
grating the Motion guidance leads to richer video motion that aligns with the prompt.
25

---

Published as a conference paper at ICLR 2025
E ADDITIONAL ABLATION STUDIES ON THE DESIGN OF TRAINING DATASETS
Table 12: Ablation studies the design of training datasets. Based the results in Table 2 further conduct experiments on OV VG and OV VG WV to corroborate the results.
VCM
T2V-Turbo Turbo-r2 22 wio MG
WV
WV
WV
WV
Quality Score
83.62
24
81.31
80.00
82.55
83.43
82.95
84.04
82 28
83.41
82.32
84.08
83.86
82.35
Semanicc Score
61.93
58.06
55.51
46.53
60.65
57.52
51,98
68.73
72.22
73.01
75.74
78.33
69.25
76,80
Total
We further evaluate the performance of conduct experiments on the OV VG and OV VG WV datasets to corroborate our conclusions in Table 2 Sec. 4.2
As shown in the table, VCM achieves a high Quality Score on the OV + VG dataset, similar train- ing on pure OV data, but adding the lower-quality WV data slightly decreases this score. Conversely, incorporating reward feedback in T2V-Turbo-V significantly improves the Semantic Score for the OV VG WV dataset, while the gains for OV + VG remain comparatively moderate. These find- ings align with our discussion the main text: RMs with short context lengths operate optimally on datasets with shorter captions, highlighting the importance of aligning dataset characteristics with RM capabilities. Furthermore, the results justify our decision to exclude OV data when training the main
models, incorporating OV into VG + WV datascts negatively impacts model performance.
26

---

Published as a conference paper at ICLR 2025
Motion Quality
General Preference
T2V-Turbow2 wi MG
T2V-Turbo-22
we MG
20
40
60
80 130
23
43
80
80
103
Figure 9: Human evaluation on 16-step generation of T2V-Turbo-v2 w/o MG and T2V-Turbo-v2 w/ MG shows a clear preference for the latter. By incorporating motion information extracted from the training videos, T2V-Turbo-v2 MG consistently produces videos that are favored for their superior motion quality and overall appeal.
F COMPARING OUR T2V-TURRO- v2 WITH T2V-TURBO VIA HUMAN EVALUATION
We conduct human evaluation to compare the 16-step video generation of T2V-Turbo-v2 w/o MG and T2V-Turbo-v2 MG verify the effectiveness of motion guidance. We carefully select 50 prompts that explicitly require motion generation from VBench. Each method generates 5 videos for each prompt with the same set random seeds.
We hire annotators from Amazon Mechanical Turk to answer two questions: Q1) Which video demonstrates better motion quality? Q2) Which video do you prefer given the prompt?
We form the video comparison task many batches of HITs. To ensure the annotation quality, we ensure the annotators are from English-speakiin countries, including AU, CA, NZ, GB, and the US. Each task needs around 30 seconds to complete, and pay each submitted HIT with 0.2 US dollars. Therefore, the hourly payment is about 24 US dollars. Note that the data annotation part of project is classified exempt by Human Subject Committee via IRB protocols.
The human evaluation results in Figure 9 show that videos generated by T2V-Turbo-v2 w/ MG are consistently preferred over those from T2V-Turbo w/o MG in terms of motion quality and overall appeal. These findings corroborate our automatic evaluation in Table 4, confirming that incorporat- ing motion guidance significantly enhances model performance and improves the motion quality of generated videos.
27


Reviews:
[{"review_id": "Cqm74RxN05", "reviewer": "Reviewer_PqeK", "summary": "This paper presents T2V-Turbo-v2, a method that enhances a diffusion-based text-to-video model by distilling a consistency model using high-quality training data, reward model feedback, and conditional guidance. The approach achieves state-of-the-art performance on VBench, demonstrating improved text-video alignment and motion quality through tailored datasets, diverse reward models, and optimized guidance strategies", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The results appear promising and solid.\n\nThe experiments are thorough.\n\nThe writing is easy to follow.", "weaknesses": "The method combines existing techniques such as consistency distillation and motion guidance, so its novelty is somewhat limited.\n\nVideoCrafter uses a 2D+1D decoupled spatial-temporal approach, whereas most recent advanced methods employ full 3D attention. How would motion guidance be applied when using 3D attention?\n\nWhat is the peak training cost, such as peak GPU memory, compared to training a single model? How does it perform on less powerful video diffusion models like Zeroscope—can it still achieve results comparable to VideoCrafter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKGh3ieNvj", "reviewer": "Reviewer_N6o1", "summary": "This paper introduces T2V-Turbo-v2, an improved text-to-video model that enhances video generation by distilling a consistency model from a pretrained T2V model during the post-training phase. The method integrates multiple supervision signals—including high-quality training data, reward model feedback, and conditional guidance—into the consistency distillation process. Experiments show a new state-of-the-art result on VBench.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow. \n2. The method employing motion guidance is logically sound, and the experimental results showing improved semantic scores effectively validate its effectiveness.", "weaknesses": "1. The early work Energy-guided stochastic differential equations[1] first present a framework that utilize an energy function to guide the generaion process for diffusion model. Please cite this paper.\n2.\tIn Figure 2, does the DDIM inversion require k forward passes for each training step? If so, does this introduce excessive computational cost?\n3.\tPlease provide .mp4 files for visual comparisons, as Vbench cannot fully substitute for a user study. Including video files will allow reviewers and readers to better assess the performance and quality of the proposed method.\n\n[1] The method employing motion guidance is logically sound,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8s6MgofOz4", "reviewer": "Reviewer_y5Jq", "summary": "This paper introduces T2V-Turbo-V2, aiming at improving the video quality and alignment with prompts by focusing the post-training phase. It distills a pre-trained T2V model using various supervision signals, including, 1. the reward models feedback from pre-trained vision language models (CLIP and InternV2) for both image and video levels. 2. the self-consistency loss used in many other distillation models, and injecting the classifier-free guidance and energy function into the self-consistency loss. \nThis paper also optimize the data preprocessing and the reward feedback, allowing the T2V-Turbo-V2 to achieve the state-of-the-art results on the Bench and outperform previous models like VideoCrafter2, Gen-3, and Kling.", "questions": "1. What are the evaluation datasets when comparing the T2V-Turbo-V2 with the SOTA methods (Table 1)?\n2. Since both the OV and VG dataset contain the high visual quality data, and the Quality Score is high when only using OV, it seems that OV is a good dataset to improve the visual quality, I want to know  the metrics of using OV + VG and OV + VG + WV (in Table 2 setting) and why the author does not use the OV dataset?\n3. For the motion guidance, how the values of  λ, τ were chosen?\n4. Can the author give more details on the dataset processing, like the needed computation resource?\n5. If replace the base pre-trained video generation model with other models, can the T2V-Turbo-V2 method still achieve good results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The model achieves high scores across multiple metrics and outperforms proprietary systems, demonstrating the effectiveness of proposed modules.\n2. T2V-Turbo-V2 introduce an effective post-training process to enhance video quality and prompt alignment, and this process if architecture agnostic, potentially can be used on other pre-trained video generation models.\n3. This paper provide detailed ablation studies on various factors like the dataset selection, reward model configurations, the effectiveness of motion guidance, and so on.", "weaknesses": "1. The model appears optimized for single-caption, short-context prompts, its ability to generate longer or more complex video context may be limited.\n2. When generating the dataset for the motion guidance, it may require considerable computation resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nm7dhkwIg6", "reviewer": "Reviewer_dKgW", "summary": "This paper present motionclone-based consistency distillation, using motion guidance to improve temporal and spatial coherence.", "questions": "1. I’m curious why this method, although based on distilling VideoCrafter2, outperforms VideoCrafter2 across multiple tasks, such as in the three metrics shown in Table 1.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This method demonstrates strong performance, achieving state-of-the-art (SOTA) results on VBench. Visualized video outputs appear smooth and high-quality, reflecting its effective design.\n\n2. The paper is clearly written, allowing reviewers to easily understand the authors' intent.\n\n3. This method is simple and effective, making it generally more practical.", "weaknesses": "This method is overly engineering-focused and lacks novelty, as the motion guidance and consistency distillation techniques involved are already established, making it appear less innovative.\n\nAdditionally, while it conducts extensive ablation experiments on motion guidance and reward models, this does not constitute a significant contribution of the paper. I am unclear about the paper's contributions; is it providing more interesting insights? It would be helpful if the authors could briefly summarize this in their response.\n\nRegarding the contribution summary of the paper (L117), it seems to emphasize the advantages of existing work and the potential of extracting motion priors. And, I do not see any strong insights that stand out; motionclone has already demonstrated this fairly clearly. If the authors mean that motion priors are particularly useful during T2V training, they should provide more experiments. For example, training SVD and VideoCrafter2 shows that the insights presented in T2V-Turbo are quite limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QspwI3FP6i", "reviewer": "Reviewer_XVsE", "summary": "In the paper, the authors present T2V-Turbo-v2, a method to enhance diffusion-based text-to-video models by distilling a consistency model from a pre-trained T2V model. This approach integrates high-quality training data, feedback from multiple reward models, and motion guidance into the distillation process. Through ablation studies, it emphasizes the importance of high-quality datasets and diverse reward models to improve visual quality and text-video alignment. The method also verifies the effectiveness of incorporating motion guidance to enhance video motion quality. T2V-Turbo-v2 achieves a state-of-the-art total score of 85.13 on VBench, outperforming advanced text-to-video models like Gen-3 and Kling.", "questions": "1. The pseudo-code in Algorithm 2 for training includes theta-, but the method states that EMA is not needed. Please update either the text or the algorithm to ensure consistency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The experiments are comprehensive and thorough, with detailed analysis.\n2. The analysis of minimizing CD loss using entire datasets while restricting reward optimization to short-captioned datasets is interesting and meaningful, which may encourage future work.\n3. This paper establishes a new SOTA total score on VBench, leveraging open-source models to outperform some advanced text-to-video models.\n4. The paper is well-written and easy to understand.", "weaknesses": "1. The integration of MotionClone into T2V-Turbo is an interesting direction. However, while the contribution highlights the potential for diverse forms of energy functions, this study primarily utilizes the motion representation from the MotionClone work without substantial modifications to the energy function's format. The other enhancements are relatively minor, such as the reward model used, which only adds a CLIP compared to T2V-Turbo, and the removal of the EMA model. It might be beneficial to explore further variations to strengthen the contribution.\n2. It would be beneficial to include a discussion on the performance differences between VCM and the proposed method across various datasets. For instance:\n    - Could you explore why VCM achieves the best performance using only OV, while the proposed method does not attain similar results? It appears that different methods may lead to varying conclusions regarding dataset choices. Additionally, how would the results differ if T2V-Turbo (not v2) was used?\n    - Considering OV and VG are both high-quality video datasets, it would be insightful to analyze why OV+WV exhibits poorer performance compared to VG+WV, which performs quite well.\n3. In the section on data preprocessing, the approach involves using DDIM Inversion on all videos to obtain the necessary motion guidance for training, which is effective in reducing training time. Nevertheless, this approach does not significantly simplify the overall complexity. It would be better to explore improvements to the motion guidance strategy itself to enhance training efficiency.\n4. It would be valuable to include theoretical or experimental results to analyze why the EMA model in consistency distillation is unnecessary.\n5. It would be better to conduct a user study to further verify the performance from a human perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)