T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback
Authors: Jiachen Li
Weixi Feng
Tsu-Jui Fu
Xinyi Wang
S Basu
Wenhu Chen
William Yang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Video generation; consistency model
Primary area: diffusion_based_models
TL;DR: We introduce T2V-Turbo, achieving fast and high-quality video generation by breaking the quality bottleneck of the video consistency model.

Abstract: Diffusion-based text-to-video (T2V) models have achieved significant success but continue to be hampered by the slow sampling speed of their iterative sampling processes. To address the challenge, consistency models have been proposed to facilitate fast inference, albeit at the cost of sample quality. In this work, we aim to break the quality bottleneck of a video consistency model (VCM) to achieve **both fast and high-quality video generation**. We introduce T2V-Turbo, which integrates feedback from a mixture of differentiable reward models into the consistency distillation (CD) process of a pre-trained T2V model. Notably, we directly optimize rewards associated with single-step generations that arise naturally from computing the CD loss, effectively bypassing the memory constraints imposed by backpropagating gradients through an iterative sampling process. Remarkably, the 4-step generations from our T2V-Turbo achieve the highest total score on VBench, even surpassing Gen-2 and Pika. We further conduct human evaluations to corroborate the results, validating that the 4-step generations from our T2V-Turbo are preferred over the 50-step DDIM samples from their teacher models, representing more than a tenfold acceleration while improving video generation quality.

T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback
Jiachen Li', Weixi Feng', Tsu-Jui Fu', Xinyi Sugato Basu?,
Wenhu Chen William Yang Wang'
UC Santa Barbara, GGoogle University of Waterloo
[ [jiachen_11 11, weixifeng, tsu-juifu, xinyi_wang, william]@ca ucsb. Saugato@google.coom com com "venhuchen@uuaterllo..c
Project
Videos: click to play in
Adobe Acrobat
VCM
4-step
F2V- Turbo
VCM
8- 8-step
T2V- Turbo
Figure 1: By integrating reward feedback during consistency distillation from VideoCrafter2 Chen et al. [024], our T2V-Turbo (VC2) can generate high-quality videos with 4-8 inference steps, break- ing the quality bottleneck of VCM [Wang et al. 2023a]. Appendix F Fincludes the corresponding text prompts.
Abstract
Diffusion-based text-to-video (T2V) models have achieved significant success but continue be hampered by the slow sampling speed of their iterative sampling processes. To address the challenge, consistency models have been proposed facilitate fast inference, albeit at the cost of sample quality. In this work, we aim to break the quality bottleneck achieve both fast and high-quality video generation. We introduce T2V-Turbo, which integrates feedback from a mixture differentiable reward models into the consistency distillation (CD) process of pre-trained T2V model. Notably, we directly optimize rewards associated with single-step generations that arise naturally from computing the CD loss, effectively bypassing the memory constraints imposed by backpropagating gradients through an iterative sampling process. Remarkably, the 4-step generations from our T2V-Turbo achieve the highest total score VBench [uang et al. 2024 even surpassing Gen-2 [Esser et al. 2023 and
video consistency model (VCM) to
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pika [Pika Labs 2023]. We further conduct human evaluations to corroborate the results, validating that the 4-step generations from our T2V-Turbo are preferred over the 50-step DDIM samples from their teacher models, representing more than tenfold acceleration while improving video generation quality.
1 Introduction
Diffusion model (DM) Sohl-Dickstein et al. 2015 Ho et al.
2020]
has emerged as a powerful frame-
work for neural image [Betker et al., 2023 Rombach et al.
2022
Esser et al. 2024 Saharia et al.
2022] and video synthesis [Singer et al. 2022 Ho et al. 2022a He et al. 2022
Wang et al. 2023b (T2V) models like
Zhang et al., 2023). leading to the development of cutting-edge
text-to-vides
Sora [Brooks et al. 2024), Gen-2 [Esser et al., 2023) and Pika (Pika Labs
2023]. Although the itera-
tive sampling process of these diffusion- based models ensures high-quality generation, it significantly slows down inference. hindering their real-time applications. On the other hand, existing open-
sourced T2V models including VideoCrafter [Chen et al. 2023 2024 and ModelScope [Wang trained on web-scale video datasets, e.g.. WebVid 10M [Bain al. 202 with
et al. 2023c] varying video qualities. Consequently, the generated videos often appear visually unappealing and fail to align accurately with the text prompts, deviating from human preferences.
Efforts have been made to address the issues listed above. To accelerate the inference process, Wang et al. [023a applies the theory of consistency distillation (CD) [Song et al., 2023 Song
and Dhariwal 2023 Luo et al. 2023a] to distill video consistency model (VCM) from teacher T2V model, enabling plausible video generations in just 4-8 inference steps. However, the quality of VCM's generations naturally bottlenecked by the performance the teacher model, and the reduced number of inference steps further diminishes its generation quality. On the other hand, to aligm generated videos with human preferences, Instruct Yuuan et al. [2023 draws inspiration from image generation techniques [Dong et al. 2023 Clark al. 2023 Prabhudesai et al. 2023 and proposes backpropagating the gradients differentiable reward model (RM) through the iterative video sampling process. However, calculating the full reward gradient is prohibitively expensive, resulting in substantial memory costs. Consequently, Instruct Video truncates the sampling chain by limiting gradient calculation only the final DDIM step, compromising optimization accuracy. Additionally, InstructVideo limited by its reliance on an image-text RM, which fails to fully capture the transition dynamic of video. Empirically, Instruct only conducts experiments on limited set of user prompts, the majority of which are related to animals. As a result, its generalizability to broader range of prompts remains unknown.
In this paper, we aim to achieve fast and high-quality video generation by breaking the quality bottleneck of VCM. We introduce T2V-Turbo, which integrates reward feedback from mixture of RMs into the process distilling a VCM from a teacher T2V model. Besides utilizing an image-text RM align individual video frames with human preference, further incorporate reward feedback
Gradient from Reward Maximization
Single-Step Generation
Mixed Reward Feedback
Gradient from Distillation
Jimg (0)
fo
Diffusion
Juia
EMA
ODE Solver
t
(Teacher)
A majestic, white horse gallops gracefully across beach....
LeD (0. 10:07:0
Consistency Distillation
Figure 2: Overview of the training pipeline of our T2V-Turbo. We integrate reward feedback from both an image-text RM and video-text RM into the VCD procedures by backpropagating gradient through the single-step generation process of our Tur-Turbo.

---

from video-text RM to comprehensively evaluate the temporal dynamics and transitions in the generated videos. We highlight that our reward optimization avoids tackling the highly memory- intensive issues associated with backpropagating gradients through an iterative sampling process. Instead, we directly optimize rewards of the single-step generations that arise from computing the CD loss, effectively bypassing the memory constraints faced by conventional methods that optimize a DM (Yuan et al. 2023 Xu et al. 2024a Clark et al. 2023 Prabhudesai et al. 2023).
Empirically, we demonstrate the superiority of our T2V-Turbo in generating high-quality videos within 4-8 inference steps. To illustrate the applicability of our methods, we distill T2V-Turbo (VC2) and T2V- (MS) from VideoCrafter2 [Chen et al. [024] and ModelScope [Wang] al. 2023cl. respectively. Remarkably. the 4-step generation results from both variants of our
T2V- Turbo outperform SOTA models on the video evaluation benchmark VBench [Huang et al.
2024 even surpassing proprietary systems such as Gen-2 [Esser et al., 2023) and Pika [Pika Labs
2023
that are trained with extensive resources. We further corroborate the results by conducting
human evaluation using 700 prompts from the EvalCrafter (Liuu et al. 2023] benchmark, validating
that the 4-step generations from T2V-Turbo are favored by human over the 50-step DDIM samples
from their teacher T2V models, which represents over tenfold inference acceleration and enhanced video generation quality.
Our contributions are threefold:
Learn a T2V model with feedback from a mixture of RMs, including a video-text model. To the best of our knowledge, we are the first to do so.
Establish a new SOTA on the VBench with only 4 inference steps, outperforming proprietary models trained with substantial resources.
4-step generations from our T2V-Turbo favored over the 50-step generation from its teacher T2V model as evidenced by human evaluation, representing over 10 times inference acceleration with quality improvement.
2 Preliminaries
Diffusion models (DMs). In the forward process, DMs progressively inject Gaussian noise into the original data distribution Paam( (x) (x(x)) and perturb it into marginal distribution (x(x;) with the transition kernel Pos (x, Xy) = (x (t)x),, (t) at timestep t. (t) and B(t) correspond to the noise schedule. In the reverse process, DMs sequentially recover the data from noise sampled from the prior distribution (x(x)) := A (xxr(0, 0, (T))). The reverse-time SDE can be modeled by an ordinary differential equation (ODE), known the Probability Flow (PF-ODE) (Song al. 2020a
dx: = (t)()x x: - I - (t) log Pt (xx) dt, XT ~ ~NO (T)))
where vx)) and (.) are the drift and diffusion coefficients, respectively, with the following properties
(t) = d lllo a(l) dt
0²(() d)3(1) log a1l) (1).
(2)
dt
di
The PF-ODE's solution trajectories, when sampled at any timestep t, align with the distribution pe (xx)).
Empirically, a denoising model ea (x,, t) is trained to approximate the score function V log Pr (x,) via score matching. During the sampling phase, one begins with a sample xT pT (xT) and follows the empirical PF-ODE below to obtain a sample Xa.
1 2 (x(x))) dt, xr x 3
dx;= p (t)x
33)
In this paper, we focus on diffusion-based T2V models, which operate on the video latent space Z and train a denoising model (z,, c, f) conditioned on the text prompt c, where 2, is obtained by perturb- ing the image latent E(x), E Z and is a VAE [Kingma and Welling 2013) encoder. The T2V models employ Classifier- Free Guidance (CFG) Ho and Salimans [021 to enhance the quality of conditional sampling by substituting the noise prediction with a linear combination of conditional and unconditional noise predictions for denoising, i.e., ēØ (2,, w. c, () (1 +w) --eee (Zz, -) (2,, Ø, (). where wi is the CFG scale. After the completion of the inference process, we can generate a video by xo = D(zo) with the VAE decoder D corresponding to E.
3

---

Consistency Distillation. Conventional methods [Ho et al. 2020 Song et al. 2020b generate their samples by solving the PF-ODE sequentially, leading to DM's ' slow inference speed. To tackle this problem, consistency models (CM) (Song et al. 2023 Song and Dhariwal 2023 propose to learn consistency function f (xt: 1) H> x to directly map any x on the PF-ODE trajectory to its origin, where e is fixed small positive number. And thus, the consistency function has the following self-consistency property
(4) where X, and x are from the same PF-ODE. We can model f with a CM fo- When tackling the PF-ODE of T2V model that operates on the video latent space 2, we aim to learn video consistency model (VCM) [Luo et al. 2023a Wang et al. 2023a fo (Ze, w. c, t) + Zo Z. To ensure fn w. c, t) = Z, we parameterize fa as
((x:, f(x,, ("), Vt, E [e,
f (z,
Ckkp(()z Coul
c,
where Cakip (1) and Cou are differentiable functions with Cakip()) and Cour(e) 0. and Fş is modeled as a neural network. We can distill a Sa from a pre-trained T2V DM by minimizing the consistency distillation (CD) [Song et al. 2023 Luo et al. 2023a loss as below
LCD (0, 0 ; Be,c.u, d (fo W (,444) Se- (2% w (,
(6)
where d(---) is distance function. 0- is updated by the exponential moving average (EMA) of 0, i.e., 0- f- stop_grad (nu + (1 -1- =))0). &,n is an estimate of 2:,, obtained by the numerical augmented PF-ODE solver V parameterized by v and k is the skipping interval
Ztn
-kack (1 w) (Z tet bny e; vb) w (tt, +k +k: kntk: t,s 2: 1))
(7)
We follow the LCM paper [Luo al. 2023a to DDIM (Song al. 2020b the ODE solver V
and defer the formula the DDIM solver to Appendix A
3 Training T2V-Turbo with Mixed Reward Feedback
In this section, we present the training pipeline to derive our T2V-Turbo. To facilitate fast and high-quality video generation, we integrate reward feedback from multiple RMs into the LCD process when distilling from teacher T2V model. Figure
2 provides an overview of our framework. Notably, we directly leverage the single-step generation 2n = fo (Zen4a:44 w. C, ft+tk) arise from computing the
CD loss LCD (6) and optimize the video Xn D(2n) decoded from it towards multiple differentiable RMs. As result,
avoid the challenges associated with backpropagating gradients through an iterative sampling process. which is often confronted by conventional methods optimizing DMs (Clark
al. 2023 Xu al. 2024a Yuan et al.. 2023
In particular, we leverage reward feedback from an image-text RM to improve human preference on each individual video frame (Sec. (3.1) and further utilize the feedback from video-text RM to improve the temporal dynamics and transitions in the generated video (Sec. 3.2
3.1 Optimizing Human Preference on Individual Video Frames
Chen et al. [2024] achieve high-quality video generation by including high-quality images as single- frame videos when training the T2V model. Inspired by their success, we align each individual video frame with human preference by optimizing towards differentiable image-text RM Rimg. In particular, we randomly sample batch of M frames {x...... M from the decoded video Xo and maximize their scores evaluated by Ring as below
M
Rimg (x1) ))].
Jimg (0) Exo.o
X0 = DD= (So (Zi C, cntxk))
(8)
3.2 Optimizing Video-Tee Text Feedback Model
Existing image-text RMs [w et al. 2023a Xu al. 2024a Kirstain et al.) 2024 are limited to assessing the alignment between individual video frames and the text prompt and thus cannot evaluate

---

through the temporal dimensions that involve inter-frame dependencies, such as motion dynamic and transitions [Huang et al. 2024 Liu et al. 2023]. To address these shortcomings, we further leverage a video-text RM Rvvi to assess the generated videos. The corresponding objective Jvia is given below
(0) = Ee [Rvid (Ro: c)], &o - (fo (Zen w, kn))))
3.3 Summary
To this end, we can define the total learning loss of our training pipeline as linear combination of the LCD in (6) Jung in (8) and Joia in (9) with weighting parameters Ping and Brid-
L (0,0
LCD (0, : Bing: img (0) (id)
(10)
To reduce memory and computational cost, we initialize our Torboo with the teacher model and
only optimize the LoRA weights [Flu et al. 2021 Luo et al. (023b] instead of performing full model
training. After completing the training, we merge the LoRA weights so that the per-step inference cost of our T2V-Turbo remains identical to the teacher model. We include pseudo-codes for our
training algorithm in Appendix B
4 Experimental Results
Our experiments aim to demonstrate our T2V-Turbo's 's ability to generate high-quality videos with 4-8 inference steps. We first conduct automatic evaluations on the standard benchmark VBench (Huang et al. 2024) to comprehensively evaluate our methods from various dimensions (Sec. 4.1) against broad array of baseline methods. We then perform human evaluations with 700 prompts from the EvalCrafter [Liu et al. 2023) to compare the 4-step and 8-step generations from our T2V-Turbo with the 50-step generations from the teacher T2V models as well as the 4-step generations from the baseline VCM (Sec. 4.2) Finally, we perform ablation studies on critical design choices (Sec. 4.3
Settings. We train T2V-Turbo (VC2) and Tur-Turbo (MS) by distilling from the teacher diffusion- based T2V models VideoCrafter2 [Chen et al. 2024] and ModelScopeT2V (Wang et al. [2023c]. respectively. Similar both teacher models, we conduct our training using the Web VidToM |Bin et al. 2021] datasets. We train our models on 8 NVIDIA A100 GPUs for 10K gradient steps without gradient accumulation. We set the batch size of training videos HPSv2.1 [Wu et al., (003al] as our image-text RM Ring. When distilling from VideoCrafter2, we utilize the 2nd Stage model of Intern Video2 (Intern Vid2 S2) [Wang et al.) 2024] as our video-text RM Ryia. When distilling from ModelScope we set Rvia to be ViCLIP [Wang et al., 2023d]. To optimize Jimg (8). we randomly sample frames from
for each GPU device. We employ
video by setting M = For the hyperaarameters (HP), we set learning rate le 5 and guidance scale range [Wiin: Wmax] 55 15).
We use DDIM [Song et al.) 2020b] as our ODE solver and set the skipping step = 20. For Tor-o (VC2), we set Bimg 1 and Byid = 2. For T2V-Turbo (MS), we set Bimg 2 and Brid = 3. We include further training details in Appendix A
4.1 Automatic Evaluation on VBench
We evaluate our
(VC2) and TuV-Turbo (MS) the standard video evaluation benchmark VBench (Huang et al. 2024] to compare against a wide array of baseline methods. VBench designed to comprehensively evaluate T2V models from 16 disentangled dimensions. Each dimension in VBench is tailored with specific prompts and evaluation methods.
Table compares the 4-step generation of our methods with various baselines from the VBench leaderboard] including Gen-2 [Esser et al. 2023), Pika (Pika Labs 2023). J. VideoCrafterl [Chen et al.
2023]. VideoCrafter2 [Chen et al.
2024].
Show-1 [Zhang et al. 2023 La [Wang et al. 2023b Table 4 in Appendix further compares our methods with
and ModeIScopeT2V [Wang et al.
2023c
VideoCrafter0.9 [He et al. [2022]
La Vie-
Interpolation (Wang et al.) (2023b). Open-Sora (Open-Sora
2024), and CogVider [Hong et al. 2022]. The performance of each baseline method is directly
reported from the VBench leaderboard. To obtain the results of our methods, we carefully follow VBench's 's evaluation protocols by generating 5 videos for each prompt to calculate the metrics. We
5

---

Table 1: Automatic Evaluation on VBench [Huang et al. 2024]. We compare our T2V-Turbo (VC2) and T2V- (MS) with baseline methods across the 16 VBench dimensions. A higher score indicates better performance for a particular dimension. We bold the best results for each dimension and underline the second-best result. Quality Score is calculated with the 7 dimensions from the top table. Semantic Score calculated with the 9 dimensions from the bottom table. Total Score weighted sum of Quality Score and Semantic Score. Further details can be found in Appendix C Both our T2V-Turbo (VC2) and T2V-Turbo (MS) surpass all baseline methods with 4 inference steps in terms of Total Score, including the proprictary systems Gen-2 and Pika.
Total Score
Quality Score
Subject Consist.
BG
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models
Consist.
ModeeSccopeT2V
75.75
78.05
89.87
95.29
98.28
95.79
$2.06
66.39
58.57
Vaee
77.08
78.78
91.41
97.47
98.30
96.38
54.94
49.72
61.90
Show-1
78.93
80.42
95.53
98.02
99.12
98.24
57.35
44.44
58.66
VideoCrafterl
79.72
81.59
95.10
98.04
98.93
95.67
62.67
55.00
65.46
Pika
80.40
82.68
96.76
98.95
99.77
99.51
63.15
37.22
62.33
VideoCrafter2
80.44
82.20
96.85
98.22
98.41
97.73
63.13
42.50
67.22
Gen-2
80.58
82.47
97.61
97.61
99.56
99.58
66.96
18.89
67.42
VCM (MS)
75.84
78,80
93.06
97.30
98,51
98.00
48,99
46.11
61.98
Our T2V-Turbo (MS)
80.62
82.15
94.82
98.71
97.99
95.64
60.04
66.39
68.09
VCM (VC2)
73.97
78.54
94.02
96.05
99.06
98.84
54.56
42.50
52.72
Our T2V- bo (VC2)
81.01
82.52
96.28
97.02
97.48
97.34
63.04
49.17
72.49
Semantic Score
Object Class
Multiple Ohjects
Human Action
Spatial Relation.
Appear. Style
Temporal Style
Overall
Models
Color
Scene
Consist.
ModelScopeT2V
66.54
82.25
38.98
92.40
81.72
33.68
39.26
23.39
25.37
25.67
LaVic
70.31
91.82
33.32
96.80
86.39
34.09
52.69
23.56
25.93
26.41
Show-1
72.98
93.07
45.47
95.60
86.35
$3.50
47.03
23.06
25.28
27.46
VideoCrafterl
72.22
78.18
45.66
91.60
93.32
58.86
43.75
24.41
25.54
26.76
Pika
71.26
87.45
46.69
88.00
85.31
65.65
44.80
21.89
24.44
25.47
VideoCrafter2
73.42
92.55
40.66
95. 00
92.92
35,86
$5.29
25.13
25.84
28.23
Gen-2
73.03
90.92
55.47
89.20
89.49
66.91
48.91
19.34
24.12
26.17
VCM (MS)
63.98
83,18
24.85
87,20
85.72
31.57
42.44
23.20
23.30
24.18
Our T2V-Turbo (MS)
74.47
93.34
58.63
95.80
89.67
45.74
48.47
23.23
25.92
27.51
VCM (VC2)
55.66
63.97
10.81
82.60
79.12
23.06
18.49
25.29
22.31
25.15
Our T2V-Turbo (VC2)
74.76
93.96
54.65
95.20
89.90
38.67
55.58
24.42
25.51
28.16
further train VCM (VC2) and VCM (MS) by distilling from VideoCrafter2 and ModelScopeT2V, respectively, without incorporating reward feedback, and then compare their results.
VBench has developed its own rules to calculate the Total Score, Quality Score, and Semantic Score. Quality Score is calculated with the 7 dimensions from the top table. Semantic Score is calculated with the 9 dimensions from the bottom table. And Total Score is a weighted sum of Quality Score and Semantic Score. Appendix Cprovides further details, including explanations for each dimension of VBench. As shown in Table 1 the 4-step generations of both our T2V-Turbo (MS) and results are particularly remarkable given that we even outperform the proprietary systems Gen-2 and Pika, which are trained with extensive resources. Even when distilling from a less advanced teacher model, ModelSSopeTT2V our T2V-Turbe (MS) attains the second-highest Total Score, just below our T2V (VC2). Additionally, our T2V-Turbo breaks the quality bottleneck of VCM by outperforming its teacher T2V model, significantly improving over the baseline VCM.
(VC2) surpass all baseline methods on VBench in terms of Total Score. These
4.2 Human Evaluation with 700 EvalCrafter Prompts
To verify the effectiveness of our T2V-Turbo, we compare the 4-step and 8-step generations from our T2V-Turbo with the 50-step DDIM samples from the corresponding teacher T2V models. We further compare the 4-step generations between our T2V-Turbo and their baseline VCMs when distilled from the same teacher T2V model. We leverage the 700 prompts from the EvalCrafter [Liu et al. 2023 video evaluation benchmark, which are constructed based real-world user data.

---

Visual Quality
Dext-Video Alignment
General Preference
T2V-Turbo (VC2) 4 step
$3.0
$2.6
54.1
CiaeoCrafter2 50 steps
T2V-Turbo
B1.0
59.6
$2.0
CidetCrafter2
T2V-Turbe (VC2) step
73.1
70.0
70.7
VCM (VC2)
staps
T2V-Turbo
86.9
60.4
61.7
ModeeSScope T2V 50 stops
(MS) 4 stop
T2V-Turbo (MS)
68.9
63.7
64.1
po
T2V-Turbo
VCM (MS)
(MS) step
stops
Proference [%6
Preferonce [N]
Figure 3: Human evaluation results with the 700 prompts from EvalCrafter [Liu et al. 2023]. We compare the 4-step and 8-step generations from our T2V-Turbo with their teacher T2V model and their baseline VCM. Top: results for T2V-Turbo (VC2). Bottom: results for T2V-Turbo (MS).
Pikachu snowboarding
dog wearing goggles boat
VCM (4-step)
Teacher (50-step)
T2V- 12V-Turbo Turbo (4-8tep)
T2V T2V-Turbo Turbo
(e-)tep)
Figure 4: Qualitative comparisons between the 4-step VCM, 50-step teacher T2V, 4-step T2V-Turbo and 8-step T2V-Turbo generations. Left: (VC2), Right: (MS).
We hire human annotators from Amazon Mechanical Turk to compare videos generated from different models given the same prompt. For each comparison, the annotators need to answer three questions: QI) Which video is more visually appealing? Q2) Which video better fits the text description? Q3) Which video do you prefer given the prompt? Appendix D includes additional details about how we set up the human evaluations.
Figure provides the full human evaluation results. We also qualitatively compare different methods in Figure Due to limited space, we include additional qualitative comparison results in Appendix F Notably, The 4-step generations from our T2V-Turbo are favored by humans over the 50-step generation from their teacher T2V model, representing 12.5 times inference acceleration with improving performance. By increasing the inference steps to 8, we can further improve the visual quality and text-video alignment of videos generated from our T2V-Ture our 8-step generations are more likely to favored by the human compared to our 4-step generations in terms of all 3 evaluated metrics. Additionally, our T2V-Turbo significantly outperforms its baseline VCM, demonstrating the effectiveness of our methods, which incorporate mixture of reward feedback into the model training.
reflected by the fact that

---

Table Ablation studies on the effectiveness of Rig and Rid We bold the highest score for each dimension for methods with the same teacher model. While incorporating feedback from Ring is effective at improving both Quality Score and Semantic Score, integrating reward feedback from Rvia can further improve the semantic score.
Total Score
Quality Score
Subject Consist.
BG Consist.
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Models
VCM (MS)
75.84
78.80
93.06
97.30
98.51
98.00
48.99
46.11
61.98
VCM (MS) Rysd
77.28
78.76
93.24
97.67
98.49
97.27
51.70
55.00
56.40
VCM (MS) Ring
79.51
81.81
97.64
99.59
98.46
95.83
64.69
38.33
68.86
Our T2V-Turbo (MS)
80.62
82.15
94.82
98.71
97.99
95.64
60.04
66.39
68.09
VCM (VC2)
73,97
78,54
94.02
96.05
99.06
98.84
54,56
42,50
52.72
VCM (VC2)= Ri
77.57
80,08
95.46
96.69
98.78
98.79
58.66
25,00
65.75
VCM (VC2)+ Rimg
80,42
82.59
96.52
97.31
97.50
97.29
63.08
47.50
72.91
Our T-rbu (VC2)
81.01
82.57
96.28
97.02
97.48
97.34
63.04
49.17
72.49
Models
Semantic
Object
Multiple Objects
Human Action
Color
Spatial Relation.
Scene
Appear Style
Temporal Style
Overall
Score
Class
Consist.
VCM (MS)
63.98
83.18
24.85
87.20
85.72
31.57
42.44
23.20
23.30
24.18
VCM (MS)+ Resa
71.35
91.14
45.64
94.60
86.97
39.74
48.55
22.90
25.91
26.81
VCM (MS) Ring
70.32
91.30
56.
94.80
76.45
46.04
47.56
21.30
23.47
25.98
Our T2V-Turbo (MS)
74.47
93.34
58.63
95.80
89.67
45.74
48.47
23.23
25.92
27.51
VCM (VC2)
55.66
63.97
10.81
82.60
79.12
23.06
18.49
25.29
22.31
25.15
VCM (VC2) + Raid
67.55
87.77
30.38
93.00
86.90
28.81
39.07
25.75
24.65
27.57
VCM (VC2) Rung
71.70
93.13
46.20
95.00
84.
37.78
$1.34
23.65
24.62
27.75
Our T2V-Turbo (VC2)
74.76
93.96
54.65
95.20
89.90
38.67
55.58
24.42
25.51
28.16
Table 3:
Effect of different choices of Rvid- Our T2V-Turbo can always outperform VCM Ring with either ViCLIP or Intern Vid2 S2 as Rvia. Table 5 5| Appendix [ includes further details.
T2V-Turbo (VC2)
T2r-Turbo (VC2)
T2V-Turbo (MS)
T2V-Turbo (MS)
Rys = ViCLIP
Rvia InternVid S2
Rvia = VICLIP
Rysa Intern Vid
Total Score Quality Score
80.92
81.01
80.62
79.90
82.77
82.57
82.15
82.27
Semantic Score
73.52
74.76
74.47
70.41
4.3
Ablation Studies
We are interested the effectiveness of each RM, and especially in the impact of the video-text RM Rvia. Therefore, we ablate Ring and Rvia and experiment with different choices of Rvia- In Appendix we further experiment with different choices of Ring.
Ablating RMs Rimg and Ruid. Recall that the training of our T2V-Turbo incorporate reward feedback from both Ring and Rvid. To demonstrate the effectiveness of each individual RM, we perform ablation study by training VCM (VC2) + Rys and VCM (VC2) + Ring. which only incorporate feedback from Reja and Ramg. respectively. Again, we evaluate the 4-step generations from different methods on VBench. Results in Table 2 shoow that incorporating feedback from either Rang or Res leads performance improvement over the baseline VCM. Notably, optimizing Rang alone can already lead to substantial performance gains, while incorporating feedback from Rvid can further improve the Semantic Score on VBench, leading to better ext-video alignment. In Appendix H we qualitatively compare the videos generated by our T2V- and VCM Ring. corroborating the effectiveness of our mixture of RMs design.
Effect of diff'erent choices of R We investigate the impact of different choices of Ryid by training T2v-Turbo (VC2) and Tur-Turbo (MS) by setting Riju as ViCLIP (Wang al.) 2023d and the second stage model of Intervideo2 (Intern Vid2 S2). In terms of model architecture, VICLIP employs the CLIP [Radford al.) 2021] text encoder while Intern Vid2 S2 leverages the BERT-large [Kenton] and Toutanoval 2019] text encoder. Additionally, Intern Vid2 S2 outperforms VICLIP in several zero-shot video-text retrieval tasks. As shown in Table 3 T2V-Turbo (VC2) can achieve decent performance on VBench when integrating feedback from either VICLIP or InternVid2 S2, Conversely, T2V-Turbo (MS) performs better with VICLIP (Wang et al. 2023d]. Nevertheless, with Intern Vid2 S2, our T2V-Turbo (MS) still surpasses VCM (MS) Ring.
8

---

5 Related Work
piffusion-based T2V Models. Many diffusion-based T2V models rely on large-scale image datasets
for training (Ho et al. 2022a Wang al. 2023c Chen et al. [023] inherit weights from pre-trained text-to-image (T21) models [Zhang et al., 2023; Blattmann et al.
2023 Khachatryan et al. 2023].
The scale of text-image datasets [Schuhmann et al. [022] is usuall lly more than ten times the scale of open-sourced video-text datasets [Bain et al.] [022]] Wang et al. 2023d] and with higher spatial resolution and diversity [Wang et al. [023c]]. For example, Imagen Video [Ho et al.) [022b] discovers that joint training on a mix of image and video datasets improves the overall visual quality and
enables the generation of videos in novel styles. Models trained with WebVid- [Bain al. 2021
like ModelScope' single-frame video, and use them to improve video qualities. Vaei [Wang et al. 2023b) initialize the training with Vid-id-I0M and LAION-5B and then continue the training with a curated internal dataset of 23M videos. To overcome the data scarcity of high-quality videos, VideoCrafter2 (Chen] et al. 2024) proposes to disentangle motion from appearance at the data level so that it can be trained on high-quality images and low-quality videos. The data limitation of high-quality videos and aligned, accurate video captions has been longstanding bottleneck of current T2V models. In this paper, we propose to combat this challenge by leveraging reward feedback from a mixture of RMs.
[Wang et al.) 2023e] or VideoCrafter [Chen et al. [023] also treat images as a
Accelerating inference of Diffusion Models. Various methods have been proposed to accelerate the sampling process of DM, including advanced numerical ODE solvers (Song al. 2020b Lu et al., 2022a b] Zheng et al. [022] Dockhorn et 2022 Jolicceur-Martinaauu et al. [022 and distillation
techniques [Luhman and Luhman| 2021] Salimans and Ho| 2021 Meng et al. 2023 Zheng et al.
2023]. Recently, Consistency Model [Song et al., 2023] Luo et al. 2023a is proposed to facilitate
fast inference by learning a consistency function to map any point at the ODE trajectory to the origin.
Li et al. [2024) proposes to augment consistency distillation with an objective to optimize image-text
RM to achieve fast and high-quality image generation. Our work extends it for T2V generation. incorporating reward feedback from both an image-text RM and video-text RM.
Vision-and-language Reward Models. There have been various open-sourced image-text RMs that are trained to mirror human preferences given a text-image pair, including HPS (Wu et al. 2023b|a]. Reward [Xu et al. 2024a], and PickScore [Kirstain al. 2024), which are obtained by finetuning a image-text foundation model such as CLIP [Radford et al. 2021 and BLIP [Li al. 2022). on human preference data. However, to the best of our knowledge, no video-text RMs, e.g.. T2VScore (Wu et al. 2024]. that mirrors human preference on a text-video pair has been released to the public. In this paper, we choose HPSv2.1 as our image-text RM and directly employ the video foundation models ViCLIP [Wang et al.) 2023d] and Inter Vid S2 [Wang et al., 2024) that are trained for general video-text understanding as our video-text RM. Empirically, we show that incorporating feedback from these RMs can improve the performance of our T2V-Turbo.
Learning from Human/AI Feedback has been proven as effective way to align the output from a generative model with human preference [Liike et al. 2018 Ziegler et al.) 2019 Ouyang et al., 2022,
Stiennon et al. 2020 Rafailov al.) 2024| Xu et al.) [2024b] In the field of image generation,
various methods have been proposed to align a texl-to-image model with human preference, including
RL (Sutton and Barto
2018 Li et al. 2020 2023a] based methods [Fan et al. 2024 Prabhudesai
et al.
2023 Zhang et al. 2024] and back packpropagation-basee reward finetuning methods [Clark et al.
2023
Xu et al. 2024a| Prabhudesai et al. 2023). Recently, InstructVideo Yuuan et al. 2023 extends
methods to optimize a T2V model. However, it still employs an image-text RM to provide reward feedback without considering the transition dynamic of the generated video. In contrast, our work incorporates reward feedback from both an image-text and video-text RM, providing comprehensive feedback to our T2V T2V-Turbo.
the reward-finctune
Conclusion and Limitations
In this paper, we propose T2V-Turbo, achieving both fast and high-quality T2V generation by breaking the quality bottleneck of a VCM. Specifically, we integrate mixed reward feedback into the VCD process of a teacher T2V model. Empirically, we illustrate the applicability of our methods by distilling T2V-Turbe (VC2) and T2V-Turbo (MS) from VideoCrafter2 (Chen et al. 2024 and ModelScope (Wang et al. 2023c], respectively. Remarkably, the 4-step generations from both our -2V-Turbo outperform SOTA methods VBench (Huang et al.) 2024], even surpassing their

---

teacher T2V models and proprietary systems including Gen-2 Esser et al. 2023] and Pika (Pika
Labs 2023]. Our human evaluation further corroborates the results, showing the 4-step generations from our T2V-Turbo are favored by humans over the 50-step DDIM samples from their teacher, which represents over ten-fold inference acceleration with quality improvement.
While our T2V-Turbo marks critical advancement in efficient T2V synthesis, it is important to recognize certain limitations. Our approach utilizes mixture of RMs, including video-text RM Rvid. Due to the lack of an open-sourced video-text RM trained to reflect human preferences on video-text pairs, we instead use video foundation models such as ViCLIP [Wang et al., 2023d and Intern S2 [Wang et al. 2024] as our Rvia- Although incorporating feedback from these models has enhanced our T2V PEV-Twe Turbo performance, future research should explore the use of more advanced Rvia for training feedback, which could lead to further performance improvements.
Acknowledgement
The work funded by unrestricted gift from Google, and we are grateful for their generous sponsorship. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the sponsors' official policy, expressed inferred.
References
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024.
Xiang Wang. Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint arXiv:2312.09100 2023a.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guice -guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7346-736675 7356, 2023.
Pika Labs. Accessed september 25, 2023, 2023. URL https://:/www wika. htth:pp:://ww..p....rr.. art/
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256 2266-2265. PMLR, 2015.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
James Betker, Gabriel Goh, Li Jing, TimBrooks, Jianfeng Wang, Linjie Li, LongOuyang, Jun- tangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu. Yunxin- Jiao, and Aditya Ramesh. Improving image generation with better captions. 2023. URL
https: //api. semanticschool. org/ warg/Corpuslin 26440403224
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pultern recognition, pages 10684-10695, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.08 032006, 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 15:36479-36499 2022.
10

---

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.1445 14792, 2022.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022a.
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint 2rXhv:2211.1322. 2022.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint 2xXiv:2309.15003 2023b.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai com/research/ video- liteeo-gnnerationaee ceterrattioonndeess--as as worrddssimulttrr
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 2023.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308..665 06571, 2023c.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. IEEE International Conference on Computer Vision, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. International conference on machine learning, 2023.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2023.
Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models:
Synthesizing high-resolution images with few-step inference. arXiv preprint 10X0422310.04378 2023a.
Hangjie Yuan, Shiwei Zhang, Xiang Wang. Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. arXiv preprint 1rXiv:2312.12-900 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024a.

---

Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprini 2r 1440, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020a.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14,661111110011 2013.
arXiv preprint
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020b.
Xiaoshi Wu, Yiming Hao, Kegiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.000 09341, 2023a.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao Lcm-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.00 1.55556, 2023b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvide02: Scaling video foundation models for multimodal video understanding. arXiv preprini arXiv:2200. 2403.1337. 2024.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint ar S1Vi:2337.00644 2023d.
Open-Sora. Open-sora: Democratizing efficient video production for all, 2024. URL. https
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale
pretraining for text-to-video generation via transformers. In The Kleventh International Conference on Learning Representations, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-IILT, pages 4171-4186, 2019.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22563-22575 2023.
12

---

Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zeno zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15954-15964 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 105225778-252949 2022.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 2022b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 2.00002 2022b.
Huangjje Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion- based adversarial auto-encoders. arXiv preprint arXiv:2202.06771, 2022.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 11.30150-30166 2022.
Alexia Jolicoeur-Martineaaa Ke Li, Rémi Piché- Taillefer, Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint (itv:2101.0233000 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pages 42390-42402. PMLR, 2023.
Jiachen Li. Weixi Feng, Wenhu Chen, and William Yang Wang. Reward guided latent consistency distillation. arXiv preprint 2xXx:2440311122, 2024,
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2096-2105, 2023b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image
pre-training for unified vision-language understanding and generation. ICMI., 2022.
Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao et al. Towards a better metric for text-to-video generation. arXiv preprint arXiv:2401 240110777771111 2024.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent 2018.
alignment via reward modeling: research direction. arXiv preprint
13

---

Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (et.::999900999 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Wenda Xu, Jiachen Li, William Yang Wang, and Lei Li. Bpo: Supercharging online preference learning by adhering to the proximity of behavior LLM. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024b.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. Advances in neural information processing systems, 33:6197-666 6210, 2020.
Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. Of- fline reinforcement learning with closed-form policy improvement operators. In International Conference on Machine Learning, pages 20485-20525 PMLR, 2023a.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances Neural Information Processing Systems, 36, 2024.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401.1222 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
Zhen Li, Zuo-Liang Zhu, Ling- Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9801-9810 10, 2023b.
Zachary Teed and Jia Deng. Raft:
Recurrent all all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVE international conference on computer vision, pages 5148-5157, 2021.
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.002880 2022.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efffcien video foundation models. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 19948-19960, 2023c.
14

---

Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compeench: A compre- hensive benchmark for open-world compositional text-toiimage generation. Advances in Neural Information Processing Systems, 36.78723-78747 2023a.
Xinyu Huang, Youcai Zhang. Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text Guiding vision-language model via image tagging. In The Twelfth International Conference Learning Representations, 2023b.
15

---

Appendix
A Experiment and Hyperparameter (HP) Details
When performing qualitative comparisons between different methods, ensure to the same random seed for head-to-head video comparisons.
As mentioned in Sec. we train T2V-Turbo (VC2) and T2V-Turbo (MS) by distilling from the teacher diffusion-barrd T2V models VideoCrafter2 [Chen et al. [024] and the less advanced ModelScopeeT2V (Wang et al. 2023c), respectively. Specifically, VideoCrafter2 supports video FPS as an input and generates videos at the resolution of 512x320. For simplicity, we always set FPS to 16 when distilling our T2V-Turbo (VC2) from VideoCrafter2. On the other hand, ModelScopeT22V always generates video at 8 FPS at resolution of 256x256.
We conduct our training with the WebVidI0M [Bain et al. 2021 datasets. Note that both teacher T2V models are also trained with Web WedVid. We train our models for 10K gradient steps with 6 NVIDIA A 100 GPUs without gradient accumulation and set the batch size to I for each GPU device.
That is, we load video with 16 frames. At each training iteration, we always sample 16 frames from the input video. We employ HPSv2. [Wu et al. 2023a] as our image-text RM Ring When distilling from VideoCrafter2, we utilize the 2nd Stage model of Intern Video2 [Wang et al.) 2024] as our video-text RM Rvia. When distilling from ModelScopeT2V,
set Rvia to be ViCLIP Wang al., 2023d]. To optimize Jing (8). we randomly sample 6 frames from the video by setting
For the hyperparameters, we set learning rate le 5 and guidance scale range [Wmin: Wmmx] = 5, 15
We use DDIM [Song et al.) 2020b] as our ODE solver and set the skipping step = 20. For
T2V-Turbo (VC2), we set Ping 1 and Bvia 2. For T2V-Turbo (MS), set Simg 2 and 3rid =3
As mentioned in Sec. we employ the DDIM [Song et al. 2020b ODE solver DDIM by following 2023a]. Its formula from intk to t,, is given below
the practice Luo et al.
Ot
Ben |Bant Css,,
-1) Ess (Zentu C, butk)
V DDIM (Ztwsu: b In,
Zlnth
2tath
alatk
Ctap
Ben
DDIM Estimated
(11)
where éy denotes the noise prediction model from the teacher T2V model. We refer interested readers the original LCM paper [Luo al. 2023a] for further details.
16

---

B Psudoe-codes for Training our T2V-Turbo
We include the pseudo-codes for training our T2V-Turbo in Algorithm highlight the difference from the standard (latent) consistency distillation
[
We use the red color to
Luo et al. 2023a Song
al.
2023
Algorithm 1
Training Pipeline
Require: text-video dataset D, initial model parameter 0, learning rate 1, ODE solver V distance metric d, EMA rate jr, noise schedule (t)), 3(1), guidance scale (iiin Wmax), skipping interval
VAE encoder &, decoder D, image-text RM Ring. video-text RM Rvia, reward scale Pung and Byia- E(x:), (x,
Encoding training data into latent space: D= ((2,c)
(2,, c) E D
0- 5- 0
repeat
Sample (2) ~ D2, U - k] and [Wmin Wmax
Sample Statk N (a (tn+k)=; 2; [t)+x)])
2WWW Ztnte WW (Zta4x Entks tni c) wilh (Zenpar (ntks Eny 2)
&o = =D(fo (Zenaxi W.C,ttnte
Jung (0) = Exo.c Em=i Ring (80) c)
(0) Ex (Rvid (x0,
LCD = d (fo (Zin e, Entt) ff (2th " c, tn
(0, ; V) < T.CD - -Big Jing (0) (1))
A t 0 Vo (0,0-)
stop-grad (n6 + - -}]
unfil convergence
17

---

C Further Details about VBench
We provide brief introduction of the metrics included in VBench [Huang et 2024 followed by introducing the derivation rules for the Quality Score, Semantic Score and Total Score. We refer interested readers to read the VBench paper for further details.
The following metrics are used to construct the Quality Score.
Subject Consistency (Subject Consist.) calculated by the DINO (Caron al. 2021
feature similarity across video frames.
Background Consistency (BG Consist.) is calculated by CLIP Radford et al. 2021 feature similarity across video frames.
Temporal Flickering (Temporal Flicker.) is computed by the mean absolute difference across video frames.
Motion Smoothness (Motion Smooth.) is evaluated by motion priors in the video frame interpolation model [Li et al. 2023b].
Aesthetic Quality is calculated by mean of aesthetic scores evalauted by the LAION
aesthetic predictor [Schuhmann et al. 2022].
Dynamic Degree is calculated using RAFT [Teed and Deng 2020
Image Quality is evaluated by the MUSIQ Ke et al. 2021 image quality predictor.
Quality Score is calculated as the weighted sum of the normalized scores of each metric mentioned above. The weight for all metrics is 1, except for Dynamic Degree, which has weight of 0.5.
The following metrics used construct the Semantic Score.
Object Class is calculated by detecting the success rate of generating the object specified
by the using GRiT [Wu et al.
2022
Multiple Object is calculated by detecting the success rate of generating all objects specified in the prompt using GRiT [Wu et al. 2022].
Human Action is evaluated by the UMT model [Li al. 2023c
Color is caleulated by comparing the color caption generated by GRiT (W et al. 2022
against the expected color.
Spatial Relationship (Spatial Relation.) calculated by rule-based method similar
to [Huang et al., 2023a
Scene is calculated by comparing the video captions generated by Tag2Text Huuan et al. 2023b] against the scene descriptions in the prompt.
Appearance Style (Appear Style.) calculated by using VICLIP [Wang et 2023d to compare the video feature and the style description in the user prompt.
Temporal Style is calculated based on the similarity between the video feature and the style descrption feature provided by ViCLIP [Wang al. 2023d].
Overall Consistency (Overall Consist.) is calculated based on the similarity between the video feature and the entire text prompt feature provided by ViCLIP (Wang et al.) 2023d).
ViCLIP [Wang et al. 2023d)
Semantic Score is simply calculated as the mean the normalized scores of each metric mentioned above. And the Total Score is the weighted sum of Quality Score and Semantic Score, which is given by
4 Quality Score + Total Score
Total Score
(12)
18

---

Table 4: Automatic Evaluation on VBench (Huang et al., 2024 We compare our T2V-Turbo (VC2) and T2V-Turbo (MS) with baseline methods across the 16 VBench dimensions. A higher score indicates better performance for a particular dimension. We bold the best results for each dimension and underline the second-best result. Quality Score is calculated with the 7 dimensions from the top table. Semantic Score is calculated with the 9 dimensions from the bottom table. Total of Quality Score and Semantic Score. Both our T2V-Turbo (VC2) and T2V-Turbo (MS) surpass all baseline methods with inference steps terms Total Score, including the proprietary systems Gen-2 and Pika.
Score weighted
Total Score
Quality Score
Subject Consist. Consist.
BG
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image
Models
Quality
Cog Video
67.01
72.06
92.19
96.20
97.64
96.47
38.18
42.22
41.03
VideoCrafter0.9
73.02
74,91
86.24
97.60
91.79
44.41
89.72
57.22
ModeIScopeT2V
75.75
78.05
89.87
95.29
98.28
95.79
$2.06
66.39
58.57
Open-Sora
75.91
78.82
92.09
97.39
98.41
95.61
48.61
61.51
LaVie
77.08
78.78
91.41
97.47
98.30
96.38
54.94
49.72
61.90
LaVie--nterpolation
77.12
79.07
92.00
97.33
98.78
97.82
54.00
46.11
59.78
Show-
78.93
80.42
95.53
98.02
99.12
98.24
57.35
44.44
58.66
VideoCrafterl
79.72
81.59
95.10
98.04
98.93
95.67
62.67
55.00
65.46
Pika
80.40
82.68
96.76
98.95
99.77
99.51
63.15
37.22
62.33
VideoCrafter2
80.44
82.20
96.85
98.22
98.41
97.73
63.13
42,50
67.22
Gen-2
80.58
82.47
97.61
97.61
99.56
99.58
66.96
18.89
67.42
VCM (MS)
75.84
78.80
93.06
97.30
98.51
98.00
48.99
46.11
61.98
Our T2V-Turbo (MS)
80.62
82.15
94.82
98.71
97.99
95.64
60.04
66.39
68.09
VCM (VC2)
73.97
78.54
94.02 96.28
96.05
99.06
98.84
54.56
42,50
52.72
Our T2V-Turbo (VC2)
81.01
82.57
97.02
97.4
97.34
63.04
49.17
72.49
Semantic
Object Class
Multiple Objects
Human Action
Spatial Relation.
Appcar. Style
Temporal Style
Overall
Models
Color
Scene
Score
Consist.
CogVideo
46.83
73.40
18.11
78.20
79.57
18.24
28.24
22.01
7.80
7.70
VideoCrafter0.9
65.46
87.34
25.93
93.00
78.84
36.74
43.36
21.57
25.42
25.21
ModeIScopeTI2
66.54
82.25
38.98
92.40
81.72
33.68
39.26
23.39
25.37
25.67
Open-Sora
64.28
74.98
33.64
85.00
78.15
43.95
37.33
21.58
25.46
26.18
LaVie
70.31
91.82
33.32
96.80
86.39
34.09
52.69
23.56
25.93
Vnterlooterooltiion
69.31
90.68
80.80
30.06
52.62
23.53
26.01
Show-1
72.98
93.07
45.47
95.60
86.35
$3.50
47.03
23.06
25.28
27.46
VideoCrafterl
72.22
78.18
45.66
91.
93.32
58.86
43.75
24.41
25.54
26.76
Pika
71.26
87.45
46.69
88.00 38.00
85.3
65.65
44.80
21.89
24.44
25.47
VideoCrafter2
73.42
92.55
40.66
95.00
92.92
35.86
55.29
25.13
25.84
28.23
Gen-2
73.03
90.92
$5.47
89.20
89.49
66.91
48.91
19.34
24.12
26.17
VCM (MS)
63.98
83.18
24.85
87.20
85.72
31.57
42.44 44
23.20
23.30
24.18
Our T2V-Turbo (MS)
74.47
93.34
58.63
95.80
89.67
45.74
48.47
23.23
25.92
27.51
VCM (VC2)
55.66
63.97
10.81
82.60
79.12
23.06
18.49
25.29
22.31
25.15
Our T2V-Turbo (VC2)
74.76
93.96
$4.65
95.20
89.90
38.67
55.58
24.42
25.51
28.16
19

---

D Human Evaluation Details
Figure 3 shows the user interface displayed the labelers when conducting our human evaluations. Each method generate videos of 16 frames using the 700 prompts from EvalCrafter [Liu et al. [202]] For our T2V-Turbo (VC2), we collect its 4-step and 8-step generations and compare them with the 50-step DDIM samples from its teacher VideoCrafter2. For our T2V-Turbo (MS), we collect its 2-step and 4-step generations and compare them with 50-step DDIM samples from teacher ModelScopeTT2V. We also compare the 4-step generations between our T2V-Turbo their baseline VCM, demonstrating the significant quality improvement our methods.
As mentioned in Sec. 4.2 we hire labelers from Amazon Mechanical Turk platform and form the video comparison task many batches of HITs. Specifically, we choose labelers from English- speaking countries, including AU, CA, NZ, GB, and the US. Each task needs around 30 seconds to complete, and we pay each submitted HIT with 0.2 US dollars. Therefore, the hourly payment is about 24 US dollars.
We note that the data annotation part of our project is classified as exempt by Human Subject Committee via IRB protocols.
You ss as
titte that mor apoesing.
the
Reass compan thair quality basedd quartion.
inrsucter Across son ecctie matting pesooaaee
Cuadlos Aiss shedl
d rnd sbsc ponart A 0
Figure 5: User interface of our human evaluation experiments.
20

---

E Additional Ablation Studies
which can be found in We further examine the effect of different choices of Ring- In the initial stage of our project. we tram VCM (VC2) + Rimg with several different image-text RMs, including HPSv2.1 [Wu et al. 2023a PickScore [Kirstain et al., [024], and ImageReward [Xu et al.) 2024a]. We collect the 4-step generations from each method and qualitatively compare them with the 4-step generation from baseline VCM. As shown in Figure 6 incorporating reward feedback from any of these Rimg leads to quality improvement over the baseline VCM (VC2). It worth noting that HPSv2. and PickScore fine-tuned from CLIP with human preference data. Therefore, learning from CLIP might also lead to better performance than the baseline VCM.
In this section, we provide the full ablation results performed in Table 3
Table 5
Table 5: Effect of different choices of Rvid- T2V-Turbo (VC2) can achieve decent performance on VBench when integrating feedback from either ViCLIP or Intern Vid2 S2. On the other hand, T2V- (MS) achieves better result with ViCLIP Wann et al.
2023d].
Models
Total Score
Quality Score
Subject Consist.
BG
Temporal Flicker.
Motion Smooth.
Aesthetic Quality
Dynamic Degree
Image Quality
Consist.
T2V-Turbo (MS), Riia VICLIP
80.62
82.15
94.82
98.71
97.99
95.64
60.04
66.39
68.09
T2V-Turbo (MS), R.i internVid22 $2
79.90
82.27
96.68
99.36
97.74
95.66
65,30
52.22
68.23
T2V-Turbo (VC2), R.a VICLIP
80.92
82.77
96.93
97.47
98.03
97.48
63.38
43.61
72.94
T2V-Tubo (VC2), Rya Vnder $2
81.01
82.57
96.28
97.02
97.48
97.34
63.04
49.17
72.49
Semantic
Object Class
Muttiple Objects
Human
Spatial Relation.
Scene
Appear. Style
Temporal Style
Overall
Score
Action
Consist.
Turho (MS), R.g VICIIIP
45.74
T2V-Turbo (MS), R.
InternVid2 S2
70.41
94.05
48.73
92.60
81.69
45.41
48.15
21.45
23.84
26.24
T2V-Turbo (VC2), R.u VICLIP
73.52
94.05
50.52
94.40
89.85
36.77
54.17
23.81
25.34
28.11 11 28.16
T2V-Turbo (VC2), R.,
InterVVi22
74.76
95.20
38.67
24.42
25.51
F Qualitative Results
We provide additional qualitative comparisons between our T2V-Turbo, the baseline VCM, and their teacher T2V models in Figures 7 and 10
The prompts for the top two and bottom two rows in Figure ! are given below:
With the style of low-poly game art, A majestic, white horse gallops gracefully across moonlit beach.
Kung Fu Panda posing in cyberpunk, neonpunk style.
G Broader Impact
The ability to create highly realistic synthetic videos raises concerns about misinformation and deepfakes, which can be used manipulate public opinion, defame individuals, perpetrate fraud. Addressing these concerns requires robust regulatory frameworks and ethical guidelines to ensure the technology is used responsibly and for the benefit of society. Therefore, installing safeguard when releasing our models. Specifically, we will require users to adhere to usage guidelines.
committed to
Despite the challenges, the impact of our T2V-Turbo is profound, offering a scalable solution that significantly enhances the accessibility and practicality of generating high-quality videos at remarkable speed. This innovation not only broadens the potential applications in fields ranging from digital art to visual content creation but also sets new benchmark for future research in T2V synthesis, emphasizing the importance of human-centric design in the development of generative AI technologies.
21

---

a caf drinking beer
VCM (VC2)
VCM (VC2) HPSv2,1
VCM (VC2) + PickScore
VCM (VC2) ImgRwd
a dog wearing vr goggles on a boat
VCM (VC2)
VCM (VC2) HPSv2,1
VCM (VC2) PickScoree
VCM (VC2) ImgRwd
With the style of low- low-poly game art, A majestic, white
horse gallops gracefully across a moonlit beach.
VCM (VC2)
VCM (VC2) HPSv2,1
VCM (VC2) PickScore
VCM (VC2) ImgRwd
Figure 6: Ablation study on the choice of the methods. The three Ring we tested can all improve the video generation quality compare to baseline VCM (VC2).
Ring- We compare the 4-step generations from each
22

---

A wise torfoise in a tweed hat and spectacles reads a
newspaper, Howard Hodgkin style
VCM (MS) (4-step)
VideoCrafter2 (50-step)
Turbo
(VC2) 4-step
T2V-Turbo
(VC2) 8-step
drove viewpoint, fireworks above the Parthenon
VCM (MS) (4-step)
VideoCrafter2 (50-step)
T2V-Turbo (VC2) 4-step
T2V-Turbo (VC2) 8-step
Iron Man is walking towards the camera in the rain at night, with lot of fog behind him. Science fiction movie. close-up
VCM (MS) (4-step)
VideoCrafter2 (50-step)
T2V-Turbo
(VC2) 4 steep
T2V-Turbo
(VC2) 8-step
Figure 7: Additional qualitative comparison results for our T2V-Turbo (VC2).
23

---

With the style of sketch, A sophisticated monkey in beret and striped shirt paints in French artist's studio.
VCM (MS) (4-step)
VideoCrafter2 (50-step)
T2V-Turbo (VC2) 4-step
T2V-Turbo (VC2) 8-step
a cyborg standing on top of a skyscraper, overseeing the city, back view, cyberpunk vibe, 2077. NYC, intricate details, 4K
VCM (MS) (4-step)
VideoCrafter2 (50-step)
T2V-Turbo (VC2) 4-step
T2V-Turbo (VC2) 8 8-step
Egyptian tomp hieroglyphics painting of A regal lion, decked out in jeweled crown, surveys his kingdom.
VCM (MS) (4-step)
VideoCrafter2 (50-step)
T2V-Turbo
(VC2) 4-stap
T2V-Turbo
(VC2) 8-step
Figure 8: Additional qualitative comparison results for our T2V-Turbo (VC2).
24

---

Macro len style, A tiny mouse in dainty dress holds a parasol to shield from the sun.
VCM (MS) (4-step)
Shuft ook shutter sk shutter R sk shutt stock shutters
ModeScopeT2V (50-step)
(MS) s-step
T2V T2V- Turbo
(MS) 8-step
T2V- Turbo
pop art style, A photo of an astronaut riding a horse in the forest. There is river in front of them with water lilies.
VCM (MS) (4-step)
ModeScopeT22 (50-step)
(MS) 4 1-step
T2V T2V-Tuh Turbo
(MS) 8-step
T2V Turbo
Figure 9: Additional qualitative comparison results for T2V-Turbo (MS).
25

---

Mickey Mouse is dancing on white background
VCM (MS)
(4-step)
shut skul uck shu uck shut ok shu ock
ModeScope T2V
(50-step)
she
shu ckksh
Skkshh sck
4-step
T2V- T2V-Tuue Turbo
shut
sckishu bcck Lock
sck tock
(MS)
8-step
T2V- T2V-Turbo Turbo
shut
sckth ockk shut cck
sck sck
(MS)
man looking at a distant mountain in Sci-fi style
VCM (MS)
(4-step)
ModeScope
(50- step)
4-step
T2V T2V-Tuse Turbo
(SS)
(MS) 8-step
T2V Turbo
Figure 10: Additional qualitative comparison results for our T2V-Turbo (MS).
26

---

H
Comparing videos generated by T2V-Turbo and VCM + Rimg
Please click to play videos in Adobe Acrobat.
Prompt: panda standing on a surfboard in the ocean in sunset.
T2V-Turbo (VC2)
VCM (VC2) Rimg
Analysis The panda on the right is instead sitting on the surfboard.
Prompt: raccoon is playing the electronic guitar.
T2V-Turbe (VC2)
VCM (VC2) Rimg
Analysis The right video fails on playing the electronic guitar.
Prompt: A motorcycle accelerating to gain speed.
T2V-Turbo (VC2)
VCM (VC2) + Ring
Analysis The motorcycle on the right actually moving backward.
27

---

Prompt: squirrel eating a burger.
T2V-Turbo (VC2)
VCM (VC2) Ring
Analysis The squirrel on the right looks more like is holding burger.
Prompt: Mars rover moving on Mars.
T2V-Turbo (VC2)
VCM (VC2) Ring
Analysis The hills on the right in the background also move.
Prompt: A horse galloping across an open field.
T2V-Turbo (VC2)
VCM (VC2) Ramg
Analysis Another horse suddenly runs into the scene of the right video.
28

---

Prompt: A black vase.
T2V-Turbo (VC2)
VCM (VC2) Ring
Analysis The right video shows two vases instead of one.
Prompt: Happy dog wearing a yellow turtleneck, studio, portrait, dark background.
T2V-Turbo (VC2)
VCM (VC2) Ring
Analysis The dog on the right doesn't 't look happy.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made sure to accurately illustrate our main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We address the limitation of work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have described our experimental settings in detail Sec. and Appendix A We will also release the codes and models.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: We include our codes in the supplementary materials. We have released our models and codes in httpps//ggithu wttp://iaceeennitt/ee--------e
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental details in both Sec. 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We only have the computational resources to run the training for one time. And thus do include error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the information in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarrrsccccoooaaa//aaaaaa
Answer: [Yes]
Justification:
We conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of our work in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have discussed this in Appendix G
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have make sure that we have followed all the rules.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided detailed usage guidance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
We include further details including full text of instructions given to participants and screenshots in
Justification: We performance human evaluation on our methods in Sec. 4.2
Appendix D
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The data annotation part of the project classified as exempt by Human Subject Committee via IRB protocols.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "X9WuwliLcz", "reviewer": "Reviewer_niV6", "summary": "The paper proposes T2V-Turbo, a model aiming to achieve both fast and high-quality text-to-video generation by breaking the quality bottleneck of a video consistency model (VCM). It integrates mixed reward feedback from one image and one video reward model into the consistency distillation process of a teacher T2V model. The 4-step generations from T2V-Turbo outperform SOTA methods on the VBench benchmark and are favored by humans over the 50-step DDIM samples from the teacher model, achieving over ten-fold inference acceleration with quality improvement.", "questions": "- In Line 228, the authors write “InternVid2 S2 outperforms ViCLIP in several zero-shot video-text retrieval task”.  While in Table 1, we know that ModelScopeT2V (MS) has less total score than VideoCrafter2 (VC2), which means that MS is a relatively weaker model than VC2, then why choose ViCLIP as the video RM for MS rather than using a stronger InternVid2 S2?", "limitations": "The authors have adequately addressed the limitations of the work in the paper. They acknowledge the lack of an open-sourced video-text reward model trained to reflect human preferences on video-text pairs and discuss the potential use of a more advanced video reward model in the future. They also mention the concerns about misinformation and deepfakes raised by the ability to create highly realistic synthetic videos and commit to installing safeguards when releasing the models, such as requiring users to adhere to usage guidelines.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and clearly presents the problem, the proposed method, the experimental setup, and the results. The use of figures and tables helps to illustrate the concepts and results.\n- The automatic evaluation results on the VBench benchmark and human evaluation results with the 700 prompts from EvalCrafter demonstrate the effectiveness of the proposed method, with T2V-Turbo outperforming baseline methods and proprietary systems in terms of total score and human preference.\n- The ability to generate high-quality videos quickly has significant impacts in various fields, such as digital art and visual content creation, and sets a new benchmark for future research in T2V synthesis.", "weaknesses": "- The method mainly combines the consistency distillation in the Video Consistency Model (VCM) with multiple reward models. Although it has achieved good results in solving the existing problems of the T2V model, this combination is relatively conventional and may be somewhat lacking in originality.\n- The citation format of this paper should adhere to the NeurIPS standard, the authors misuse \\citet throughout the paper, especially in Sections 2 and 5, making it hard to read smoothly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2nuI07bU9m", "reviewer": "Reviewer_x94N", "summary": "The paper introduces T2V-Turbo to enhance the quality of video consistency models in text-to-video generation. The authors address the slow sampling speed of diffusion-based T2V models and the low quality of generated video by integrating feedback from a mixture of differentiable reward models into the consistency distillation (CD) process of a pre-trained T2V model. This integration allows for the optimization of single-step generations, bypassing the memory constraints of backpropagating gradients through iterative sampling processes. T2V-Turbo demonstrates significant improvements in both speed and quality, achieving high performance on the VBench benchmark and surpassing leading models such as Gen-2 and Pika.", "questions": "- Regarding reward models for video generation. Could the author provide more details about how the reward models should be designed in order to obtain better video generation or evaluating video generation models?\n- It seems like only consistency distillation for VC2 degrades the performance (Table 1), while the score remains the same for ModelScope. What if we do not use consistency distillation, and only use reward feedback learning for VC2 or MS? Then this may have a higher score than T2V-Turbo? Suppose it is true, then what if we distill after the feedback learning? To summarize my points, does the joint training of feedback learning and consistency distillation is better than sequential fine-tuning?", "limitations": "The paper highlights the limitation of relying on existing video-text reward models that are not explicitly trained to reflect human preferences on video-text pairs. Instead, the authors use video foundation models like ViCLIP and InternVid S2 as substitutes. While incorporating feedback from these models has enhanced T2V-Turbo’s performance, the authors acknowledge that the development of more advanced video-text reward models could further improve the results. Additionally, the complexity of integrating mixed reward feedback into the CD process could hinder the broader adoption and scalability of the approach.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- **Efficiency and Quality**: T2V-Turbo achieves impressive results with 4-step generations, offering a tenfold acceleration in inference speed while improving video quality compared to 50-step DDIM samples from teacher models.\n - **Comprehensive Evaluation**: The authors conduct extensive experiments, including automatic evaluations on VBench and human evaluations with 700 prompts from EvalCrafter, to validate the effectiveness of T2V-Turbo.", "weaknesses": "- **Technical contribution**: The proposed approach is simply a mixture of previous works, i.e., consistency distillation and reward feedback learning, which deflates the technical contribution of the paper. However, I acknowledge that this is one of the pioneering approaches in text-to-video generation.\n- **Reward models**: While the proposed method heavily depends on the reward models, the reward models that the paper used are actually not designed to function as reward models. This is because of the absence of video-text reward models in comparison to image generation (as the author mentioned in their limitations). However, it is a more right approach to first design a good reward model for video generation than developing a video generation model to align with such reward models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback on our work! Please find our detailed response below.\n\n> Q1: The proposed approach is simply a mixture of previous works, i.e., consistency distillation and reward feedback learning\n\nWe emphasize that our method is NOT simply combining the video consistency model and reward feedback learning. Previous methods, such as InstructVideo [1], require backpropagating gradients through an iterative sampling process, which can lead to substantial memory costs. In contrast, our method cleverly leverages the single-step generation arising from consistency distillation. By optimizing the rewards of the single-step generation, our method avoids the highly memory-intensive issues associated with passing gradients through an iterative sampling process.\n\nAdditionally, we would like to emphasize the contribution of our mixture of RMs design, which enables us to break the quality bottleneck in VCM without access to a $\\mathcal{R}\\_\\text{vid}$ trained to mirror human preference. By learning from an image-text RM $\\mathcal{R}\\_\\text{img}$ and a video foundation model $\\mathcal{R}\\_\\text{vid}$, such as ViCLIP and InternVid2 S2, we achieve both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.\n\n[1] Yuan et al., InstructVideo: Instructing Video Diffusion Models with Human Feedback. CVPR 2024\n\n> Q2: However, it is a more right approach to first design a good reward model for video generation than developing a video generation model to align with such reward models.\n\nWe fully recognize the importance of designing a good video-text RM $\\mathcal{R}\\_\\text{vid}$. However, creating an effective $\\mathcal{R}\\_\\text{vid}$ might require long-term efforts. A video is more complex than an image due to its additional temporal dimension. We envision that it requires multiple iterations to derive an effective $\\mathcal{R}\\_\\text{vid}$. Specifically, one iteration involves 1) Training $\\mathcal{R}\\_\\text{vid}$ by collecting new preference data from a video generator and 2) Training the video generator to align with $\\mathcal{R}\\_\\text{vid}$. **Our method provides an efficient way to accomplish the second step.** \n\nOn the other hand, even without a $\\mathcal{R}_\\text{vid}$ trained to reflect human preferences on video-text pairs,  we highlight that our method can still enhance the generation quality of a T2V model by aligning it with video-text foundation models, such as ViCLIP and InternVid2 S2.\n\n> Q3: Could the author provide more details about how the reward models should be designed in order to obtain better video generation or evaluating video generation models?\n\nWe thank the reviewer for the question. In addition to our response to Q2,  we conjecture a $\\mathcal{R}_\\text{vid}$ effective for training video generators might need to be finetuned from existing video foundation models, such as ViCLIP and InternVid2 S2. This approach mirrors the success seen with image-text RMs, including HPSv2, ImageReward, PickScore, and AestheticScore, which have proven effective for training image generators. These models are finetuned from image-text foundation models, such as CLIP and BLIP, using human preference data.\n\nMoreover, learning a multi-dimensional reward to reflect fine-grained human preferences might also be helpful. For example, each dimension of the reward vector could be trained to reflect visual quality, transition dynamics, text-to-video alignment, etc.\n\n> Q4: Does the joint training of feedback learning and consistency distillation is better than sequential fine-tuning?\n\nFirst, sequential fine-tuning is computationally more expensive than joint training. According to the InstructVideo paper, their reward feedback learning phase costs more than 40 hours. If we add the distillation time cost further, the total time cost for sequential fine-tuning can easily exceed 50 hours. Conversely, our joint training only requires less than 10 hours, representing a fivefold reduction in terms of training wall-clock time.\n\nSecond, as mentioned in Sec. 1 of our paper, finetuning a diffusion T2V model towards a differentiable RM requires backpropagating gradients through the diffusion model's iterative sampling process. Therefore, calculating the full reward gradient is prohibitively expensive, resulting in substantial memory costs. Conversely, our method leverages the single-step generation that arises naturally from computing the CD loss, effectively bypassing the memory constraints.\n\n> Q5: It seems like only consistency distillation for VC2 degrades the performance (Table 1)\n\nIn terms of the performance degradation of distilling VC2, we conjecture it can be alleviated by performing full model training, or further optimizing the hyperparameters related to learning the LoRA weights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OjyO53xOLW", "reviewer": "Reviewer_MMwk", "summary": "This paper aims to achieve a video consistency model with both fast and high-quality generation. Specifically, the authors introduce T2V-Turbo, which integrates feedback from a mixture of differentiable reward models into the consistency distillation (CD) process of a pre-trained T2V model. The differentiable reward models consists of an image-text reward model and a video-text reward model. Experiment results verify the effectiveness of proposed method.", "questions": "See the weakness part.", "limitations": "The authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow and understand.\n\n- The idea is reasonable and simple.\n\n- Experiment results are good.", "weaknesses": "- The novelty is limited. The paper seems to simple combine the video consistency model and the differentiable reward models. Different from previous works, the authors utilize a mixture of reward models, where a video-text reward model is additionally added to encourage the diffusion model to better model the temporal dynamics.  However, to me, this contribution is quite incremental. \n\n- For the paper presentation, I think it is better to show what the text prompt is in Fig. 1 to help reader better understand the difference between different models. \n\n- For the quantitative comparison, it can be seen that the proposed method actually doesn't perform the best for most of the evaluation metrics. I am not sure if comparing total score (i.e., a weighted sum of Quality Score and Semantic Score) only is enough to show the superiority of proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments!\n\n>The paper seems to simple combine the video consistency model (VCM) and the differentiable reward models\n\nWe emphasize that our method is NOT a simple combination of VCM and differentiable RM. Previous works focus on aligning a pretrained DM to the preference given by RMs. In contrast to previous methods, e.g., InstructVideo, aligning a pretrained T2V model by backpropagating gradients through the memory-intensive iterative sampling process, our method cleverly leverages the single-step generation that naturally arises from consistency distillation. By optimizing the rewards of the single-step generation, our method avoids the highly memory-intensive issues associated with passing gradients through an iterative sampling process.\n\nAdditionally, our joint optimization technique is notably computationally efficient. Previous approaches, such as InstructVideo, require over 40 hours of training to align a pretrained T2V model. In contrast, our method reduces training time to less than 10 hours, achieving a fourfold increase in efficiency in wall-clock training time.\n\nEmpirically, we have shown that our approach outperforms existing methods on comprehensive benchmarks, including VBench and EvalCrafter. In contrast, previous reward learning methods for video generation, such as InstructVideo, conducted experiments on a limited set of user prompts, most of which were related to animals.\n\n> the authors utilize a mixture of reward models, where a video-text reward model is additionally added to encourage the diffusion model to better model the temporal dynamics. However, to me, this contribution is quite incremental.\n \nWe would like to clarify the contribution of our mixture of RMs design, which enables us to break the quality bottleneck in VCM without access to a $\\mathcal{R}\\_\\text{vid}$ trained to mirror human preference. By learning from an image-text RM $\\mathcal{R}\\_\\text{img}$ and a video foundation model $\\mathcal{R}\\_\\text{vid}$, such as ViCLIP and InternVid2 S2, we achieve both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika.\n\nAdditionally, our ablation study in Table 2 further provides valuable scientific insights into the effectiveness of different RMs. While leveraging feedback from $\\mathcal{R}\\_\\text{img}$ alone (VCM + $\\mathcal{R}\\_\\text{img}$) is sufficient to match the visual quality (**Quality Score**) of our T2V-Turbo, the additional feedback from $\\mathcal{R}\\_\\text{vid}$ further enhances text-video alignment, resulting in higher **Semantic Score**. Qualitative evidence is provided in the attached PDF. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.\n \n> I think it is better to show what the text prompt is in Fig. 1 to help reader better understand the difference between different models.\n\nWe thank the reviewer for the suggestion. We will include the corresponding prompts in our revised manuscript. The prompt for the top two and bottom two rows are\n1. With the style of low-poly game art, A majestic, white horse gallops gracefully across a moonlit beach.\n2. Kung Fu Panda posing in cyberpunk, neonpunk style\n\n>   For the quantitative comparison, it can be seen that the proposed method actually doesn't perform the best for most of the evaluation metrics. I am not sure if comparing total score (i.e., a weighted sum of Quality Score and Semantic Score) only is enough to show the superiority of proposed method.\n\nFirst, we emphasize that our automatic evaluation results in Table 1 are corroborated by the human evaluations in Fig 3, where the 4-step generation from both T2V-Turbo (VC2) and T2V-Turbo (MS) are preferred over the 50-step generations from their teacher VideoCrafter2 and ModelScopeT2V.\n\nSecond, we highlight that the **Total Score, Quality Score, and Semantic Score are sufficient proxies for human preference**. Consider the comparison between VideoCrafter2 and our T2V-Turbo (VC2). Figure 3 indicates that human annotators favor the 4-step generation from our T2V-Turbo (VC2) in terms of **Visual Quality**, **Text-Video Alignment**, and **General Preference**. These preferences align with the higher **Quality Score**, **Semantic Score**, and **Total Score** of our T2V-Turbo (VC2), thereby validating these metrics' effectiveness in reflecting our method's superiority.\n\nLastly, we address why our T2V-Turbo does not perform the best across all evaluation metrics yet still achieves the highest overall scores. We extracted the performance of VideoCrafter2 and our T2V-Turbo (VC2) from Table 1 of our paper. Although VideoCrafter2 scores slightly higher on 5 out of 7 dimensions constituting the **Quality Score** and 4 out of 9 dimensions constituting the **Semantic Score**, the differences are not significant. In contrast, our T2V-Turbo (VC2) significantly outperforms VideoCrafter2 in metrics such as `Dynamic Degree`, `Image Quality`, and `Multiple Objects`, contributing to its overall superiority.\n\n| Models|Total Score|Quality Score|Subject Consist.|BG Consist.|Temporal Flicker|Motion Smooth.|Aesthetic Quality|Dynamic Degree|Image Quality|\n|-|-|-|-|-|-|-|-|-|-|\n| VideoCrafter2| 80.44|82.20|**96.85**|**98.22**|**98.41**|**97.73**|**63.13**|42.50|67.22|\n| $\\texttt{T2V-Turbo}$ (VC2)|**81.01**|**82.57**|96.28|97.02|97.48|97.34|63.04|**49.17**|**72.49**|\n\n| Models|Semantic Score|Object Class|Multiple Objects|Human Action|Color|Spatial Relation.|Scene|Appear. Style|Temporal Style|Overall Consist.|\n|-|-|-|-|-|-|-|-|-|-|-|\n|VideoCrafter2|73.42|92.55|40.66|95.00|**92.92**|35.86|55.29|**25.13**|**25.84**|**28.23**|\n|$\\texttt{T2V-Turbo}$ (VC2)|**74.76**|**93.96**|**54.65**|**95.20**|89.90| **38.67**|**55.58**|24.42|25.51| 28.16|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7JArZg4np", "reviewer": "Reviewer_61hZ", "summary": "This paper presents a distillation method for text-to-video models. In short, it builds upon latent consistency models (more specifically, adapting the paper \"Reward Guided Latent Consistency Distillation\" from image to video models). The method involves the usual consistency model objective, in addition to that feedback from both an image-text reward model (e.g. HPSv2.1) and a video-text model (e.g. InternVideo) is used to enhance the quality of the generated videos. Overall, the proposed approach not only is a able to few-step video generation (4/8), but is also outperforming existing open-source video generators such as ModelScope and VideoCrafter.", "questions": "Overall, I definitely lean towards accepting the paper. Despite being \"obvious\", I think that having an open-weight model+implementation for this would definitely help research in the field. However, I would really like the authors to point out if I have missed out details in their contributions or any other novel aspects of their work, since that is the major shortcoming I see currently.", "limitations": "Limitations are sufficiently discussed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The biggest strength of the paper would definitely be the strong results. Having a 4 step model that is able to achieve results comparable/better to 50 step models is indeed commendable.\n\nIn addition to this, the evaluation seems quite thorough: there are quantitative results on V-Bench with several categories, user studies comparing against standard video models, and also ablation studies which makes the paper quite sound. \n\nFinally, the presentation of the paper is also quite clear, everything is easy to grasp and understandable.", "weaknesses": "The most obvious weakness about this paper is that it is a direct application of Li et al. (\"Reward Guided Latent Consistency Distillation\") to video models. As far as I can understand, the changes involve replacing SD2.1 with VideoCrafter/ModelScope and incorporating an additional video-text RM in addition to the image-text RM. Similarly, the main difference I can see with Wang et al. 2023 (VideoLCM) is the utilization of the reward objectives. \n\nGiven that previous works have shown the efficacy of Latent Consistency Models for video generation, and the benefits of reward models for LCM distillation in images, it's not entirely novel to combine these 2 ideas into a single distilled video model. \n\n[Minor] I don't find too many qualitative examples, and I couldn't find more videos in the supplementary (apart from the code). It would definitely be nicer if there were more examples that were provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's positive feedback on our work. Please find our detailed response below.\n\n> I would really like the authors to point out if I have missed out details in their contributions or any other novel aspects of their work.\n\nWe appreciate the opportunity to clarify our contributions. We would like to emphasize the importance of our mixture of RMs design, which enables us to achieve significant empirical results even without access to a $\\mathcal{R}_\\text{vid}$ trained to reflect human preference. \n\nSpecifically, our method leverages feedback from both an image-text RM $\\mathcal{R}\\_\\text{img}$ and a video foundation model $\\mathcal{R}\\_\\text{vid}$, such as ViCLIP and InternVid2 S2. This combination allows our T2V-Turbo to break the quality bottlenecks in the video consistency model, resulting in both fast and high-quality video generation. Notably, the 4-step generations from our T2V-Turbo achieve the SOTA performance on VBench, surpassing proprietary models, including Gen-2 and Pika.\n\nAdditionally, our ablation study in Table 2 empirically provides valuable scientific insights into the effectiveness of different RMs. While leveraging feedback from $\\mathcal{R}\\_\\text{img}$ alone (VCM + $\\mathcal{R}\\_\\text{img}$) is sufficient to match the visual quality (**Quality Score**) of our T2V-Turbo, the additional feedback from $\\mathcal{R}\\_\\text{vid}$ further enhances text-video alignment, resulting in higher **Semantic Score**. Qualitative evidence is provided in the attached PDF. To the best of our knowledge, we are the first to improve video generation using the feedback from video-text RMs. We believe that future advancements in video-text RMs will further enhance the performance of our methods.\n\nOur joint training pipeline is computationally efficient. For example, InstructVideo requires over 40 hours of training to align a pretrained T2V model. In contrast, our method reduces training time to less than 10 hours, achieving a fourfold increase in efficiency in wall-clock training time.\n\nLastly, our method is broadly applicable to a diverse set of prompts. We have empirically shown that our approach outperforms existing methods on comprehensive benchmarks, including VBench and EvalCrafter. In contrast, previous methods, such as InstructVideo, conduct experiments on a limited set of user prompts, most of which are related to animals.\n\n> I don't find too many qualitative examples, and I couldn't find more videos in the supplementary (apart from the code). It would definitely be nicer if there were more examples that were provided.\n\nThank you for your suggestions! We have now included more videos in the attached PDF. We promise to include more videos in our revised manuscript and create a website to better present the videos generated by our methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTADtzMWP9", "reviewer": "Reviewer_Kq5y", "summary": "The paper presents T2V-Turbo, a training strategy where additional reward models are introduced during the consistency distillation process to enhance the T2V consistency model's quality. With such enhancement, the trained model achieves favorable results in both VBench and human evaluations.", "questions": "1. Why use different video reward models for ModelScope and VideoCrafter?\n2. What leads to different hyperparameter choices in L164-165?", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is presented clearly and easy to follow;\n2. The supplementary material provides abundant background information and results;\n3. According to the figures, the proposed method significantly boosts the video quality.", "weaknesses": "1. Regarding the technical contribution, the proposed method seems like a straightforward extension of the video consistency model. The idea of adding direct supervision on the clean samples in consistency distillation has also been explored in previous works, e.g., [1];\n2. It's hard to evaluate the motion quality improvements according to the static frames in the paper. It's highly recommended to include the corresponding mp4 files;\n3. I wonder which reward model contributes more to quality improvement from Fig.4 row 1 and Fig.4 row 3? This cannot be inferred from Tab.2 and it's recommended to present some qualitative results in the ablation study;\n4. The paper lacks in-depth discussions on the choice and influence of the image/video reward models. For instance, would it be better to CLIP/GAN-discriminator compared to the human preference score? Why do MS and VC2 perform differently when combined with InternVidS2?  Section 4.3 only provides empirical results.\n\n[1] Adversarial Diffusion Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback!\n\n> Regarding the technical contribution, the proposed method seems like a straightforward extension of the video consistency model. The idea of adding direct supervision on the clean samples in consistency distillation has also been explored in previous works, e.g., [1] Adversarial Diffusion Distillation (ADD).\n\nWe would like to clarify that ADD is not based on consistency distillation. Instead, ADD's distillation loss corresponds to score distillation sampling [2]. Moreover, adversarial training is prone to instability [3] and might require massive hyperparameter tuning [4]. In contrast, our methods optimize towards differentiable RMs enjoying a stable training process. To the best of our knowledge, we are the first to add direct supervision on the clean samples when distilling from a video diffusion generator. Additionally, we are the first to learn video generators from the feedback of video-text models.\n\n[2] Poole et al. \"Dreamfusion: Text-to-3d using 2d diffusion.\" ICLR 2023.\n\n[3] Yue et al. \"On the algorithmic stability of adversarial training.\" NeurIPS 2021\n\n[4] Pang et al. \"Bag of tricks for adversarial training.\" ICLR 2021\n\n> It's highly recommended to include the corresponding mp4 files\n\nWe appreciate the suggestions! We promise to include all corresponding mp4 files in our revised manuscript and create a website to better present the videos generated by our methods.\n\n>  I wonder which reward model contributes more to quality improvement from Fig.4, row 1, and row 3? \n\nFor Fig. 4, the image-text RM HPSv2.1 contributes more to our T2V-Turbo's (row 3) quality improvement over the baseline VCM (row 1). In the attached PDF, we empirically show that incorporating feedback from the $\\mathcal{R}\\_\\text{vid}$ improves the video quality.\n\n> This cannot be inferred from Tab.2 and it's recommended to present some qualitative results in the ablation study\n\nFirst, we would like to clarify that Table 2 provides strong quantitative evidence for the effectiveness of each RM. For both VC2 and MS variants, leveraging feedback from $\\mathcal{R}\\_\\text{img}$ alone (VCM + $\\mathcal{R}\\_\\text{img}$) matches the **Quality Score** of our T2V-Turbo, indicating the similarly high visual quality of the generated videos. However, VCM + $\\mathcal{R}\\_\\text{img}$ still falls behind our T2V-Turbo in terms of **Semantic Score**. Further incorporating feedback from $\\mathcal{R}\\_\\text{vid}$ can bridge this gap, leading to better text-video alignment.\n\nThe attached PDF corroborates the results in Table 2 with video examples. Specifically, it compares videos generated by VCM (VC2) + $\\mathcal{R}\\_\\text{img}$ and our T2V-Turbo (VC2). The results show that while the visual quality of both methods is generally similar, additional feedback from $\\mathcal{R}_\\text{vid}$ significantly enhances the text-to-video alignment in T2V-Turbo.\n\n> The paper lacks in-depth discussions on the choice and influence of the image/video RMs. For instance, would it be better to CLIP/GAN-discriminator compared to the HPS?\n\nWe thank the reviewer for the insightful question! Firstly, we would like to remind the reviewer that we have experimented with additional image-text RMs, including PickScore and ImageReward, as detailed in Appendix E. We qualitatively show that incorporating reward feedback from any of these image-text RMs leads to quality improvement over the baseline VCM (VC2). It is worth noting that HPSv2.1 and PickScore are fine-tuned from CLIP with human preference data. Therefore, learning from CLIP might not lead to better performance compared to learning from image-text RMs.\n\nWe did not experiment with a GAN-discriminator, as training a GAN-discriminator can suffer from instability and may require substantial hyperparameter tuning. However, given the success of ADD, we acknowledge the potential benefit of learning from a GAN-discriminator. Consequently, we consider this a promising direction for future work.\n\nLastly, we promise to include more in-depth discussions in our revised manuscript on the choice and influence of image/video RMs.\n\n> Why do MS and VC2 perform differently with InternVidS2?\n\nFirstly, we would like to emphasize that when training T2V-Turbo (MS) with InternVid2 S2, T2V-Turbo (MS) still shows improvement over both VCM (MS) and VCM (MS) + $\\mathcal{R}\\_\\text{img}$. However, we acknowledge that ViCLIP is more effective in enhancing T2V-Turbo (MS) compared to InternVid2 S2, as demonstrated in Tables 2 and 3. In contrast, ViCLIP and InternVid2 S2 work similarly well for T2V-Turbo (VC2). \n\nWe conjecture that this difference might be due to the quality disparity between the teacher models ModelScopeT2V and VideoCrafter2. For example, ModelScopeT2V suffers from low resolution, and the generated videos contain watermarks, which may limit VCM (MS)'s ability to learn from InternVid2 S2. Nonetheless, further tuning of the $\\beta_\\text{vid}$ parameter might help improve the performance of T2V-Turbo (MS) with InternVid2 S2.\n\n>  Why use different video RMs for ModelScope and VC2?\n\nOur main purpose is to **examine our methods with a diverse set** of $\\mathcal{R}\\_\\text{vid}$. As we did not have access to a video-text RM trained to reflect human preference on video, we instead chose to experiment with video foundation models, including ViCLIP and InternVid2 S2. Note that we perform an ablation study and report results with both ViCLIP and InternVid2 S2 in Table 3.\n\n> What leads to different hyperparameter choices in L164-165?\n\nCompared to VideoCrafter2, ModelScopeT2V has not been trained on high-quality video or image datasets. Additionally, ModelScopeT2V suffers from low resolution, and the generated videos contain watermarks. Thus, ModelScope suffers from lower video quality. As a result, T2V-Turbo (MS) requires larger weighting parameters $\\beta_\\text{img}$ and $\\beta_\\text{vid}$ to get stronger external supervision from the RMs to improve its generation quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their time and constructive feedback on our work. We have responded to individual reviews below and would like to highlight additional qualitative results in the attached PDF. **Please download and open it with Adobe Acrobat** to click and play the videos.

The attached PDF **corroborates the results in Table 2** with video examples. To demonstrate the effectiveness of our mixture of RMs, we compare 8 pairs of videos generated by our $\texttt{T2V-Turbo}$ and $\text{VCM}$ + $\mathcal{R}\_\text{img}$. Due to space constraints, we focus on the VC2 variants and did not include results for $\text{VCM}$ and $\text{VCM}$ + $\mathcal{R}_\text{vid}$, as our $\texttt{T2V-Turbo}$'s results are significantly better. Additionally, we include videos corresponding to Figure 4 of our paper to better showcase the superiority of our $\texttt{T2V-Turbo}$ over its teacher model and the baseline $\text{VCM}$.

We would also like to re-iterate the contributions of our methods:
1. To the best of our knowledge, we are the first to improve video generation using feedback from video-text RMs.
2. We emphasize the importance of our mixture of RMs design, which enables us to break the quality bottlenecks in VCM without access to a $\mathcal{R}\_\text{vid}$ trained to mirror human preference. The 4-step generations from our $\texttt{T2V-Turbo}$ set the SOTA performance on VBench, surpassing proprietary models like Gen-2 and Pika.
3. Our training pipeline is NOT a simple combination of VCM and differentiable reward. Our method cleverly leverages the single-step generation arising from consistency distillation, avoiding the need to backpropagate gradients through the memory-intensive iterative sampling process required by traditional methods, such as InstructVideo.
4. Our method is notably **computationally efficient**. For example, InstructVideo requires over 40 hours of training to align a pretrained T2V model. In contrast, our method reduces training time to less than 10 hours, achieving a fourfold increase in efficiency in wall-clock training time.
5. Our method is **broadly applicable to a diverse set of prompts**. We have empirically shown that our approach outperforms existing methods on comprehensive benchmarks, including VBench and EvalCrafter. In contrast, InstructVideo's experiments are conducted on a limited set of user prompts, most of which are related to animals.

Meta-review: The paper received two borderline rejects, one weak accept and two accepts. The concerns were primarily centered around lack of technical novelty since the paper's summary can be distilled to "applying reward models to distillation". But all the reviewers point out the impressive results with the approach using very few steps. The paper applies multiple image and video reward models during consistency distillation to avoid memory bottlenecks. It also showcases that a true video reward model (which don't exist publicly) is not needed, and using video-text models as reward models also boosts performance. The AC agrees with the majority of reviews and feels that given the simplicity of implementation and the impressive results, the paper merits acceptance.
Decision: Accept (poster)