Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation
Authors: Yuanhao Zhai
Kevin Lin
Zhengyuan Yang
Linjie Li
Jianfeng Wang
Chung-Ching Lin
David Doermann
Junsong Yuan
Lijuan Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: consistency distillation
video diffusion models
diffusion distillation
text-to-video generation
Primary area: diffusion_based_models
TL;DR: Our motion consistency model not only accelerates text2video diffusion model sampling process, but also can benefit from an additional high-quality image dataset to improve the frame quality of generated videos.

Abstract: Image diffusion distillation achieves high-fidelity generation with very few sampling steps. However, directly applying these techniques to video models results in unsatisfied frame quality. This issue arises from the limited frame appearance quality in public video datasets, affecting the performance of both teacher and student video diffusion models. Our study aims to improve video diffusion distillation and meanwhile enabling the student model to improve frame appearance using the abundant high-quality image data. To this end, we propose motion consistency models (MCM), a single-stage video diffusion distillation method that disentangles motion and appearance learning. Specifically, MCM involves a video consistency model that distills motion from the video teacher model, and an image discriminator that boosts frame appearance to match high-quality image data. However, directly combining these components leads to two significant challenges: a conflict in frame learning objectives, where video distillation learns from low-quality video frames while the image discriminator targets high-quality images, and training-inference discrepancies due to the differing quality of video samples used during training and inference. To address these challenges, we introduce disentangled motion distillation and mixed trajectory distillation. The former applies the distillation objective solely to the motion representation, while the latter mitigates training-inference discrepancies by mixing distillation trajectories from both the low- and high-quality video domains. Extensive experiments show that our MCM achieves state-of-the-art video diffusion distillation performance. Additionally, our method can enhance frame quality in video diffusion models, producing frames with high aesthetic value or specific styles.

Motion Consistency Model:
Accelerating Video Diffusion with Disentangled Motion-Appearanee Distillation
Yuanhao Zhai
Zhengyuan Yang! Linjie Li Jianfeng Wang* David Doermann" Junsong Yuan Lijuan Wang! State University New York Buffalo 'Microsoft
Kevin Lin
Chung- Ching Lin
{yzhai6, chungehing lin, jijuann]mmii crosoft. (keli, zhengyang, lindsey. .1, 1 jianfw,
https: //yzhai, github. comuubiio/nnn io/mcm/
Abstract
Image diffusion distillation achieves high-fidelity generation with very few sam- pling steps. However, applying these techniques directly to video diffusion often results in unsatisfactory frame quality due the limited visual quality in public video datasets. This affects the performance of both teacher and student video diffusion models. Our study aims improve video diffusion distillation while improving frame appearance using abundant high-quality image data. We pro- pose motion consistency model (MCM), single-stage video diffusion distillation method that disentangles motion and appearance learning. Specifically, MCM in- cludes video consistency model that distills motion from the video teacher model, and an image discriminator that enhances frame appearance to match high-quality image data. This combination presents two challenges: (1) conflicting frame learn- ing objectives, video distillation learns from low-quality video frames while the image discriminator targets high-quality images; and (2) training-inference discrepancies due the differing quality of video samples used during training and inference. To address these challenges, we introduce disentangled motion distillation and mixed trajectory distillation. The former applies the distillation objective solely to the motion representation, while the latter mitigates training- inference discrepancies by mixing distillation trajectories from both the low- and high-quality video domains. Extensive experiments show that our MCM achieves the state-of-the-art video diffusion distillation performance. Additionally, our method can enhance frame quality in video diffusion models, producing frames with high aesthetic scores specific styles without corresponding video data.
1 Introduction
Diffusion models [22, 54, 42, 23, 28, 45] have significantly advanced the quality text-to-image
generation [45, 8, 13], enabling the creation of high-fidelity images and allowing for diverse stylistic
variations [3, 2, 1]. The recent development
image diffusion distillation [55, 34, 48, 38, 31] significantly reduces the inference cost by letting distilled student models to perform high-fidelity generation with extremely low sampling steps.
Due to the additional temporal dimension, the sampling time for video diffusion models [21, 24, 9, 62, 19, 65] is considerably longer than that for image diffusion models. However, directly extending image distillation methods to speed up the video diffusion models results in unsatisfactory frame
Work done during an internship at Microsoft.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Teacher steps)
step
steps
steps
LCM
shuf mrufee eereers ttsck
MCM (ours)
Figure Qualitative result comparisons with latent consistency model (LCM) [38] using Mod- elScope'T2V [62] as teacher. Our MCM outputs clearer video frames using few-step sampling, and also adapt different image styles using additional image datasets. Corresponding video results are in supplementary materials.
appearance quality. Specifically, unlike image diffusion models that benefit from high-quality image datasets such LAION [49], public video datasets [70, 57, often suffer from lower frame quality, including issues like low resolution, significant motion blur, and watermarks. This discrepancy poses great challenge, quality and the distillation process. An example is shown in Fig. 1, where both the teacher and the latent consistency model (LCM) student [38, 66] learn the watermark from the video dataset.
the quality of the training data directly impacts both the teacher generation
To address this challenge, we propose motion consistency model (MCM), a single-stage video diffusion distillation method that enables few-step sampling and can optionally leverage high- quality image dataset to improve the video frame quality. We showcase our results in Fig. bottom row and Fig. 6, and illustrate the distillation process in Fig. 2.
To simultaneously achieve the goal of diffusion distillation and frame quality improvement, we build our method upon a video LCM [38, 66] combined with an image adversarial objective 48), due their proven effectiveness in the two tasks [38, 66, 61, 76, 27], respectively. However, directly combining them presents two essential problems. (1) Conflicted frame learning objectives. The LCM aims to learn and represent characteristics low-quality video frames, while the adversarial objective pushes the frames toward high-quality image data. This opposition creates conflicts between preserving low-quality visual features and enhancing image quality. (2) Discrepant training training-inferenc distillation input. During training, only low iow-quality video samples used for LCM distillation, whereas during inference, the input is generated high-quality video samples. These two problems greatly hinders the distillation process and the generated frame quality.
To address the first problem, we propose disentangled motion distillation. Specifically, we disentangle the motion components from the video latent, and only distill the motion knowledge from the teacher. Meanwhile, the image adversarial objective used exclusively to learn frame appearance. This approach effectively mitigates the conflict between learning objectives by ensuring that motion and ap- pearance are learned independently. For the second problem, we propose mixed trajectory distillation. We simulate the probability flow ordinary differential equation (PF-ODE) trajectory during inference

---

High-quality image data Static image
T2V Teacher
Low-quulity video data
T2V Student
Generated video
Low frome quality
Dynamic video Motion consistency model
High frome quality
Dynomic video
sampling
Fast sampling
Figure 2: Illustration of our motion consistency model distillation process, which not only distills the motion prior from teacher to accelerate sampling, but also benefit from additional high-quality image dataset to improve the frame quality of generated videos.
and sample generated high-quality videos from this simulated trajectory for distillation. By mixing distillation between low-quality video samples and generated high-quality video samples, we ensure better alignment between training and inference phases, thereby enhancing overall performance.
In summary, our contributions are as follows:
We propose motion consistency model (MCM), a single-stage video diffusion distillation method that accelerates the sampling process and can optionally leverage an image dataset to enhance the quality of generated video frames.
We introduce disentangled motion distillation to resolve the conflicting frame learning objec- tives between video diffusion distillation and image adversarial learning, ensuring better motion consistency and frame appearance.
We propose mixed trajectory distillation, which simulates the PF-ODE trajectory during infer- ence. By combining low-quality video samples with generated high-quality video samples during distillation, our approach improves training-inference alignment and enhances generation quality, We conduct extensive experiments, demonstrating that our MCM significantly improves video diffusion distillation performance. Furthermore, when leveraging an additional image dataset, our MCM better aligns the appearance of the generated video with the high-quality image dataset.
2 Preliminaries
Diffusion models. Diffusion models [22] consists of forward and reverse process. The forward
process corrupts the data progressively by interpolating a Gaussian noise W I) and the data sample zo. T; = g( e, () a;æ0 + o,e, where a and a, are pre-defined noise schedule. The reverse process learns time-conditioned model to gradually removes the noise. Recent methods [56, 37] propose to design numerical solvers of ordinary differential equations (ODE) to reduce the number of sampling steps.
Consistency model. Consistency model (CM) [55] is a new family of generative models that enables few-step sampling. The essence of CM f(x,, f) is the self-consistency property, that maps any point in the same probability flow ODE (PF-ODE) trajectory for Vt, t' E [c. 7), where is fixed small positive number. The boundary condition f(x <) = ensures the CM always predicts the origin PF-ODE trajectories. CM is parameterized as the weighted sum of the data sample and model output F parameterized by 0:
same point. That is, f(x,, t) f t')
f [wr t) = Cskip(t) + Cour(t) Fg (Zx,,)) where Caki (() and Coar (l) are differentiable functions [55, 28] with (asp(t) and Grar (<) 0.
(I)
CM can be learned via consistency distillation (CD), where teacher model ø and ODE solver &b estimates the previous sample in the empirical PF-ODE trajectory, with being the step size:
( f; t).
(2)
Let 0 be the student model parameter, and its exponential moving average (EMA) 0- + - ( 2)0 be the target model parameter. CD is trained enforce the self-consistency property, which minimizes the distance between student and target outputs:
LCD Exo.e.t (Jo 1), So- (0f -1)]]
(3)
3

---

In this paper, we follow previous methods [38, 61] to use Huber loss as the distance function: d(a,, æ/) = v a 2/1 + 82 6, where 8 is a threshold hyperparameter.
3 Motion consistency model
We present motion consistency model (MCM), novel video diffusion distillation method designed for accelerating the text-to-video diffusion sampling process. Additionally, MCM can leverage different image dataset to adjust the frame appearance, e.g., frame quality improvement, watermark removal or frame style transfer. An illustration of the objective is shown in Fig. 2.
video-caption dataset V = {(v, c) and an image dataset MI = {m our objective is twofold.
Problem formulation. Given pre-trained text-to-video diffusion model
1. Distill the motion knowledge from teacher e into student 0 using the video dataset V, enabling ftw-step video generation.
2. Adjust the appearance of the generated video frames using the image dataset M.
In this paper, we assume that the appearance of video frames in V has lower quality compared the images in M. For convenience, we refer to the appearance of video frames as following a "low-quality" distribution and the images in MI following thigh-quality distribution.
When the image dataset M consists of frames extracted from the video dataset V, this task simplifies into a straightforward video diffusion distillation process.
3.1 Video latent consistency model meets image discriminator
straightforward way to achieve the aforementioned goal to adopt two-stage training strategy [61]] Le., train a 2D backbone with the image dataset, and then temporally inflate the backbone to fine-tune temporal layers on videos. However, not only this approach can time-consuming, but the disjointed training stages can lead to suboptimal motion modeling and degraded appearance learning. To address this problem, we propose single-stage, end-to-end learnable approach.
To build a strong baseline, we adopt video latent consistency model (LCM) [38] f- with frame adversarial objective [47,45 48]. The LCM has proven effective in diffusion distillation [38, 66, 61], and the adversarial objective has been widely used for style transfer [76, 27] and diffusion distillation [48]. By combining these two approaches, we aim achieve both few-step sampling and improved frame appearance control.
For the discriminator D(-) used in the adversarial learning, adopt pixel-space discriminator following previous methods [47, 48]. However, densely applying the adversarial objective to all frames is computationally expensive, and can lead degraded temporal smoothness [72]. To mitigate this, drawing inspiration from sparse sampling-basse action recognition [52, 64, 63], we apply the adversarial loss only to set of sparsely sampled frames.
Specifically, let ag = fo (xx, t) represent the LCM- --M--reditted video latent, and by = ((a)) be the decoded video, where 9 is the latent decoder. We randomly sample frames from the video, denoted 0.s (00,0) ..., ob,2) and apply the adversarial loss these sampled frames:
1 i
Σ Dos De
EExn,e, e,e
(4)
where vb is the learnable parameter in D(-). The discriminator is trained to minimize
CD Exo.e.t max (0. 0,1 + 1 ΣΣ D. ]
+ Em, [max(0, 1 - Dy (m,)))
(5)
We omit the gradient penalty term [18, 48] for conciseness. Please find details on the discriminator design in [47, 48] or in Appendix C. Note that the discriminator's 's objective is to distinguish between the generated video frame and the images from the image dataset M, thereby aligning the video frame with the high-quality distribution.
The overall baseline learning objective is a weighted sum of the CD loss (Eq. (3) and the adversarial loss: Lrase LCD Andy C where Antv is the weighting hyperparameter.

---

Target
Add noise
M(-}
Low-quality distribution
Real video
Teacher reverse
ODE trajectory
Moving average
LMCD Extract motion
fo ()
fol.)
Generated video ODE trajectory
Exee
fol)
M(-
High-quality distribution
Student
Gauu
udy
fo-()
2ig
500
L
adv
fol.
Data
Noise
Figure 3: Left: framework overview. Our MCM features disentangled nottio-appearance distillation, where motion learned via the motion consistency distillation loss CMCD. and the appearance is learned with the frame adversarial objective Lady Right: mixed trajectory distillation. We simulate the inference-time ODE trajectory using mudent--enerann video (bottom green line), which mixed with the real video ODE trajectory (top green line) for consistency distillation training.
3.2 Disentangled motion consistency distillation
While combining LCM and the adversarial loss is straightforward, their training objectives conflict. The LCM aligns the appearance the predicted video with the low-quality distribution, while the adversarial objective adjust the appearance to match the high-quality distribution. Moreover, as
Tanget
EMA
EMA
CMCD
LCM relies solely on boundary conditions (Sec. 2) generate clean video frames, it typically leads to blurry frame as approaches T [55, 38, 61]. This blurriness conflicts with the adversarial loss objective, which seeks to produce clean frames at all steps.
Student
MLP
To mitigate these conflicts, we propose a disentangled approach: separating the motion presentation from the video latent, and applying the LCM objec- tive exclusively to the motion representation to ensure temporal smoothness. Meanwhile, we use the adversarial objective to learn the frame appearance. This disentanglement allows us to leverage the strengths of both objectives without them interfering with each other. An illustration shown Fig. left.
Figure 4: Learnable motion representation.
Specifically, we replace the original consistency distillation loss LCD in Loase with the following motion consistency distillation (MCD) objective LMCD:
(M (M (fo- -))]]
LMCD Exp.e.t
where M(-) indicates motion representation extraction function. In this paper, consider the following implementations.
Latent difference: the difference between temporally consecutive frame latents. This was used in action recognition [52, 64, 75] as motion representations.
Latent correlation: 4D cost volume tensor containing pairwise dot-product between consecutive
frame latents, widely used in optical flow estimation [26, 51, 50].
Latent low/high-frequency components: this is inspired by recent work [69], which shows different frequency components the latent have different impacts on the motion.
Learnable representation: we apply two-layer MLP after the LCM output to extract motion (Fig. 4). This MLP does not contain temporal layers, and preserves the motion while avoiding direct applying appearance loss to the latent. The parameters of the target MLP head updated using exponential moving average, and the MLP discarded during inference. This design resembles the projection head used in self-supervised learning [17, 11, 12, 10].
examine all choices in Sec. 4.3, and find that the learnable representation works the best, enabling effective motion consistency modeling while preserving frame appearance quality.
5

---

Table I: Video diffusion distillation comparison on the Web mini validation set.
StDe +
CLIPSIM Step
Teacher
Method
8
2
8
DDIM [54]
5228
2580
1222
858
20.05
20.61
24.04
28.89
DPM++ [37]
2082
1142
843
975
22.13
24.78
29.38
30.75
AnimateDiff [19] 512 x 512 x 16
ICM [38] (our impl.)
1242
978
1006
909
27.32
28.95
29.94
29.86
AnimateLCM [61]
1575
1333
998
946
24.65
27.43
28.98
28.77
mnimatDDif-LLghhnnnn [35]
1288
1289
1283
1355
28.29
29.07
30.01
29.69
MCM (ours)
1025
827
821
30.36
30.70
30.85
29.88
DDIM
7231
23009
T229
652
20.47
20.06
23.68
28.46
ModeIScopeT2V [62] 256 256 16
DPM++ [37]
2030
1142
477
506
22.38
24.79
29.70
31.00
LCM [38] (our impl.)
854
658
603
637
27.50
29.28
30.48
30.73
MCM (ours)
526
450
456
503
29.81
30,63
30.55
29.58
3.3 Mixed trajectory distillation
While the disentangled motion distillation mitigates the appearance learning objective conflicts, another training-inference discrepancy persists. During training, all ODE trajectories used for MCD are sampled by adding noise to the low-quality videos in 5 (Fig. 3 top right). In contrast, during inference, the ODE trajectories will be sampled in the high-quality video space. As result, such discrepancy leads to degraded performance during inference. Furthermore, since we do not have access to high-quality videos, cannot directly sample ODE trajectories from this distribution for training, making it challenging to resolve this training-inferenne discrepancy.
To address this problem, we propose to simulate such high-quality video ODE trajectories by using the consistency model multi-step inference process [55]. Specifically, starting from random noise 2т (0 I), first run single-step inference to get the high-quality data sample a and then add noise to to sample high-quality ODE trajectories. An illustration is shown Fig. bottom right.
Formally, the starting point ž for MCD is sampled
a ag + O,E, where z fo(er, and ær (O,
In this way, the MCD loss can be formulated by replacing the starting data points from those sampled from low-quality trajectories with those sampled from simulated high-quality video trajectories:
MCD = Exr,ett (dl (M (fo(E) .M (Jo- (4 -s))
where 0..0 2 ( Ø). Similarly, the adversarial loss is also applied to the model d;-s
prediction from the samples on the high-quality trajectories, which we denote as CGGgen By applying the MCD and adversarial objectives to the simulated high-quality trajectory, it not only mitigates the waining-inferene trajectory discrepancy, but also ensures zero terminal signal-to-noise ratio æT <N I) to avoid the data leakage problem [33] and helps plain video distillation.
In practice, found that only training from the generated video ODE trajectory potentially leads to mode collapse (Appendix B.1,, and mixing the training from real samples and generated samples achieves the best results. Thus, the final loss L is formulated as a weighted sum of the distillation objectives using real samples and generated samples:
Areal (EMCD + C + (1 - >reat) (ARRD) + Audy G,gen adv
where Areal E [0, 1] is hyperparameter to balance the distillation in different trajectories.
4 Experiments
Implementation details. We choose two text-to-video diffusion models for experiments: Mod- e2S [62] and Animate [19] with StableDiffusion v1.5 [45]. We use DDIM solver [54] with 50 sampling steps as the ODE solver Φ. We train LoRA [25] with rank 64 as MCM, follow- ing [38, 61]. Following the teacher model setting, we generate 16-frame videos, with resolution 256x256 for ModelScopeTT2V, and 512x512 for AnimateDiff. The learning rates for the diffusion model and discriminator are set to 56 and 5e respectively, with batch size 128, Adam opti- mizer [30], and 30k training steps. The weight hyperparameters are determined via grid search:

---

Table 2: Zero-shot video diffusion distillation comparison on the MSRVTT validation set.
FVD@ Step
CLIPSIM Step
Teacher
Method
4
DDIM [54]
4782
4350
2774
20.90
20.94
22.87
27.36
DPM++ [37]
2004
1447
876
794
22.93
24.5
27.62
29.10
AnimateDiff [19] 512 x 512 x 16
LCM [38] (our impl.)
1276
1180
956
830
25.75
27.33
28.37
28.65
Animatel CCM [6]]
1578
1278
824
740
27.56
28.52
29.58
27.67
AnimateDiff-Lighmaeee [35]
1260
1259
892
932
27.38
28.77
29.12
28.77
MCM (ours)
1197
1036
801
675
28.95
29.40
29.64
29.13
DDIM [54]
6439
2305
1445
841
21.49
20.33
22.57
26.76
ModelScope122 [62] 256 256 16
DPM++ [37]
2039
1336
467
23.48
24.85
28.51
29.70
LCM [38] impl.)
1094
26.78
28 .55
29.01
MCM (ours)
501
434
414
482
28.37
29.02
28.86
28.28
Teacher 50 steps
Ours
DPM++
AnimateLCCM
AnimateDiff- Lighttiniin
2 steps
4 steps
steps
steps
2 steps
4 steps
2 steps
4 steps
young woman a yellow sweater uses VR glasses, sitting on the shore of pord on background dark waves. A strong wind develops her hair, the sun's rays reffected from the water.
Female running at sunset. Healthy fitness concept.
Figure 5: Qualitative comparison of video diffusion distillation with AnimateDiff [19] the teacher model. The first and last frames are sampled for visualization. Our MCM produces cleaner frames using only 2 and 4 sampling steps, with better prompt alignment and improved frame details. Corresponding video results are supplementary materials.
Andy and Aveal 0.5. The experiments are conducted on machine equipped with H100 GPUs. The project is developed using PyTorch [4], Diffusers [60], and PEFT [40].
Evaluation metrics. We use FVD [59] to measure the video quality, and CLIP [44] similarity score (CLIPSIM) for video-prompt alignment measurement, where FID [20] to measure the frame quality and the similarity to the image dataset.
is used. In Sec. 4.2, we
4.1 Comparisons with distilled video diffusion models
Experiment settings. In this section, we compare MCM with other video diffusion models validate the effectiveness of the disentangled motion aottion-ppperance distillation. For fair comparison, we use the WebVid 2M [5] as both the vidco and image training dataset, without using any additional image datasets. We postpone the experiments with different image datasets in Sec. 4.2. For testing, we ran- domly sample 500 validation videos from WebVid 2M (WebVid mini) for in-distribution evaluation; we also follow common practice [62, 16] to use 2900 validation videos from MSRVTT [70] for zero-shot generation evaluation. We report standard metrics of FVD and CLIPSIM.
Results. We compare our method with training-free samplers DDIM [54] and DPM++ [37], and trainable models LCM [38], Animatel LCM [61] and AnimateDiff-Lightnn [35]. We present the quantitative results on Web and MSRVTT Tab. and Tab. respectively. On both datasets, and using both teacher models AnimateDiff [19] and Mo [62], our MCM outperforms

---

Teacher
Ours WebVid
LAION-aes steps
Anime 4 steps
Realistic 4 steps
cartoon steps
50 steps
4 steps
Aerial view. mid-air flight over and clean mountain river sunny Summer morning. Green trees and rays on horizon. Direct
Back of woman in shorts going pure creek in beautiful mountains.
Ri
Beautiful colours of autumn enhanced by the golden sunset hour of light washing the trees and fields near a small rural scottisk village with the occasional car driving along a quiet road.
Figure 6: MCM frame quality improvement results using different image datasets with Mod- e2ScopeT2V [62] teacher. The first and last frames are sampled for visualization. Our MCM effectively adapts different distributions with 4 steps. Corresponding video results are in supple- mentary materials.
all competing distillation methods in terms of video quality and video-prompt alignment at sampling steps 1, 2, and 4. We compare the qualitative results in Fig. 5, where method outputs clean video frames using and 4 sampling steps. Compared with AnimateDiff-Lighnin [35], we achieve better prompt-alignment and frame details, demonstrating our effectiveness.
4.2 Video distillation with improved frame quality
Experiment settings. this section, we unleash the full capability of MCM by equipping the model with diverse types of image datasets. We showcase the effectiveness of using image datasets to boost frame quality as well as adapting the model new styles without requiring corresponding video data. Specifically, continue to use WebVid 2M [5] as the video dataset. For image dataset,

---

Table 3: Comparison on frame quality improvement. Performance mea- sured by FID using different numbers of sampling steps.
Table 4: Ablation study on each components in MCM. Mod- elScopeT2V [62] is used as teacher.
PVDesSiep
CLIPSIM Step
Adv.
MCD
Mixed
4
8
4
LCM
854
658
603
637
27.50
29.28
30.48
30.73
CAmine)
703
650
767
760
28.70
30.37
30.90
30.77
Tip MCM (64m))
119
100
185
30.11
89
76
78
71
67
526
450
456
503
29.81
30.63
30.55
29.58
Table 5: Ablation study on motion representation.
Table 6: Ablation study on the mixed trajectory training.
Step L
CLIPSIM@ Step
Motion
2
Raw latent
703
650
767
760
28.70
30.37
30.901
30.77
FVDDoSSepp
CLIPSTMM
Dift.
628
640
722
779
29.17
30.39
31,03
30.55
Awi
1
28.83
30.39
30.62
30.62
563
526
548
579
29.30
30.40
30.11
29.52
Low freq.
677
667
753
919
29.25
30.32
30.35
29.79
29.81
30.55
High frey.
28.76
30.01
556
498
483
523
28.77
29.73
29.93
29.53
Leamable
563
526
548
579
29.30
30.40
30.11
29.52
consider the following options. LAION-aes, 4M subset sampled from the high- aesthhett image dataset LAION- -esthetic 6+ [49]; datasets in anime, realistic, and 3D cartoon styles generated by fine-tuned StableDiffusion [3, 2, I], each dataset containing 500k images with LAION captions.
Results. Due to the lack of prior arts with comparable capabilities, we validate by comparing MCM with designed two-stage baseline. Following Animatel.CMM LCM [61], we implement a two-stage method: first training the spatial layers with the image dataset, then training the temporal layers with the video dataset. We use real image dataset, LAION-aes, and generated anime-style dataset for quantitative comparison. As shown in Tab. 3, our MCM achieves lower FID scores on both datasets, demonstrating its superior adaptability to different image dataset distributions. We present our qualitative results in Fig. 6, where our method demonstrates high frame quality using only four sampling steps, achieving sharp frames with various styles.
4.3 Ablation study
We conduct the distillation ablation study on WebVid mini [5] using ModelScope [62] as teacher. The main results are summarized in Tab. 4. Starting from LCM [38], we progressively add adversarial learning, motion consistency distillation loss, and mixed trajectory distillation. Each addition shows an improvement in video quality, demonstrating the effectiveness of these components.
Disentangled motion distribution. We compare different motion representations in Tab. 5. All representations, except for the latent low-frequency components, help reduce FVD and improve CLIPSIM at low sampling steps. This indicates that disentangling motion from the raw latent space for consistency learning enhances video quality. Among these, the learnable motion representation performs the best, demonstrating its effectiveness compared to handcrafted motion representations.
Mixed trajectory distribution. Tab. 6 compares the performance between using only real video (Rret = 1), only generated video (Aveal = 0), or a mixed of both. The results reveal that mixing the trajectories achieves the best results. While using only generated videos also reduces FVD at all steps, it potentially leads to mode collapse, producing nearly all-black videos (Appendix B. highlighting the importance of incorporating real videos.
4.4 Applications
We demonstrate that our proposed MCM is versatile and applicable to various tasks, including high- resolution video generation with different aspect ratios and conditioned video generation. In Fig. MCM generates high-resolution videos up to 576x 1024 pixels and supports various aspect ratios such as square, portrait, and landscape formats. Moreover, by incorporating controllable modules, MCM enables conditional video generation. For example, in Fig. 8, integrating ControlNet [73] allows for pose-conditioned video generation. Notably, both videos are generated using only sampling steps, highlighting the effectiveness and versatility of our MCM.

---

Figure 7: Our MCM allows for high-resolution video generations with different aspect ratios, using 4 sampling steps. The left three videos (576x 1024) are in portrait format, the top right videos (768x512) are in landscape format, and the bottom right two videos (768x768) are in square format.
Figure 8: Our MCM can incorporate ControlNet [73] to enable pose-conditioned video generation. The videos are generated using 4 sampling steps.
5 Conclusion
We introduce motion consistency model (MCM) to tackle the challenge of low frame quality in video diffusion distillation. Our method leverages disentangled motion distillation to separate motion learning from appearance learning, and mixed trajectory distillation to align training and inference through simulating the inference ODE trajectory. MCM achieves state-of-the-an video diffusion distillation results, and effectively improves frame quality when using high-quality image dataset. Broader impacts and limitations. MCM accelerates and enhances video generation for creative applications. However, as a data-driven approach, our model is sensitive to the distribution and diversity of the training data, which influences model's ability to generate varied video frames, potentially limiting generalization ability. MCM also carries potential risks such deepfakes creation. Implementing responsible deployment is essential to mitigate these risks.
Acknowledgement
This work supported in part by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011200 1200012.. Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA).
10

---

References
[1] Disney pixar cartoon type b v1.0.
https://diviiaaii.ccmmmoeell///
2023,
[2] Realistic vision v6.0 bl.
https://ddvitaal.co/mmodlls////
realistio-vision-v600-- 2023.
[3]
Toonyou - beta 6. https://civitaat htt.a//c.c..o/eeeaaa//////////// 2023.
[4] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Vozne- sensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceed- ings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 929-947, 2024.
[5] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Int. Conf. Comput. Vis., pages 1728-1738, 2021.
[6] Yogesh Balaji. Martin Rengiang Min, Bing Bai, Rama Chellappa. and Hans Peter Graf. Condi- tional gan with discriminative filter generation for text-to-video synthesis. In ICCAI, volume 1, page 2, 2019.
[7] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.06500. 2022.
[8] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3 pdf. 2(3):8, 2023.
[9] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., pages 22563-22575, 2023.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self--upervised vision transformers. In Int. Conf. Comput. Vis., pages 9650-9660, 2021.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. pages 1597-1607. PMLR, 2020.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020.
[13] Xiaoliang Dai, Ji Hou, Chih-Yar Ma, Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023.
[14] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
[15] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Adv. Neural Inform. Process. Syst., 36, 2024.
[16] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. Int. Conf. Compur. Vis., pages 22930-22941, 2023.
[17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Adv. Neural Inform. Process. Syst. 33:2 22112222244 2020.
[18] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Adv. Neural Inform. Process. Syst., 30, 2017.
[19] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint axXiv:2307.04222 2023.

---

[20]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Adv. Neural Inform. Process. Syst., 30, 2017.
[21] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint wxXiv:2210002300 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adv.
Neural Inform. Process. Syst., 33:6840-6851. 2020.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 025.0.../299.. 2022.
[24] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Adv. Neural Inform. Process. Syst., 35:8633-8646, 2022.
[25] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv
preprint 200.::2000.006855, 2021.
[26] Zhaoyang Huang. Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In Eur. Conf. Compur. Vis., pages 668-685. Springer, 2022.
[27] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conf. Comput. Vis. Puttern Recog,, pages 1 112--1134, 2017.
[28] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. Adv. Neural Inform. Process. Syst. 35:26565-26577, 2022.
[29] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Int. Conf. Comput. Vis., pages 15954-15964, 2023.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.690 69800, 2014.
[31] Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, and Ali Thabet. Imagine flash: Accelerating emu diffusion models with backward distillation. arXiv preprint 24fi::2000052224 2024.
[32] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Adv. Neural Inform. Process. Syst., 36, 2024.
[33] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps flawed. In JEEE Winter Conf. Appl. Comput. Vis., pages 5404-5411, 2024.
[34] Shanchuan Lin, Anran Wang, and Xiao Yang, Sdxl-lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.11322 2024.
[35] Shanchuan Lin and Xiao Yang. Animatedif--lghhmingg Cross-model diffusion distillation. arXiv preprint arXiv:24033 /2706, 2024.
[36] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.099 09778, 2022.
[37] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Adv. Neural Inform. Process. Syst. 35:5775-5787, 2022.
[38] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[39] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. arXiv preprint arXiv:2312.00852. 2023.
[40] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: tthee- parameter-efficien fine-tuning methods. https: //gi titthb..coo com con//nnggingfaace//ftttt 2022.
12

---

[41] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. IEEE Conf. Comput. Vis. Puttern Recog-, pages 14297-14306, 2023.
[42]
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. pages 8162-8171. PMLR, 2021.
[43] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafranice, Vasil Khalidov, Pierre Feraandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 4748-8763, PMLR, 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 10684-10699 10695, 2022.
[46] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXivv2202.0005: 2022.
[47] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. pages 30105-30118 PMLR, 2023.
[48] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17044 2023.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Ady. Neural Inform. Process. Syst. 35:25278-2553 25294,, 2022.
[50] Xiaoyu Shi, Zhaoyang Huang. Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In Int. Conf. Comput. Vis., pages 12469-12480, 2023.
[51] Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++: Masked cost volume autoencoding for pretraining optical flow estimation. In JEEE Conf. Comput. Vis. Pattern Recog,, pages 1599- 1610, 2023.
[52] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Adv. Neural Inform. Process. Syst., 27, 2014.
[53] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3626-3636, 2022.
[54] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
[55] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint FrXiiv:3003.01469 2023.
[56] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[57] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.04000 2012.
[58] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In IEEE Conf. Comput. Vis. Puttern Recog., pages 1526-1535, 2018.
13

---

[59] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprini arXiv: /8/2.0177, 2018.
[60] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https: htte:///gihie ggubb www.coo/////o// con/huggingface) diffusers, 2022.
[61] Fu- Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning. arXiv preprint arXiv:2402. 2402.00699 2024.
[62] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang. and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06577 2023.
[63] Limin Wang. Yuanjun Xiong. Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 4325 4334, 2017.
[64] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In Eur. Conf. Comput. Vis., pages 20-36. Springer, 2016.
[65] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Adv. Neural Inform. Process. Syst., 36, 2024.
[66] Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint 12.v:2211..0909, 2023.
[67] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3an: Disentangling appearance and motion for video generation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5264-5273, 2020.
[68] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. diffusion models for text-to-video generation. In Int. Conf. Compur. Vis., pages 7623-7633, 2023.
One-shot tuning of image
[69] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint ar arXiv:2312.075377 2023.
[70] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5288-52999 2016.
[71] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou, Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:231 I. 1.09257, 2023.
[72] Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. arXiv preprint
2023.
[73] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Int. Conf. Comput. Vis., pages 3836-3847, 2023.
[74] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
[75] Yue Zhao, Yuanjun Xiong, and Dahua Lin. Recognize actions by disentangling components of dynamics. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6566-6575 2018.
[76] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Int. Conf. Comput. Vis., pages 2017.
14

---

A Related work
Video generation. Early methods mainly rely on generative adversarial networks (GANs) [58, 6, 67, 53], which struggled with scalability and generalization to out- dfsttii-iiitttiion domains. Building the recent success of image diffusion models [22, 45], several methods leverage diffusion models for video generation [21, 24, 9, 62, 19, 65]. This paper focuses two popular text-to-vided diffusion models: ModelScopeT22 [62] and AnimateDiff [19]. ModelScope112V [62] inserts spatio-temporal blocks into 2D backbones and trains the whole backbone end-to-end, ensuring adaptability varying frame lengths with smooth motions. AnimateDiff [19] introduces a plug-and-play motion module that learns motion priors from real-world videos and can be applied to personalized image diffusion
models for video generation.
Diffusion distillation. Training- free methods primarily focus developing advanced ODE solvers to reduce the number of sampling steps [54, 37, 7, 36, 74]. Another approach employs smaller backbones [39, 15, 32] to improve structural efficiency. To further reduce the inference steps, progressive distillation [46, 41, 34, 35] is first proposed to distill the two or more steps into one. Consistency models [55, 38, 66, 61] generalize progressive distillation by mapping any point the PF-ODE trajectory to the same origin. A parallel line of work [71, 48, 34, 35, 31] directly applies constraints on the model output by leveraging adversarial losses. However, these methods heavily rely on the quality of the training dataset, and low-quality datasets inevitably degrade the generation
quality.
Frame quality adjustment. Unlike image generation, which benefits from high-quality datasets [49], popular public video datasets [57, 70, 5] often exhibit lower frame quality, including issues like motion blur, low resolution, and watermarks. This discrepancy hinders the development of high-quality video generation in single-stage, end-to-end manner. A straightforward way to improve frame quality is two-stage approach [9, 19, 61], which involves training an image backbone with a high-quality image dataset and then adding temporal layers for fine-tuning on video datasets Alternatively, some methods [29, 68] animate pre-trained image diffusion models with low-shot fine-tuning. In this paper, we propose single-stage, end-to-end method address this problem, while simultaneously achieving few-step sampling with video diffusion distillation.
B
Additional experiments
B.1
Mixed trajectory distillation
We empirically find that only using generated videos for trajectory sampling (i.e., Areal = 0) may sometimes lead to near all-black videos, accounting for 5% generated videos. Some examples are shown in Fig. 9. Such a result demonstrates the importance of leveraging real video data for the distillation.
C Additional implementation details
C.1 Image discriminator
We design our image discriminator based on StyleGAN-T [47] and ADD [48]. Specifically, given an input image, the discriminator first extracts features using a pre-trained, frozen DINOv2 [43, 14] model. Features extracted from layers 2, 5, 8, and 11. These features then passed through set of trainable, lightweight discriminator heads to distinguish images at the patch level.
Following ADD [48], we use the VIT-S/14 variant for feature extraction. To improve video-caption alignment, we also incorporate CLIP embeddings for additional discriminator conditioning, following StyleGAN-T [47] and ADD [48]. For this, embeddings.
CLIP-ViT-g 14 text encoder to compute text
C.2 Training details
Both teachers use classifier-free guidance [23], and we follow L.C [38] to set dynamic guidance scale range for distillation. Specifically, set the guidance range values to [5, 15] for Mod-
15

---

1 step
aelbenstrsc
2 steps
steps
Viet wer
steps
naati reebre
>yeal =
Ayent=
Areal Flag of the country verde flutters on green Background. 3d rendering
Areal
Areal 4k video footage showing moving air conditioning elements
Areal
Stage lighting, lighting devices, stage lighting in hall, beams of light, bright beam of light
Figure 9: Failure case in mixed trajectory distillation when only generated videos are used for ODE trajectories sampling, i.e., Areal = O.
elScopeTTV [62], and [7, for AnimateDiff [19]. We follow respective teachers e-prediction,
and use linear beta scheduling for AnimateDiff following AnimateDiff-Lighhnin [35].
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction were carefully written to include literature review, our motivation, our technical details, and our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We thoroughly discussed the limitations of work in Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: YYs]
Justification: We included sufficient implementation details our paper, and our model, code, and data will be made public available upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification:
We will release our model, code, and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Co/eeeodaeeeeeeeeee
//nips.. wwtppcccpcccccccccccccccccccclct Cotaai/uaddeeeeeeeumiiiiii ssssssoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We included all implementation details in Sec. 4 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did not include error bars the experiments computationally expensive, and existing literatures do not have the convention to report error bras.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We included the details Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttp///eeiippppeeeeee wwwpurip..ccppuullleeeeeeeeeeeee httppuublc/tttite / ines?
Answer: [Yes]
Justification:
We have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discussed broader impacts in Sec. 5,
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our method can leverage the safeguards in existing diffusion models to prevent the generation of inappropriate content.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cited all code, data, and models used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not introduce assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "LBVdxQ8NlW", "reviewer": "Reviewer_VLKW", "summary": "This paper proposes a video diffusion distillation method that disentangles motion and appearance learning. Basically, it proposes to enhance the appearance generation with high-quality image data and distill motion knowledge from the video teacher model.", "questions": "Please refer to the weaknesses.", "limitations": "Broader impacts and limitations have been discussed in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method can distill motion knowledge from video diffusion models and improve the appearance quality through disentangled motion distillation.\n2. The mixed trajectory distillation is proposed to improve training-inference alignment and enhance generation quality.\n3. This paper is technically clear and the organization is good.", "weaknesses": "1. The introduction of gaps between the training and inference distillation inputs is not so straightforward.\n2. The introduction to related work needs to be significantly enhanced, especially in terms of the idea of decoupling appearance and motion, which is no longer uncommon and has many related works.\n3. It simply provides the conclusion that \"learnable representation works the best\" without giving specific analysis as to why. Such an analysis may be more helpful for following research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for identifying our key strengths and providing insightful comments. We address your concerns point by point.\n\n**Weakness 1: Gaps in training and inference distillation inputs.** Thank you for raising the concern! There exist two major gaps in the mixed trajectory distillation.\n- Distribution mismatch gap. During training, we assume all inputs are noisy low-quality videos; during inference, the input will be noisy high-quality videos.\n- Information leakage gap. During training, the original consistency distillation takes as input noisy videos that contain ground-truth video information; during inference, the sampling will start from pure noise, containing zero signal.\n\nOur mixed trajectory distillation simultaneously addresses these two problems. We will make this clear in the revised version.\n\n**Weakness 2: Related work in decoupled appearance and motion.** Thank you for the suggestions! While disentangle appearance and motion is well-established in video understanding, our work addresses unique challenges in video diffusion distillation.\n- Video diffusion models suffer from long sampling time due to the additional temporal dimension.\n- Popular open-source video datasets contain low-quality frames, such as watermarks, motion blur, and low resolution.\n- Our MCM simultaneously achieves video diffusion acceleration and frame quality improvement by leveraging additional high-quality image data.\n- We introduce disentangled motion distillation and mixed trajectory distillation to overcome specific challenges in video diffusion models. \n\nWe will expand our related work section to include additional references and contextualize our contributions in video diffusion distillation.\n\n\n**Weakness 3: Analysis on learnable motion representation.** Thank you for the advice! Our Table 5 shows that most representations, except latent low-frequency components, reduce FVD and improve CLIPSIM at low sampling steps. This indicates that disentangling motion from the raw latent space for consistency learning enhances video quality. The learnable motion representation performs best, outperforming handcrafted ones. We attribute this to its ability to:\n- Adaptively learn optimal motion features, capturing complex motion patterns that generic handcrafted representations might miss.\n- Better separate motion from appearance, reducing conflicts in learning high-quality frames.\n\nThis learned disentanglement allows for more effective motion consistency modeling while preserving frame appearance quality.\nWe will elaborate more on this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Q5MKc7bIo", "reviewer": "Reviewer_FQyx", "summary": "This paper proposed a single-stage video diffusion distillation method that can disentangle motion and appearance learning, thus improving frame appearance using various high-quality image data. The proposed mixed trajectory distillation mitigates the training-inference differences in terms of video quality. The extensive experiments demonstrate the superior performance in enhancing frame quality in the video diffusion model.", "questions": "1. In Fig. 6, why do the \"Ours w/ Webvid\" results also have watermarks?", "limitations": "The authors have discussed the limitations of this paper", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed disentangled motion distillation and mixed trajectory distillation are intuitive and novel.  \n2. The experiments are thorough. They are conducted across various datasets and show superior results in terms of video diffusion distillation. The ablation study shows the effectiveness of the proposed disentangled motion distribution and mixed trajectory distribution modules.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. Motion jittering in the supplementary video. It's probably caused by the teacher model but the authors could better discuss the way to alleviate it.\n2. In Fig. 6, there is no caption to indicate which one is the result of the proposed methods and which one is the designed two-stage baseline. What are the differences between the first row and the second row?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our MCM’s contributions and providing constructive feedback. We respond to your concerns below.\n\n**Weakness 1: Motion jittering.** Thank you for the feedback! The motion jittering is largely inherited from the teacher model (the same phenomenon was also reported in AnimateDiff-Lightning [35]). To mitigate this, we could use the following methods.\n- Increase the hyperparameter $\\lambda_{\\text{real}}$ in mixed trajectory distillation, so that MCM could learn more information from real videos instead of jittering teacher output.\n- Apply additional temporal losses to maintain stable video outputs, such as constraints on the brightness or optical flow changes.\n\nWe will add additional discussion in the revised version.\n\n**Weakness 2: Fig. 6 caption.** Our Fig. 6 shows only our MCM results adapted to different image dataset styles, not two-stage results. The first and second rows represent the first and last video frames, respectively, similar to Fig. 5. We'll clarify this in the revision.\n\n**Question 1: Watermark in “Ours w/ WebVid” in Fig. 6.** Thank you for noticing the watermark. This is because all WebVid videos contain “shutterstock” watermarks. We use WebVid for a fair comparison with ModelScopeT2V-based video diffusion distillation methods, such as DDIM, DPM++, and LCM. We will elaborate more on this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0AiuHlBAIB", "reviewer": "Reviewer_sR1C", "summary": "This work presented a new consistency based framework for video diffusion model distillation.\nSpecifically, the adversarial loss is leveraged to enhance the video quality and consistency distillation loss is performed in the motion embedding space to learn the video motion patterns effectively. In addition,  the authors proposed mixed trajectory distillation to ensure better alignment between training and inference phases.\nThe experimental results demonstrate the proposed approach could produce more visual-pleasing results compared with previous distillation methods.", "questions": "Could the proposed algorithm achieves satisfactory performance on other video generation tasks such as StableVideoDiffusion (Image-to-video) and AnimateAnyone (human video generation)?", "limitations": "The proposed algorithm is not evaluated on high-resolution video generation diffusion models such as (1024x576 or 768x768).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed disentangled motion-appearance distillation is reasonable and effective.  \n2. The generated results in Fig. 5 and supp are very promising.\n3. The quantitive comparisons in Table 1 and 2 are convincing.", "weaknesses": "1. The adversarial loss is not stable and could the authors employ other manners such as perceptual loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of our novel contributions and promising results. We follow your advice to conduct the following set of experiments.\n\n**Weakness 1: Perceptual loss instead of adversarial loss.** Following your suggestion, we conducted experiments comparing perceptual loss with our approach, using ModelScopeT2V teacher on the WebVid mini val set. The results are summarized in the table below. Our observations are:\nThe adversarial loss outperforms using 1 and 2 sampling steps, benefiting low-step sampling by producing sharper details.\nBoth losses perform similarly using 4 and 8 steps.\nWe do not observe apparent training instability using adversarial loss, thanks to the well-trained feature extractor DINO v2 and the discriminator gradient penalty loss.\nWe will add more analysis and details in the revised version.\n\n\n| Method           | FVD @ 1 step | 2 steps | 4 steps | 8 steps | CLIPSIM @ 1 step | 2 steps   | 4 steps   | 8 steps   |\n| ---------------- | ------------ | ------- | ------- | ------- | ---------------- | --------- | --------- | --------- |\n| Adversarial loss | **703**      | **650** | 767     | 760     | **28.70**        | **30.37** | **30.90** | **30.77** |\n| Perceptual loss  | 826          | 693     | **748** | **742** | 27.31            | 29.32     | 30.47     | 30.71     |\n\n\n\n**Question 1: Results on image-to-video and human video generation.** Following your advice, we conduct the following image-to-video distillation experiment using Stable Video Diffusion (SVD). The quantitative results on MSRVTT are listed below. We make the following observations.\nWe outperform Euler and AnimateLCM on FVD across most sampling steps.\nCompared to AnimateLCM, which finetunes the entire SVD (1.6B parameters), we achieve better performance by training only 5.9% of parameters using a lightweight LoRA (95M parameters).\n\nWe will add these results to our revised manuscript and make our SVD-based MCM checkpoint publicly available.\n\n| Method         | # Params | FVD @ 1 step | 2 steps | 4 steps | 8 steps |\n| -------------- | -------- | ------------ | ------- | ------- | ------- |\n| Euler          | -        | 1639         | 1633    | 1268    | 1043    |\n| AnimateLCM     | 1.62B    | 772          | **442** | 253     | 242     |\n| **MCM (ours)** | **95M**  | **749**      | 463 | **246** | **235** |\n\n\nDue to limited rebuttal time, we demonstrate our method's compatibility with ControlNet for controllable human video generation. Examples are provided in the attached PDF. This integration highlights our MCM’s versatility and potential for broader applications. We will include additional results in the revised version.\n\n\n**Limitation 1: High-resolution video generation evaluation.** Following your advice, we conducted high-resolution video generation evaluations at 768x768 resolution, using our AnimateDiff-based MCM on the WebVid mini val set. The results below demonstrate that our MCM achieves state-of-the-art performance in high-resolution video generation. We've included promising examples of these high-resolution outputs in the PDF attached to our general response.\n\n| Method               | FVD @ 1 step | 2 steps  | 4 steps | 8 steps | CLIPSIM @ 1 step | 2 steps   | 4 steps   | 8 steps   |\n| --------------------- | ------------ | -------- | ------- | ------- | ---------------- | --------- | --------- | --------- |\n| DDIM                  | 5364         | 2654     | 1378    | 973     | 20.23            | 20.53     | 23.67     | 28.93     |\n| DPM++                 | 2371         | 1208     | 973     | 990     | 21.85            | 24.81     | 28.63     | **30.15** |\n| LCM                   | 1273         | 1065     | 979     | 986     | 27.81            | 28.90     | 29.87     | 29.93     |\n| AnimateLCM            | 1673         | 1348     | 1078    | 979     | 25.03            | 27.72     | 29.01     | 29.23     |\n| AnimateDiff-Lightning | 1374         | 1367     | 1297    | 1370    | 28.34            | 29.22     | 29.78     | 29.89     |\n| **MCM (ours)**        | **1108**     | **1037** | **962** | **897** | **29.86**        | **30.21** | **30.75** | 29.98     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for identifying our novel contribution in video diffusion distillation, promising qualitative and quantitative results, and well-written paper. We address all concerns in the individual responses below. Please find the attached PDF file for our additional qualitative results.

All training/inference code and model checkpoints used in our original manuscript and rebuttal will be made publicly available.

Meta-review: Reviewers either recommend acceptance or leaning towards acceptance. Reviewers appreciated the novel technique and thorough experiments. Reviewers raised some concerns related to some missing analysis and discussion and  several of which are addressed in the rebuttal. Reviewers did raise valid concerns and authors are encouraged to do the best of their abilities to address all the concerns in the final version.
Decision: Accept (poster)