Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning
Authors: Penghui Ruan
Pichao WANG
Divya Saxena
Jiannong Cao
Yuhui Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Video Generation
Diffusion Models
Primary area: diffusion_based_models

Abstract: Despite advancements in Text-to-Video (T2V) generation, producing videos with realistic motion remains challenging. Current models often yield static or minimally dynamic outputs, failing to capture complex motions described by text. This issue stems from the internal biases in text encoding which overlooks motions, and inadequate conditioning mechanisms in T2V generation models. To address this, we propose a novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Our method includes a content encoder for static elements and a motion encoder for temporal dynamics, alongside separate content and motion conditioning mechanisms. Crucially, we introduce text-motion and video-motion supervision to improve the model's understanding and generation of motion. Evaluations on benchmarks such as MSR-VTT, UCF-101, WebVid-10M, EvalCrafter, and VBench demonstrate DEMO's superior ability to produce videos with enhanced motion dynamics while maintaining high visual quality. Our approach significantly advances T2V generation by integrating comprehensive motion understanding directly from textual descriptions. Project page: https://PR-Ryan.github.io/DEMO-project/

Enhancing Motion in Text-to- Generation with Decomposed Encoding and Conditioning
Penghui Ruan 1.2, Pichao Wang"; Divya Saxena', Jiannong Cao Yuhui Shi?t
'The Hong Kong Polytechnic University, Hong Kong, China
SSouthern University of Science and Technology. Shenzhen, China
Amazon, Seattle, United States
penghui. ruan @ connect.poo pot.ct..oo.uhkk,, pichaowang warrann@@gmiil.oom (divsaxen, esjcao) @ coopp-polys... eol.uueddu.kk, shiyh @ wwwtech.edu.crr
Abstract
Despite advancements in Text-to-Vides (T2V) generation, producing videos with realistic motion remains challenging. Current models often yield static or mini- mally dynamic outputs, failing to capture complex motions described by text. This issue stems from the internal biases in text encoding, which overlooks motions, and inadequate conditioning mechanisms in T2V generation models. To address this, we propose novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Our method includes a content encoder for static elements and a motion encoder for temporal dynamics, alongside separate content and motion conditioning mechanisms, Crucially, we introduce moxtomotion and dideo-motion supervision to improve the model's ' under- standing and generation of motion. Evaluations on benchmarks such MSR-VTI UCF-101, ViebVid-10K EvalCrafter, and VBench demonstrate DEMO's superior ability to produce videos with enhanced motion dynamics while maintaining high visual quality. Our approach significantly advances T2V generation by integrating comprehensive motion understanding directly from textual descriptions. Project page:
Introduction
The field of Text-to-Vided (T2V) generation [6 46
has seen significant advancements, especially with the advent of diffusion models. These models have demonstrated impressive capabilities in generating visually appealing videos from textual descriptions. However, persistent challenge remains: generating videos with realistic and complex motions. Most existing T2V models produce outputs that resemble static animations or exhibit minimal camera movement, falling short of capturing the intricate motions described in textual inputs [2 46 19 56 74).
19
56
26
74
3
59 70
This limitation arises from two primary challenges. The first challenge is the inadequate motion representation in text encoding. Current T2V models utilize large-scale languaee models (VLMs), such as CLIP [4)]. as text encoders. These VLMs are highly effective at capturing static elements and spatial relationships but struggle with encoding dynamic motions. This is primarily due to their training focus, which biases them towards recognizing nouns and objects [35, while verbs and actions are less accurately represented [17 69 38). The second challenge the reliance on spatial-onf text conditioning. Existing models often extend Text-to-Image (T2I) generation
"The work does relate author's position Amazon. Corresponning authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

techniques to T2V tasks 21 19, 56 74, 31, applying text information through spatial cross- attention on Trame-by-frame basis. While effective for generating high-quality static images, this approach is insufficient for videos, where motion is a critical component that spans both spatial and temporal dimensions. A holistic approach that integrates text information across these dimensions essential for generating videos with realistic motion dynamics.
Recent efforts to address these challenges have involved incorporating additional control signals such as sketches [5]. strokes [9]
23
60)
1
68
database samples [72], depth maps 33 and
human poses 77 generation process. These signals are derived either from reference videos or pre-trained motion generation models 36 30]. While these approaches improve motion synthesis. they depend on external references or pre-trained models, which may not always be practical. Moreover, they introduce complexity and potential inefficiencies, as they require separate handling of additional data sources.
reference videos 62
34
65 and bounding boxes [55] into the T2V
To address these challenges, we introduce Decomposed Motion (DEMO), a novel framework designed to enhance motion synthesis in T2V generation. DEMO adopts a comprehensive approach by decomposing both text encoding and conditioning processes into content and motion components. Addressing the first challenge, DEMO decomposes text encoding into content encoding and motion encoding processes. The content encoding focuses on object appearance and spatial layout, capturing static elements such as "a girl" and "the road" in the scenario "A girl is walking to the left on the road." " Mcanwhile, the motion encoding captures the essence of object movement and temporal dynamics, interpreting actions like "walking" and directional cues like "to the left." " This separation allows the model to better understand and represent the dynamic aspects of the described scenes. Regarding the second challenge, DEMO decomposes the text conditioning process into content and motion dimensions. The content conditioning module integrates spatial embeddings into the video generation process on frame-by-frame basis, ensuring that static elements are accurately depicted in each frame. In contrast, the motion conditioning module operates across the temporal dimension, infusing dynamic motion embeddings into the video. This separation enables the model to capture and reproduce complex motion patterss described in the text. Moreover, DEMO incorporates novel text-motion and video-motion supervision techniques enhance the model's understanding and generation of motion. Text-motion supervision aligns cross- attention maps with the temporal changes observed in ground truth videos, guiding the model focus on motion information. Video- motion supervision constrains the predicted video latent to mimic the motion patterns of real videos, promoting the generation of coherent and realistic motion dynamics. These supervision techniques ensure that the model not only generates visually appealing videos but also renders the intricate motions described in the text.
To validate our framework, we conduct extensive experiments on several benchmarks, including MSR- VTT (66), UCF-101 [50]. Web 10M [[]. EvalCrafter [1I, and VBench (24]. DEMO achieves substantial improvements in metrics related to motion dynamics and visual fidelity, indicating its superior capability to generate videos that are both visually appealing and dynamically accurate.
2
Related Work
T2V Generation. The T2V domain has made substantial strides, building on the progress in T21 generation. The first T2V model, VDM (2I), introduces a space-time factorized U-Net for temporal modeling, training on both images and videos. For high-definition videos, models like Imagen Video [1), Make-A-Video [46, LaVie (59]. and Show-1 (70) use cascades of diffusion models with spatial and temporal super-resolution. Video (74). Video LDM [4, and LVDM [16] apply latent diffusion for video, working in a compressed latent space. VideoFusion (32 separates video noise into base and residual components. ModelScopeT2V uses ID convolutions and attention to approximate 3D operations. Stable Video Diffusion (SVD) 3 divides the process into T21 pre-training, video pre-training, and fine-tuning and demonstrate the necessity of a well-curated high-quality pretraining dataset for developing a strong base model. Despite these advancements, the generated videos still exhibit limited motion dynamics, often appearing largely static with minimal motion, highlighting an ongoing challenge in achieving dynamic and realistic motions.
T2V Generation with Rich Motion. Generating video with rich motion is still an open challenge
in the field of T2V generation. Existing works [15, 9 23 60 51 68, 72 35
71
6
12
62
73
34

---

[ADU] (NOUILI) [VERB] [ADV] (NOP) the [NDUNLA]
sladly
Padr
Areantly
clunetty
Oroup by [ASU]
304
A abore san crands gatcelyy beside the tree.
sleedy
NL50 0istance
steadily
cluuetty
Content
Hardon iecorsatoonn
Figure Our Pilot Study, We generated set of prompts (262144 in total) following a fixed template, grouping them according to the different parts of speech (POS). These grouped texts are then passed into the CLIP text encoder, and we calculate the sensitivity as the average sentence distance within each group. As shown the left-hand side, compared to POS representing content, CLIP is less sensitive to POS representing motion. (Results are consistent across different templates and different sets of words within each POS. Further details can be found in the appendix.)
65 55 address this challenge by incorporating additional control signals that inherently contain rich motion information. Tune-A- Video [65] proposes ppatial-temporal self-attention into the T2I backbone and trains the model on a single reference video. The model thus learns to generate new videos with motions specified by the reference video. Materzynska er al. 34 follow the idea of T21 customization [13 27 to fine-tune the model and a specific text token on small set of reference videos. The model can then recontextualize with that learned token to generate new videos with specific motions. Dream
[62] further customizes both the appearances and motions given reference images and videos. MotionDirectorr (73 proposes a dual-path Low- Rank Adaptations [22 to decouple the motions and appearances residing in the reference videos. MotionCtrl [60) incorporates object trajectories and camera poses into the T2V generation by conditioning them in the convolution and temporal transformer layers, respectively. Contrasting with these approaches, DEMO prioritizes the generation of videos that exhibit significant motions derived solely from textual descriptions without relying on additional signals.
3 Method
Latent Video Diffusion Models (LVDMs). LVDMs build the diffusion models training a 3D U-Net as the noise predictor, where a VQ-VAE (37] or a VQ-GAN [II] is employed to compress the video into low-dimensional latent space. The 3D U-net consists of nown-sample, middle, and up-sample blocks. Each of these blocks comprises multiple convolution layers augmented by spatial and temporal transformers. The spatial transformer consists of spatial self- attentotennoon, spatial cross-an coossaattention, and feed-forward layers. The temporal transformer consists of temporal self self-attention and beed-forward layers. The 3D U-Net trained with a text encoder to minimize the noise- preeecprei loss in the latent space given follows:
48
20
by
Laiffusion e=, 1 lyplllc c0 t, (p)
(1)
where is the video latent corresponding to x in the pixel space, 1 is the time step, & is a text encoder, p is a text prompt, and e is noise sampled from Gaussian distribution. zt is noisy 20 after steps diffusion forward process given by:
= 20 + vI - aye, ā, = II as
(2)
1=1
where 0g is a pre-defined noise schedule.
3.1 Decomposed Text Encoding
As shown our pilot study in Figure the CLIP text encoder can distinguish different motions, but it is not sensitive to motion as it is to content. Consequently, the text encoding focuses more

---

girl walking the the rood
Wutton Encoder
Content Enecder
Image Gebndding
Votton Drbedding
LN
Rabaad Misiion
(995)
Undin
Modio
(255)
Figure Overview of DEMO Training. As shown in the left-hand side, DEMO incorporate dual text encoding and text conditioning (for simplicity, other layers in the UNet are omitted). As shown in the right-hand side, during training, the Liextmmotion is used to enhance motion encoding, the Cieg is used to avoid catastrophic forgetting, the Lvideo-motion is to enhance motion integration. The snowflakes and flames denote frozen and trainable parameters, respectively.
on content encoding rather than motion encoding. To preserve the generalization ability of the T2V generation model, we retain the original text encoder, referring to it as the content encoder (denoted text encoder, referred
&g). Additionally, Em), which is specifically designed to capture object movement and temporal dynamics in textual descriptions (as shown in the left-hand side of Figure 2). We initialize our motion encoder from CLIP text encoder and then fine-tune it using specialized text-motion supervision, as described below.
introduce
the motion encoder (denoted as
has shown that cross cross-attention attention maps represent the structure of visual content. The cross- attention operation can viewed as a projection of text information into the visual structure domain. With this understanding, encoder's focus more toward motion information by constraining the temporal changes of cross- attention maps closely mimic those observed in ground truth videos, illustrated by the red line in Figure 2
Text-Motion Supervision. Research [27 28, 61
aim to shift the text
Formally, given a noisy video latent 2; at time step f. and a text prompt p. the cross-attention maps A E B XXXxW"xXXX where H' and W are the height and width of video latent the ith cross-attention layer, F the number of frames, is the sequence length, for a cross-attention layer i are defined as follows:
N Σ softmax Q(a) (Atinr
A'
Vdn
((n) W:(n) 21, (()) w(x) (p)
where We and Wx are projection matrices for query and key, i E (I.2, is layer index, (1, N} represents each head multi-head cross-attention, and d,, is the dimension of each head.
We empirically find that the cross-attention maps corresponding to the "[eos]" token, which aggre- gate the whole sentence's semantics, play a pivotal role in generating motion. This aligns with the understanding that motion is global concept and cannot be captured by single word. For instance, phrases like "A baby/dog is walking/running forwards/backward! demonstrate that different combi- nations of words maps related to the "[eos]" token and constrain them to mimic the motion patterns observed in the ground truth videos. This approach forms the basis of text-motion loss, defined follows:
result in significantly different motions. Hence, focus the cross-attention
M
1
Liext-motion =
- - -0
cos(o(A) (oo)) o(zo))
(5)
M
where o a function to extract motion dynamics from a video. our case, we use optical flow to represent the motion dynamics (noting that optical flow is only used during training: during

---

inference, we use only the text prompt input). In light of the potential scale differences between the cross-attention maps and video pixel values, we compute the cosine similarity between them. Additionally, for cross-attention at different spatial resolutions, we downsample the ground truth video to match the spatial resolution of the cross-attention maps.
Regularization. Recall that CLIP 40 is trained with a contrastive learning objective to match texts and images from a group of text-image pairs. However, directly fine-tuning the CLIP text encoder with Equation T and Equation D which differ significantly from the original contrastive learning objective, can easily lead to catastrophic forgetting [2)]. To mitigate this, we introduce regularization term in the penalize the text embedding it diverges from the corresponding image embedding, maintaining alignment with the original CLIP contrastive learning objective, as illustrated by the green line in Figure The regularization loss defined as follows:
objective to preserve its generalization ability. Specifically, we
Leg -Ezoop cos(EE (p), gimg (z ( (10))))
(6)
where gimg represents the CLIP image encoder. Given that there is only one text prompt for the F/2 and compute its image embedding. entire video, we select medium frame zo
3.2 Decompased Text Conditioning
DEMO employs separate content conditioning and motion conditioning modules to incorporate content and motion information. preserve the generative capabilities of our base model, we maintain the original text conditioning module, referred to here as the content conditioning module. We then strategically introduce novel temporal transformer, referred to as the motion conditioning module (detailed structure shown in Figure 2 to incorporate motion information along the temporal axis. To encourage the motion conditioning module to generate and render motion dynamics, we train this module under rideo-motion supervision, as described below.
Video-Motion Supervision. Recall that at each diffusion denosing step I, we can obtain the predicted o at time step 4, which is given by:
vI - Cree (2, Em Ec(p)
20, (t, 24,
Var
This predicted fo,r encapsulates the motion information in the video domain. We then prioritize the motion generation by constraining the predicted žo,r mimic the motion pattern in the real video, as illustrated by the yellow line in Figure 2
We define our video-motion loss as follows:
Lvideo-mation [0,, (=o) (f
where & function to extract motion features from video. Given that Laifiusson a pixel-wise denoising loss (whether raw pixel or latent pixel), choosing & as a general motion representation that is not in pixel space may lead to conflicting objectives due the differing representation spaces. Instead, we choose t as the consecutive frame difference defined as follows:
ক ((20)
= 20 20
11F--1
where FFF denotes the video latent from frame 2 to frame F, and z -1:f-1 denotes the video latent from frame frame 1.
3.3 Joint Training
Our final loss is weighted combination of Liext-motions Ereg, Lvideo-motion- and original diffusion loss Ldittusion as follows:
C Ldiffusion + Coxtt-mooton + 3 + Lvidco-mmotin
(10)
where a, 8, and scaling factors to balance different loss terms.
5

---

Sonel
Slow motion flower petals from blossom, landing softly on ground.
Kid Crafee
Mode/ScapeT22
An old man with white hair is shown speaking.
VideeCrofter2
ScapTT
REVA
Jockeys racing.
Figure 3: Qualitative Comparison. Each video is generated with 16 frames. We display frames 4, 6, 8, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials.

---

4 Experiments
4.1
Implementation Details
To rule out potential dataset bias, we use the same training dataset as our base model ModelScope T2V. Specifically, we use WebVid-I0M M, a large-scale dataset of short videos with textual descriptions as our fine-tuning dataset. The training details and hyperparameters can be found in the Appendix.
4.2 Qualitative Evaluations
In this subsection, conduct qualitative comparison among Va [59] VideoCrafter2 (8). ModeIScopeT2V (56), and DEMO. For a fair comparison, we use the same seed for each of these methods. The comparative analysis is illustrated in Figure where we showcase examples generated by these methods. Upon examination, it is evident that these models are capable of producing high- quality videos. However, a notable distinction arises in the dynamic representation of motion within the generated videos. The ModelScope model, while visually appealing, predominantly generates static scenes. For instance, in the scenario described as "Slow motion flower petals fall from a blossom, landing softly on the ground" (the first example in Figure [. the video generated by ModelScoope captures the petals landing on the ground but lacks the motion of the petals falling. contrast, DEMO significantly outperforms by capturing the essence of motion, producing video where the petals fall slowly and gently to the ground. Similarly, LaVie demonstrates similar issue, as illustrated in the third example, where the jockeys remain largely static. VideoCrafter2 exhibits relatively large motion dynamics but suffers from motion blur, as shown in the third example. Conversely, DEMO vividly captures the jockeys racing, thereby providing more realistic representation. This underscores the advanced capability of DEMO to generate videos that not only visually represent a scene but also dynamically encapsulate the ongoing motion.
4.3
Quantitative Evaluations
Table Results of zero-shot T2V generation on MSR- VTT (Evaluation protocol comparison can be found in the appendix).
Table 2: Results of zero-shot T2V genera- tion on UCF-101 (Evaluation protocol com- parison can be found in the appendix).
Model
FID (4)
F.D
CLIPSIM (T)
Model
IS (1)
FVD (J)
MagicVideo
1290
VagicVideo
74
655.00
Make Video
401
13.17
0.3049
Make Villen
33.00
367.23
Show-1 A0
13.08
538
0.3072
Show-1
200
35.42
394.46
Video LDM
4
0.2929
Vidco LDM
4
33.45
550.61
LaVie (59
0.2949
LaVie 50
$26.30
PYaCo [4
10.21-9.73
PYoCo 44
47.76
355.19
VideoFactory
0.3005
VideoFractor
58
410.00
EMU VIDEO
EMU VIDEO
45
42.70
606.20
SVD k
SVD x
242.02
ModeIScopeT2V
5G
14.89
557
0.2941
ModeIScopeT2V
37.55
17
ModelScope122 Inc-tuned
13.80
536
0.2932
ModeIScopeT2V line-tuned
37.21
612.53
DEMO
11.77
422
0.2965
DEMO
36.35
547.31
Zero-shot T2V Generation on MSR-VTT We evaluate the performance of our model on the MSR-VTT 666 test set by caleulating FID [8]. FVD [44 39] and CLIPSIM (33 metrics. For FID and FVD, in alignment with prior studies [56], we randomly sample 2048 videos and one prompt for each video from the test set. For CLIPSIM,
follow previous works [46] 64] [6] and use nearly 60k sentences from the entire test set to generate videos. As illustrated in Table [ DEMO demonstrates notable advancements over the ModelScopeT2V baseline in terms of video quality metrics. Specifically, DEMO achieves an FID score of 11.77, showing marked improvement in individual frame quality compared the baseline score of 14.89, For FVD, DEMO achieves a score of 422 compared to the baseline of 557, indicating improved overall video quality. It is important to note that the FVD is calculated using an I3D model pre-trained on the Kinetics-400 dataset [5] for action recognition. By computing the FVD over its logits, this metric not only reflects visual quality but also emphasizes motion quality in video generation. Additionally, DEMO improves the CLIPSIM
Results reproduced from our own evaluation.
7

---

Table Results T2V generation on Web 10M (Val).
Model
FID (4)
(U)
CLIPSIM (1)
ModeeScopeT2V ModelScopeT2V fine-tuned
11.14
508
0.2986
10.53
461
0.2952
DEMO
9,86
351
0.3083
Table 4: Results of shoo T2V generation on EvalCrafter.
Video Quality
Motion Quality
Model
VQAA (1)
VQAr (f)
IS (1)
Action Score (1)
Motion AC-Score (7)
Flow Score (1)
15.12
16.88
14.60
75.88
44
2.51
ModeISccppT22 T2V fine lumed
15.89
16.39
14.92
74.23
40
2.72
DEMO w/o Lydoo-mttion
18.78
15.12
17.13
76.20
48
3.11
DEMO
15.65
17.57
78.22
score from 0.2941 0.2965, further demonstrating its superior ability generate high- qugh--ualtty videos that are well-aligned with their textual descriptions.
Zero-shot T2V Generation on UCF-101. For UCF-101 50
report the IS
43 and FVD on the
101 action classes. For IS and FVD, we follow previous works 4 [4] to generate 100 videos for each of the 101 classes. We directly use the class names as prompts. As shown in Table compared with baseline ModeIScoprI2V. we improve the FVD from 628.17 to 547.31. 1. However, we observed a slight decrease in IS, which may be attributed the limited textual information provided by UCF-101 class names, such as "baby crawling" and "cliff diving." These prompts primarily suggest motion, and our model, optimized to emphasize this motion, may have over-focused on this limited information. This overemphasis potentially limited the diversity of generated content, lowering the IS.
T2V Generation on Web Vid- 10M (Val). For WebVid- 10M (1, perform T2V generation on the validation set. As shown in Table we evaluate the FID, FVD, and CLIPSIM, where we randomly sample 5K text-video pairs from the validation set. Our model achieves FIDD score of 9.86, an FVD score of 351, and a CLIPSIM score of 0.3083. These outcomes underscore our framework's substantial enhancement of video quality.
Zero-shot T2V Generation on EvalCrafter. EvalCrafter 31 provides 700 diverse prompts across categories like human, animal, objects, and landscape, each with scene, style, and camera movement description. For our evaluation, we generate one video for each of the 700 text prompts. As shown in Table 4 we have obtained significant improvement over the baseline ModelScopeTPV in both video quality and motion quality. In terms of video quality, DEMO enhances both the Video Quality Assessment for Aesthetics (VQA,) and the IS, albeit with slight decrease in the Video Quality Assessment for Technical Quality (VQAT)- For motion quality, EvalCrafter uses three metrics: Action-Score, Flow-Score, and Motion AC-Score. The Action-Scome based the VideoMAB V2 model (57] and MMAction2 toolbox, measures action recognition accuracy on Kinetics-400 classes, with higher scores indicating better human action recognition. Flow-Score and Motion AC-Score, derived from RAFT model (53) optical flows, evaluate motion dynamics. The Flow-Score measures the general motion dynamics by calculating the average magnitude of optical flow in the video, while the Motion AC-Score assesses how well the motion dynamics align with the text prompt. For motion quality, our model surpasses the baseline across all metrics (Action-Score Flow-Score, and Motion AC-Score), showcasing DEMO's superior ability to generate vidcos characterized by better motion quality and higher motion dynamies.
Zero-shot T2V Generation on VBench. VBench 24 is a comprehensive benchmark to evaluate video quality. In our evaluation of VBench, we focus specifically on motion quality. We report
Table 5: Results of zero-shot T2V generation on VBench.
Model
Motion Dynamics
Human Action (t)
Temporal Flickering (t)
Motion Smoothness(()
ModelSeope
62.50
96.02
Mode line-tuned
63.75
90.40
96.35
96.38
DEMO
68.90
90.60
94.63
96.09

---

four key metrics: Motion Dynamics, Human Action, Temporal Flickering, and Motion Smoothness. As shown in T DEMO significantly improves motion dynamics from 62.50 to 68.90. However, we observed only slight improvement in human action recognition. This indicates that while our model enhances the richness and complexity of motion, it provides limited benefit in improving the accuracy of human action representation. Additionally, we note slight decreases in temporal flickering and motion smoothness. This observation aligns with findings from the VBench paper, which suggest that increased motion dynamics can conflict with temporal flickering and motion smoothness.
4.4 Ablation Studies
Impact of Lreg and Llext-motion- As shown in Figure we compute the sensitivity of our motion encoder with different loss combinations. The red columns indicate the motion encoder with Lusttmmttion only completely loses its ability distinguish different tokens, either motion or content, indicating a serious catastrophic forgetting where the model loses its original knowledge. The green columns show that fine-tuning the motion encoder with Leg only preserves the model's generalization ability but does not increase the motion sensitivity. In contrast, the purple columns demonstrate that when training the motion encoder with both Legg and Luxt-motion- the model gain increased sensitivity to tokens representing motion without losing sensitivity to tokens representing content.
Impact of Liddenmootion- To validate the effectiveness of our video-motion loss, we perform an ablation study on the EvalCrafter dataset. As shown in Table without Livideo-motion the model shows a slight improvement in motion quality compared the baseline. This is because the motion encoder provides the model with enriched motion information for generation. However, without explicitly constraining the model mimic realistic motion, it may still focus on generating high-quality individual frames rather than coherent video sequences with rich motion dynamics. By introducing video-motion loss, the model achieves significantly higher motion quality, demonstrating the importance of this loss in guiding the model in producing videos with enhanced motion dynamics.
Table 6: Ablation study on additional parameters in motion encoder.
Benchmarkk
Metric
ModeIScopeT2V ModeIScope12V fine-tuned ModelScope122 motion encoder
DEMO
FID (1)
14,89
13,80
13.98
11.77
MSRVTT
557
536
CLIPSIM (T)
0.2941
0.2932
0.2935
0.2965
(t)
37.55
37.21
37.66
36.35
UCF-101
FVD ())
628.17
612.53
601.25
$47.31
FID
11.14
10.53
10.45
9.86
WehVid-I0M
458
3511
CLIPSIM (1)
0.2986
0,2952
0.2967
0.3083
15.89
VQ)_T())
16.88
16.39
16. 36.34
15.65
EvalCrafter
IS (T)
14.60
14.92
15.02
17.57
Action Score (1)
75.88
74.23
75.
78.22
Motion AC-Scare (1)
.4
40
46
58
Flow Score
2.51
2.72
2.44
4,89
Motion Dynamics (1)
62.50
63.75
63.50
68.90
Vbench
Human Action (?)
90.40
90.40
90.20
90.60
Temporal Flickering
96.35
Motion Smoothness (7)
96.19
96.38
96.22
96.09
Impact of additional parameters in motion encoder. To rule out the effect of additional parameters introduce by motion encoder, we evaluated the effect of training with CLIP text encoder on the overall model performance. ModelScope (2) fine-tuned version of ModelScope without additional motion encoder parameters, and (3) ModeIScopeT2V with the motion encoder while maintaining its original training loss. As shown in Table 6 observed that the performance of the model with the additional motion encoder parameters is comparable to the fine-tuned version without these extra parameters. This suggests that, without specific supervision encoder parameters is marginal. However, the DEMO model consistently outperforms all variations, demonstrating the effectiveness of our method in improving both video quality and text-video alignment.
then compared three different variations: (1) the original
additional constraints, the effect of the added text
Efficiency Analysis. To validate the efficiency of our proposed methods, we trained the baseline model for the same number of iterations and compared its performance with ours. As shown in Tables
4 and continuing to fine-tune the model results in only marginal improvements in

---

A man is standing in a kitchen talking and then a mixer and carton of milk are shown.
Figure 4: Limitations. DEMO does not support creating videos containing sequential motions specified by text. As shown the example, two motions, "a man standing in a kitchen and talking" and "a mixer and a carton of milk shown". appear simultaneously.
video quality. Additionally, we observed slight degradation in CLIPSIM, indicating that further training may not benefit text-video alignment.
5 Limitations and Future Work
Despite DEMO's efficiency in enhancing motion synthesis without relying on additional signals, it faces significant challenges generating different motions sequentially, as illustrated in Figure {
These challenges likely stem from the text encoder's difficulty in comprehending the order of actions and the motion generation model's limited capability to generate different motions. solution to this issue involves annotating each frame with specific prompt and training the model on
potential
video clips of varying lengths rather than a fixed duration. We consider exploring this direction in our future work.
6 Broader Impacts
Our model achieves higher visual fidelity and motion quality, which can benefit various fields such as content creation and visual simulation. However, our model is fine-tuned web data, specifically VedVid-10M []] As a result, the model may not only learn how to generate videos but also inadvertently learn societal biases present in the web data, which may include inappropriate or NSFW content. Potential post-processing steps, such as applying video classifier to filter out undesirable content, could help mitigate this issue.
7 Conclusion
In this paper, we have presented DEMO, an innovative framework crafted to advance motion synthesis T2V generation. By separating text encoding and text conditioning into distinct content and motion dimensions, DEMO facilitates the creation of static scenes and their dynamic evolution. To encourage our model to focus on motion encoding and motion generation, we propose novel text-motion and video-motion supervision. Our extensive evaluations across various benchmarks have illustrated DEMO's capability to significantly improve motion synthesis, showcasing its potential within the field. In future work, plan to augment T2V datasets with more detailed descriptions and delve into advanced motion embedding techniques. By focusing on these areas, we aim to advance the frontiers of research in this dynamic and rapidly evolving domain.
10

---

8 Acknowledgement
This work is partially supported by National Science Foundation of China (No. 022507106822) Shen- zhen Fundamental Research Program (No. JCY12020011014113359), NSFC/RGC Collaborative Research Scheme (No. CRS PolyU501/23), HK RGC Theme-based Research Scheme (No. PolyU T43-513/23-N) and Research Grants Council of the Hong Kong Special Administrative Region, China (No. PolyU15205924)) We also acknowledge the support from Research Institute for Artificial Intelligence of Things, The Hong Kong Polytechnic University. and Center for Computational Science and Engineering Southern University of Science and Technology.
References
[1] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. In: 2021 IEEE/CVF International Confer- ence Computer Vision (ICCV), pp. 1788-1788. IEEE, Montreal, QC, Canada (Oct https://eeeexllors ieeee
document/9711165/
[2] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song. J., Zhang. Q. Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., Karras, T., Liu, M.Y.: eDiff-I: Text-to-Image Diffusion Models with Ensemble of Expert Denoisers (Mar 2023), htttp://axxiv.oo wtt.o///g/.....o./.rreee 01324 2XXi:2211.01333 [es]
[3] Blatmann, A., Dockhorn, T., Kulal, S., Mendelevitch,
D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., Jampani, V., Rombach, R., Ai, S.: Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
[4] Blattmann, A., Rombach, R., Ling, H.. Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (Apr 2023), wwtp:/wwwwwwrg/////////////
arXiv:2304.08818 [cs]
[5] Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (Feb 2018), http::/arxix...o/ orgo abs/ 1700/ 07750 1XXvv:100.007755 [cs]
[6] Chang, D., Shi, Y., Gao, Q., Fu, J., Xu, H., Song, G., Yan, Q., Yang. X., Soleymani, M.: MagicDance: Realistic Human Dance Video Generation with Motions & Facial Expressions Transfer (Nov 2023), http: org abtp//as...1./. 21112311.11111222 12052 arXiv:2311. 1.20522 [cs]
[7] Chen, H., Xia, M., He, Y., Zhang, Y. Cun, X., Yang. S., Xing, J., Liu, Y., Chen, Q., Wang, X.,
Weng, C., Shan, Y.: VideoCrafterl: Open Diffusion Models for High-Qualin, Video Generation (Oct 2023),
http://rriiv.oor/aa/2221000 www.g//////a/aae 2310. 19512, arXiv:2310.195:
[cs]
[8] Chen, H., Zhang. Y., Cun, X. Xia, M., Wang. X., Weng. C., Shan, Y.: VideoCrafter2: Overcoming Data Limitations for Suga-Quaiity Video Diffusion Models (Jan 2024), http: //arxiv. Seaxziv.org/bb//240000 /bw//401.... 09047, arXiv:2401.0900 [cs]
[9] Chen, T.S., Lin, C.H., Tseng, H.Y., Lin, T.Y., Yang, M.H.: Motion-Conditioned Diffusion Model for Controllable Video Synthesis (Apr 2023), http:// wwwxii.org/abbs/3000 14404 arXiv:2304. 14404 [es]
[10] Dai, X., Hou, J., Ma, C.Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., Yu, M., Kadian, A., Radenovic, F., Mahajan, D., Li, K., Zhao, Y., Petrovic, V., Singh, M.K., Motwani, S., Wen, Y., Song. Y., Sumbaly, R., Ramanathan, V., He, Z., Vajda, P., Parikh, D.: Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack (Sep 2023), hattp://arx: .ww.siv.orrg.... abs / 2309. 15807 arXiv:2309.1553 [cs]
[11] Esser, P., Rombach, R., Ommer, B.: Taming Transformers for High- RegohResoo Image Synthesis (Jun 2021), http: //tp/////xvv org/ wwrg/ba//// /2022 09841] arXiv:2012.0984] [cs]
[12] Feng, M., Liu, J., Yu, K., Yao, Y., Hui, Z., Guo, X., Lin, X., Xue, H., Shi, C., Li, X., Li, A., Kang, X., Lei, B., Cui, M., Ren, P., Xie, X.: DreaMoving: A Human Video Generation Framework based on Diffusion Models (Dee 2023), htttp://axxi: http:://axiiv..o/abbsss//eee wtt./bs//////////// 05107 arXiv:2312.051 [cs]

---

[13] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An Image Worth One Word: Personalizing Text-to-Imas Generation using Textual Inversion (Aug 2022),
org/ org/abs/////oooogooo..ggg.... 2208. 2208.01618 arXiv:2208.01618 [cs]
[14] Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.B., Liu, M.Y., Balaji, Y.: Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (Aug 2023), http:://txiiii org/ htsp//bb/////5/ 10474 arXiv:2305. 10474 [cs]
[15] Guo, Y. Yang, C., Rao, A., Agrawala, M., Lin, D., Dai, B.: SparseCtrl: Adding Sparse
Controls to Text-to- Video Diffusion Models (Nov 2023), http://arxiv.orgg wwt../r////ab/11111// 16933 arXiv:2311.1699 [cs]
[16] He, Y., Yang. T., Zhang. Y., Shan, Y., Chen, Q.: Latent Video Diffusion Models for High-Fidelity Long Video Generation (Mar 2023), http://aaxiiv.orrga//////// MrXiv:2211.1322 [cs]
[17] Hendricks, L.A., Nematzadeh, A.: Probing Image-Languan Transformers for Verb Understand- ing (Jun 2021), http:://axxv..rrg///// orggabs////// /2066 091411 arXiv:2106.0914) [cs]
[18] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs Trained by
a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium (Jan 2018), http [cs, stat]
/ //arxiv. orgooogg/gaa...ooooooo /106..176..000 08500, arXiv:
[19] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., Salimans, T:: Imagen Video: High Definition Video Generation with Diffusion Models (Oct 2022), http::/aar::t http:://arrxv..og/bb////22
org/ 02303 arXiv:2210.02303 [cs]
[20] Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models (Dec 2020), http: //arxiv. org/ ht///bbs//200 11239, arXiv:2006.1 1139 [cs, stat]
[21] Ho, J., Salimans. T., Gritsemko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video Diffusson Models (Jun 2022), http: fistp://arxiv. org wwwpaaawwwvvrrrrraaa.aaaa 220///4..///.2...//// 03458, arXiv:2204.03458 [es]
[22] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., Chen, W.: LoRA: Low-Rank Adaptation of Large Language Models (Oct (021),/ttpp: http::///axxv.eeeee http://aaxivvoogaa////////
09685 arXiv:2 D.X.v:2106.09666 [cs]
[23] Huang, H.P., Su, Y.C.,, Sun, D., Jiang, L., Jia, X., Zhu, Y., Yang, M.H.: Fine-grained Controllable Video Generation via Object Appearance and Context (Dec 2023), /taxii//rxxiv.ooo org/ abs/ mrXiv:2312.029) [cs]
[24] Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu. T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., Liu, Z.: VBench: Comprehensive
Benchmark Suite for Video Generative Models (Nov 2023), htttp://rxxv.orr/ 2brrg//bb//21111
17982 ar X1..:2331111999 [cs]
[25] Kingma, D.P. Ba, Adam: A Method for Stochastic Optimization (Jan 2017), http:// arxiv. abs/ 1412. 6980, arXiv:1412.699 [cs]
[26] Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang. J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., Cheng, Y.. Chiu, M.C., Dillon, J., Essa, I., Gupta, A., Hahn, M., Hauth, A., Hendon, D.. Martinez, A., Minnen, D., Ross, D.. Schindler, G., Sirotenko, M., Sohn, K.,
Somandepalli, K., Wang, H.. Yan, J., Yang. M.H., Yang, X., Seybold, B., Jiang, L.: VideoPoet: A Large Language Model for Zero-Shot Video Generation
[27] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-Concept Customization of Text-to-Image Diffusion
[28] Li, S., de Weijer, J., Hu, T., Khan, F.S., Hou, Q. Wang, Y., Yang, J.: StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing (Aug 2023), httt:://arxxxii www.aaiivv.ogggbba.. org/abs/ 2303. 15649, arXiv:2303.15644 [cs]
[29] Li, Y., Liu, X., Kag. A., Hu, J., Idelbayev, Y., Sagar, D., Wang, Y., Tulyakov, S., Ren, J.: TextCraftor: Your Text Encoder Can be Image Quality Controller (Mar 2024), http::///aat /rrxiv org htr//ass/4400 18978 arXiv:2403.18977 [cs]
12

---

[30] Liang, J., Fan, Y. Zhang, K., Timofte, R., Van Gool, L., Ranjan, R.: MoVideo: Motion-Aware Video Generation with Diffusion Models (Nov http::/ttt/ttti/ org abs///1l//.1a/a 11325 arXiv:2311. 1113111111222 1325 Jes, eess]
[31] Liu. Y., Cun, X., Liu. X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., Shan, Y.: EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (Oct 2023),
http://arxiv.orrs org.ivi..rrraee.. h/tg///////// 2310. . 11440) arXiv:2310.1 11444 [es]
[32] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen. Y., Zhao, D., Zhou, J., Tan, T.: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation (Oct 2023). http://axxiv.ortt org/ abs/ 2303. 08320 ar arXiv:2303.0833
[cs]
[33] Jv, Huang, Y., Yan, M., Huang. J., Liu, J., Liu, Y., Wen, Y., Chen, X., Chen, S.: GPT4Motion: Scripting Physical Motions in Text-to- Video Generation via Blender-Oriented GPT Planning (Nov 2023), http: /tap:iv//axiv...e www./as.////eooooooe//.///..eea/ abs 12631 arXiv:2311.1263 [cs]
[34] Materzynska, J., Sivic, J., Shechtman, E., Torralba, A., Zhang. R., Russell, B.: Customizing Mo- tion in Text-to- Video urXiv:2312.04966 [cs]
Text-to-Video Diffusion Models (Dec 2023),
[35] Momeni, I.,, Caron, M., Nagrani, A., Zisserman, A., Schmid, C.: Verbs in Action: Improving
verb understanding in video-language models (Apr 2023), httt:://atr:tt www....wog.aa//3300000 06708 arXiv:2304.066 0.6608 [es]
[36] Ni, H., Shi, C., Li, K., Huang, S.X., Min, M.R.: Conditional Image-to- Video Generation with Latent Flow Diffusion Models (Mar 2023), http://arxiv.ors wttp:/orrrrrabb/3333331444/
arXiv:2303.153 137444 [cs]
[37] Oord, A.v.d., Vinyals, O., Kavukcuoglu, K.: Neural Discrete Representation Learning (May 2018), 88, http: //atp///rrxvv http:/arivv..rrrrra/ 0rg/003/1111..09337
N0v::171100993 [cs]
[38] Park, J.S., Shen, S., Farhadi, A., Darrell, T., Choi, Y., Rohrbach, A.: Exposing the Limits of Video-Text Models through Contrast Sets. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. pp. 3574-3586 Association for Computational Linguistics, Seattle, United States (2022). https://doi.or// wtw.////.......... org/2022. nww.aaaaccl-a----- -arcc--maiii main. 261
https: attps://aclanthooogyy
[39] Parmar, G., Zhang, R., Zhu, J.Y.: On Aliased Resizing and Surprising Subtleties in GAN Evaluation (Jan 2022), //axxv/ httg/abs/2100..1111 11222 arXiv:2104.1122
[40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, L: Learning Transferable Visual Mod- els From Natural Language Supervision (Feb 2021), (xXiv:2103.0000 [cs]
[41] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models (Apr 2022), http://arxii /atpvv//xxvvvv/// wwwiiiv..rg//aee htg/abs/2112.10755 utXiv:2112.1075: [cs]
[42] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: Fine
Tuning Text-to-lmage Diffusion Models for Subject-Driven Generation (Mar 2023), http: //arxiv.. wwwzziv.org///. www./rgg///bbbbeeeeeeeeeee abs 12242, arXiv:2208.1 12242 [cs]
[43] Salimans, T., Goodfellow, I., Zaremba, W. Cheung, V., Radford, A., Chen, X.: Im- proved Techniques for Training GANs (Jun 2016), httt:://rxxi. wwwpaawwvv...gaabb.. org/abs/ 000//6606.000 03498 AXXiv:6606003499 [cs]
[44] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T. Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarezyk, R., Jitsev, J.: LAION-5B: An open large-scale dataset for training next generation image-text models (Oct 2022), /est:://arxiz org/ abs/ 2210. 08402, arXiv:22 00.08402 [cs]
13

---

[45] Sheynin, S.. Polyak. A., Singer, U., Kirstain, Y., Zohar, A., Ashual, O., Parikh, D., Taigman,
Y:: Emu Edit: Precise Image Editing via Recognition and Generation Tasks (Nov 2023).
http: http///axiv...r org/ abs// 2311 10089
01t:22111.00000 [es]
[46] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang. S., Hu, Q. Yang. H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., Taigman, Y.: Make-A-Video: Text-to Video Generation without Text-Video Data (Sep 2022), http:///axii, org/
14792, eeXiv:2209.14799 [cs]
[47] Smith, L.N., Topin, N.: Super-Convergeence: Very Fast Training of Neural Networks Using Large Learning Rates (May 2018), http://axii.piopo///a//aaaaa wwapr//abb///ie abs/aaass/////////////////////// 07120 arXiv:708.007120 120 [cs, stat]
[48] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan. N., Ganguli, S.: Deep Unsupervised Learn ing using Nonequilibrium Thermodynamics (Nov 2015), 03585 arXiv: 150::03533 [cond-mat, q-bio, stat]
[49] Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models (Oct 2022), http: //arxiv. org/ wwgg/bs///g httpaaa..///../a//// arXiv:2010.02502 [cs]
[50] Soomro, K., Zamir, A.R., Shah, M.: UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild (Dee 2012),
0402 arXiv: 1212.0402 [cs]
[51] Su, S., Guo, L., Gao, L., Shen, H., Song, J.: MotionZeenoExxploiting Motion Priors for Zero-shot Texl-to-Video Generation (Nov 2023), http://arxxiv.rr//aa//221111 arXiv:2311. 16666331.1663551 [cs]
[52] Sun, K., Pan, J., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., Dai, J., Qiao, Y., Wang, L., Li, H.: JourneyD:: A Benchmark for Generative Image Understanding (Oct 2023), http://arrt: /arxivvvvv.. org/ abs/ 000110070000666 00716, arXiv:2307.0071 [cs]
[53] Teed, Z., Deng, J.: RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (Aug 2020),
http://axxvv..oo abs 2003. 12039
ar arXiv:2003.1 1 1209 [cs]
[54] Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: Towards
Accurate Generative Models of Video: A New Metric & Challenges (Mar 2019), http:// Saw.v..org/bbs//111/ ht///bbs/11113/ arxiv.org/ei org/
01717 arXiv:1812.01717 [cs, stat]
[55] Wang. J., Zhang. Y., Zou, J., Zeng, Y., Wei, G., Yuan, L., Li, H.: Boximator: Generating Rich and Controllable Motions for Video Synthesis (Feb 2024), http: /ttttp://axxv....///// wwtp://:/pabaa..ooooooo arXiv:2402.01566 [cs]
[56] Wang. J.. Yuan, H., Chen, D., Zhang. Y., Wang, X., Zhang, S.: ModelScope Texdeto-Video Technical Report (Aug 2023), http: wrw. abs 2 arXiv:2308.0657/ [cs]
[57] Wang, I.., Huang, B., Zhao, Z., Tong, 7., He, Y., Wang. Y., Wang, Y., Qiao, Y.: VideoMAR V2: Scaling Video Masked Autoencoders with Dual Masking (Apr 2023), http://arxxv.orr abs/2303 16727 arXiv:2303. 16727 [cs]
[58) Wang, W., Yang, H., Tuo, 7., He, H., Zhu, J., Ju, Liu, J.: VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to- Video Generation (Jun 2023), http: //tp////a/vvv///////// .rriv.org/ abs/2305. 10874 1rXiv:2305.1087. [cs]
[59] Wang. Y., Chen, X., Ma. X., Zhou, S., Huang, Z., Wang. Y., Yang, C., He, Y., Yu, J., Yang. P., Guo, Y., Wu, T., Si, C., Jiang, Y., Chen, C.. Loy, C.C., Dai, B., Lin, D., Qiao, Y., Liu, Z.: LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models (Sep 2023),
hthp:///rriizn org/ abs/ 220...2399..0 15103 arXiv:2309.151 15100 [cs]
[60] Wang, 7., Yuan, Z., Wang, X., Chen, T., Xia, M., Luo, P., Shan, Y.: MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (Dec 2023),
03641 arXiv:2312.0364 [cs]
[61] Wang 7., Sha, Z.,, Ding, Z., Wang, Y., Tu, Z.: TokenCompose: Grounding Diffusion with Token- level Supervision (Dee 2023), http: / arxiv org/abs///1111 2312. 03626 arXiv:23 12.0062 [cs]
14

---

[62] Wei, Y., Zhang, S., Qing, Z., Yuan, H., Liu, Z., Liu, Y., Zhang, Y., Zhou, J., Shan, H.: Dream Composing Your Dream Videos with Customized Subject and Motion (Dee 2023). [es]
http://arxiin org/ abs 2312. 04433 arXiv:2312.044
[63] Wu, C., Huang, L., Zhang, Q. Li, B., Ji, L., Yang, F., Sapiro, G., Duan, N.: GODIVA: Generating Open-Domaln Videos from nAtural Descriptions (Apr 2021), http:///rxixi org/ abs/2104 14806, arxiv:2204, 14806 [es]
[64] Wu, C., Liang, J., Ji, L.,, Yang, F., Fang, Y., Jiang, D., Duan, N.: NV'UW: Visual Synthesis Pre-training for Neural visUal World creAtion (Nov 2021), http://arxiv, owg. wwtp/rg/////bb/eeeooeeee ab1/
12417. arXiv:2111.122
[es]
[65] Wu, J.Z., Ge, Y., Wang, X., Lei, W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-A- Lune-A-Videos Video: One-Shot Tuning of Image Diffusion Models for Text-to- Video Generation
(Mar 2023), http:/tttp/////// htt.v..xvvvvv. peexiv.or//eee /211..221...22 11565 arXiv:2212.11565 [cs]
[66] Xu, J., Mei, T., Yao, T., Rui, Y.: MSR-VTT: Large Video Description Dataset for Bridging Video and Language. In: 2016 IEEE Conference Computer Vi- sion and Pattern Recognition (CVPR). pp. 5288-5296. IEEE, Las Vegas, NV, USA (Jun 2016). http://doi.org//// http://0o..og11...19///PPP... 1099CCPR.2016.77.... http://ieeexploort ieee org/ document/7780940/
[67] Xue, H., Hang. T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., Guo, B.: Advancing High- Resolution Video-Language Representation with Large-Scale Video Transcriptions (Jul 2022),
httt://arxiz www.aaawwi.ororaaaaaaaaaeea abs/ 2111. 10337] arXiv:21 1..:2111.11333 10337 [cs]
[68] Yin, S.. Wu, C.. Liang, J., Shi. J., Li. H., Ming, G., Duan, N.: DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory (Aug 2023), http
7/arxiv. orgg///.rr///oo ha//abb/22300 08089, arXiv:2308.0808. [cs]
[69] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to about it? (Mar 2023), http:://axii..or orrg
abs/2210 01936, arXiv:2210001936 [es]
[70] Zhang, D.J., Wu, J.Z., Liu, J.W., Zhao, R., Ran, L., Gu, Y., Gao, D., Shou, M.Z.: Show- Marrying Pixel and Latent Diffusion Models for Text-to- Video Generation (Oct 2023), http://arxiv. org/ abs 2309. 15818, arXiv:2309. 158888 [es]
[71] Zhang, M., Cai, Z., Pan, L.,, Hong, F., Guo, X., Yang, L., Liu, Z.: MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model (Aug 2022), http://arxiv.org//
2208. 15001 arXiv:2208.115 15001 [cs]
[72] Zhang. M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang. Liu, Z.: ReMoDiffuse; Retrieval- Augmented Motion Diffusion Model (Apr 2023), http://arxiv.orp: http://awivv.or///bb2/200..1.11 hbtp:///bb//3300111 arXiv:2304.011 1161111600 [cs]
[73] Zhao, R., Gu, Y., Wu, J.Z., Zhang, D.J., Liu, J., Wu, W., Keppo, J., Shou, M.Z.: MotionDirector: Motion Customization of Text-to- Video Diffusion Models (Oet 2023), http://arxiv.or org/ abs/2310, 08466 2rXiv:2310.084..
Text-to-Video
[cs]
[74] Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., Feng. J.: MagicVideo: Efficient Video Gen- eration With Latent Diffusion Models (May 2023), httt:://atxxii http:://axxvvorr//bb//2222 wttp.//rrrr//bb/2...1110. arXiv:2211. 1018 [cs]
15

---

Appendix
9.1 Training Details and Hyperparameters
As shown in 7 we train DEMO using the Adam optimizer 25 with a OneCycle scheduler 47 Specifically, the learning rate varies within the range of [0.00001, 0.00005), while the momentum oscillates between 0.85 and 0.99. We train model using Deepspeed framework with stage 2 zero optimization and cpu offloading. DEMO is trained on 4 NVIDIA Tesla A GPUs with batch size of 24 per GPU. DEMO takes 256 256 images inputs and utilizes VQGAN with compression rate of 8 to encode images into a latent space of 32x 32. DEMO trained with 1000 diffusion steps. We set the classifier-free free guidance scale as 9 with the probability of 0.1 randomly dropping the text during training. For inference, we use the DDIM sampler 49 with 50 inference steps.
Table 7: Training Hyperparameters
Hyperparameter
DEMO
LDM
Compression Rate
8
Latent Shape
32x32x 16
Channels
320
U-net
Channel Multiplier
1,2,4,4
Attention Resolutions
16, 8, 4
Head Channels
32
# of Parameters
1.68B
Dropout Rate
0.1
Architecture
CLIP ViT-H/14 77
Token Length
Motion Encoder
Token Dimension
1024
# of Parameters
354.03M
Proj In & Proj Out Normalization
Linear GroupNorm 32 GEGLU
Activation
Motion Conditioning
Channels
320
Attention Resolutions
16, 8, 4 32
Head Channels
of Parameters
238.32M
DDPM Time Steps
0, 1000
Optimizer
Adam
Learning Rate
0.00001, 0.00005
Scheduler
OneCycle Scheduler
Training
Classifier-free Guidance Scale Loss Weight a
9
0.1
Loss Weight 3
0.3
Loss Weight y
0.
Optical Flow Estimator
Raft (53)
Inference
DDIM Sampling Steps
50
9.2 Pliot Study Details
To test the sensitivity of the motion encoder to parts of speech (POS) representing con- tent and motion information, we generated set of prompts following the template: A [ADJ][NUU, [ | the [NOUN₂]. We then grouped these prompts according to their respective POS categories. Next, we calculated the pairwise sentence similarity within each group using the "[eot]" token to determine sentence similarity. The average similarity within each group, as well as across different groups, was reported. This setup groups different words with the same POS under the same context, thereby climinating potential biases introduced by the context.
16

---

Table 8: Training dataset of current T2V models.
Model
Base Model
Training Dataset
MagicVidoo
14
LDM 41
WebVid- 10M [ 10M from HD-VILA-100M. (5
Interal 7M
Make-A-Video
2.38 from Laion-5B HH WebVid-10M
10M from HD-
VILA- [LLA-100M667
Video 1DM
LDM 41
RDS for driving/WebVill 10M 1 for T2V
MadelScope
6.0
LDM 41
2.38 from Laion-5B [ WebNid-I0M
Show-1
70
DespFloyy MoueeSScupe 12V
56
WebVid 10M
1
LaVie 59)
Stable Diffusion 1.4
Laion5B [4] WehVidd 10M 1 Vimeo-25M 59
PyoCo 14
<Dift-I
1.28 text-image dataset 22.5M text-video dataset
VidooFactory
38
LDM
41
HD- VG-130M [5] WebVid-10M (I
VIDEO
licensed
dalaset
SVD 1
Stable Diffusion 2.1
C1
LVD-F 3 (152M) 250K pre.captooned video clips of high visual fidelity
DEMO
ModeIScopeT2V 58
WebVid-10M
1
Table 9: Comparison of different evaluation protocol on MSR-VTT.
Model
FID (L)
FID-CLIP (1)
(VD
CLIPSIM
Evaluation Protocol
VigioVidco 4
36.50
1290
Text prompt on test set; unknown number.
Make A Video 46
13.17
0.309
FID and CLIPSIM are evaluated on 59794 with text prompt from test set.
Show-1
13.08
538
0.3072
FID and FVD are evaluated with
2048 videos generated
CLIPSIM is ovaluate on 59794 videos with prompts.
Video LDM
4
0.2929
CLIPSIM is calculate on 2990 videos prompts
set.
La Vic 30
0.2949
CLIPSIM is calculate on 2990
videos with prompts from test set. The Make-AAViddoo
PYoCo
25.39-22.14
10.21-9.73
VideoFactery
53
0,3005
CLIPSIM is calculate on 2990 with prompts
ModelScope12V 56
14.89
557
0.2941
FID and FVD are evaluated with 2048 videos generated CLIPSIM is evaluate on 59794 videos with prompts.
ModelSScoeTT2 tune
13,80
536
0.2932
Same as ModelScopeTT2
DEMO
11.77
Same as ModelScopeT2V
We define the sensitivity of our motion encoder as one minus this similarity. The full set of different words within each POS category is defined as follows:
ADJ = { "big", "small", "tall", "short", "fat", "thin", "young". "old")
NOUN; = { ["cat", "dog", "horse", "child", "man", "woman", "bird", "fish") VERB = ("walk", "run", "jump", "crawl", "eat", "swim", "fly", "climb"}
ADV = ={"qui ("quickly", "slowly", "suddenly", "steadily", "cautiously". "briskly", "gracefully", "clumsily")
ADP (""arross", "over", "through", "beside", "against", "under", "above", "near"] NOUN; =""river", "bridge", "mountain", "tree", "house", "lake", "field", "forest")
Given these six categories with eight words each, we have total of 8 262144 prompts. It is noteworthy that we did not observe significant differences when using different templates or different sets of words within each POS. The results were consistent across different setups, and selected these prompts to try to make these prompts meaningful.
9.3 Detailed Training and Evaluation for T2V Models
As shown in Table g existing T2V models are trained using diverse datasets and strategies, leading to various evaluation standards across different datasets, as detailed in Table D and Table I0 Here,
17

---

Table 10: Comparison of different evaluation protocols on UCF-101.
Model
(7)
FVD
Evaluation Protocnl
Video K
655.00
Evaluated vidcos generated with class labels; unknown number.
Make- Aiiieo 46
33.00
367.23
One template sentence per class label; 100 videos per class.
Show-1
35.42
394.46
One template
per class label; videos per prompt for IS; FVD on 2048 sampled vidcos.
Video
33.45
label only. videos per class.
LaVic 5
526.30
Class label only; 100 vidcos class.
PYoCo
44
47.76
355.19
One template sentence per class label: videos per prompt for IS; FVD 2048 sampled videos.
VidenFactory 58
410.00
One template sentence per class lahel; 100 videos
per class.
EMU VIDEO 45
42.70
606.20
One template sentence per class label; 100 videos per class.
SVD
242.02
FVD 13,320 videos using class labels only.
ModelScoopT22
50
37.49
630.23
100 videos per class using class labels only.
ModeIScope T2V fine-tuned
37.21
612.53
100 vidcos per class using class labels only. videos per class using labels only.
DEMO
36.35
we detail and justify our evaluation standards. For our evaluation on MSR-VTV we follow the base model's approach to compute the CLIPSIM on the entire MSR-VTT dataset. For FID computation, CLIP-ViT/B 32 used extract the frame features. For FID and FVD, we randomly sample 2048 videos, following the ModelScopeTT2V paper's methodology. For our evaluation on UCF-101, to eliminate bias introduced by template sentences (as done in several previous works), we directly use the class labels to compute the IS and FVD scores.
9.4 Extended Quantitative Evaluations
To evaluate the generalization of our methods, we applied DEMO on ZeroScope, we report the performance as follows:
Table 11: Quantitative results on ZeroScope.
Benchmark
Metric
ZeroScope
DEMO+ZeroScope
FID (L)
14.57
13.59
MSRVTT
FVD (L)
812
543
CLIPSIM (1)
0.2833
0.2945
IS (t)
37.22
37.01
UCF-101
FVD (L)
744
601
FID (L)
11.34
10.03
Web Vid-
FVD (L)
615
479
CLIPSIM (1)
0.2846
0.2903
VQA A (t)
27.76
33.02
(QA_T())
33.87
37.28
IS (t)
14.20
15.28
EvalCrafter
ActionScore (t)
67.78
72.55
MotionAC-Score (t)
62
FlowScore (1)
1.10
5.25
MotionDynamics (1)
42.72
70.28
HumanAction (+)
67.36
88.34
Vhench
Temporal (L)
97.39
94.83
MotionSmoothness (1)
97.92
95.72
18

---

9.5
Extended Qualitative Evaluations
In this section, we provide extended qualitative comparison between our method and the baseline.
LoVe
VideoCraftar2
ModelScopeT 2V
climbing
Lovie
VideoCrafter2
ModelScopeT 2V
DEMO
Bird view, flying over snowy forest.
LaVie
VideoCrafter2
ModelScopeT 2V
DEMO
A boot accelerating to gain speed.
Figure 5: Extended qualitative comparison. Each video is generated with 16 frames. We display frames 2, 4, 6, 8, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials.
19

---

Lavie
VideoCrafter2
ModelScopet 2V
DEMO
woman wearing Hantu paper fan her hand.
Lable
VideoCrafter2
ModelScopeT 2V
DEMO
turkery, couterclockwiss
Lavle
VideoCrafter2
ModelScopeT
Apples and oranges, clockwise.
Figure 6: Extended qualitative comparison. Each video is generated with 16 frames. We display frames 1, 2, 4, 6, 8, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials.
20

---

Lavie
VideoCrafter2
ModetScopeT
DEMO
Three majestic giraffes graze on the leaves of tall trees in African savarnah LaVle
VideoCrafter2
ModelScopeT2V
DEMO
Superman shaking hands Spiderman with the style watercolor. LaVle
VideoCrafter2
ModetScopeT 2V
DEMO
lion is catching prey.
Figure 7: Extended qualitative comparison. Each video is generated with 16 frames. We display frames 4, 6, 10, 12, 14, 15, and 16. arranged in rows from left right. Full videos available in the supplementary materials.
21

---

Lovie
VideoCrafter2
ModeIScopeT
hockey player execuèes perfect hockey stop, spraying ice in all directions they come to sudden halt. VideoCrafter2
LaVle
ModetScopeT 2V
DEMO
green minecroft
gun.
LaVie
VideoCrafter2
ModelScopeT 2d
DEMO
A man is looking at distart mountain in sci-fi style.
Figure 8: Extended qualitative comparison. Each video is generated with 16 frames. We display frames 2, 4, 6, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials.
9.6 Human Evaluation
To further assess the qualitative performance of our proposed method, we conducted user study comparing approach (DEMO) with several state-of-the-art video generation models. We randomly
22

---

selected 50 prompts from EvalCrafter [I], ensuring diversity across scenes, styles, and objects. For each comparison, 15 annotators evaluated the generated videos in terms of three main criteria: text-video alignment, visual quality, and motion quality. The study compared our method with ModelScope LaVie, and VideoCrafter2.
The participants were asked to select their preferred video between the two models for each prompt. The comparative results summarized in Table [2 Specifically, DEMO consistently outperformed ModeIScopeT2V, Vie, and VideoCrafter2, particularly in terms of motion quality, where it achieved preference rate of 74% over ModelScopeT2V. Additionally, DEMO was favored in text-video alignment and visual quality by 62% and 66%, respectively. However, when compared to La and VideoCrafter2, DEMO showed a lower performance visual quality, which can be attributed to differences in training datasets. Vie and VideoCrafter2 use higher-quality, video and image datasets, such as Vimeo-25M [59] and JDB S2, respectively, while DEMO and ModelScope are trained on the WebVid1OM dataset, which is lower in visual quality.
Furthermore, conducted an additional user study to evaluate the effectiveness of our proposed video-motion supervision term, Lvideo-motion- The results indicated that our method with motion supervision outperformed the version without motion supervision, achieving win rates of 58%, 56%, and 72% in text-video alignment, visual quality, and motion quality, respectively. These findings highlight the significant improvements brought by the video-motion supervision in generating smoother and more realistic motion dynamics.
Table 12: User Study Results:
Comparison between DEMO and Other Models
Methods
VideoVideo Alignment
Visual Quality
Motion Quality
DEMO ModelScopeT22
62%
66%
74%
DEMO vs La
56%
46%
62%
DEMO VideoCrafter2
60%
42%
52%
DEMO vs DEMO w/o Lvideo-motion
58%
56%
72%
In summary, the user study provides strong evidence that DEMO improves motion quality without sacrificing text-video alignment. Despite the lower visual quality compared to models trained on high-quality datasets, our method demonstrates its strength motion generation, key aspect of video realism.
23

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
As shown in abstract and the fourth paragraph in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
As shown in limitations
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
24

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide code for our proposed method in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
25

---

Justification:
We provide the environment requirements and the command to run the experi- ments in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training details are shown in appendix 9
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computation resources in appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: cwwpurppsccccc..oo.oeooooooooe http://ccccccc./ttiicccctiieoaae
Answer: [Yes]
Justification:
The research conducted conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
As shown in broader impacts
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learss from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credited the models and dataset we used in the paper and we list their licenses here.
ModeIScopeT2V: cc-by-ne-4.0
Open-
Web Vid- custom license:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "7YCXlEhaqb", "reviewer": "Reviewer_uuEr", "summary": "This paper aims to improve the motion dynamics generation of the text-to-video generation models. It proposes a framework to decompose both text encoding and conditioning into content and motion components. For text encoding, a CLIP encoder is fine-tuned to encode the motion information in the text prompts better. For text conditioning,  it introduces a motion conditioning module to incorporate motion information in the denoising process.", "questions": "Please refer to the weaknesses.", "limitations": "The limitations are discussed in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The general idea is interesting, i.e. decompose the text encoding and conditioning into content and motion components to capture the motion prompts and generate motion dynamics better.\n2. The pilot study is interesting, which investigates that the CLIP encoder tends to be less sensitive to the motion instructions in the text prompts.\n3. This paper is technically clear and easy to follow.", "weaknesses": "1. For the pilot study in Fig. 1, only one kind of sentence template is applied. Will the drawn conclusion be the same when the text prompts changes to other formats? This concern is not addressed in the paper.\n2. In Fig.3, it is hard to tell which rows are generated by the base model and which rows are generated by the fine-tuned model.\n3. From line 179 to line 183, it is hard to evaluate the correctness of the conclusion. Because it is quite hard to distinguish the two generated videos shown in Fig. 3. They look almost the same.\n4. For qualitative evaluations. The improvements in generated motions in most shown cases are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Below are our responses to the concerns raised:\n\n**Q1**: In the pilot study, employing only one kind of sentence template may not be sufficient to draw the conclusion that the text encoder is less sensitive to the motion instructions in the text prompt.\n\n**Response**: We thank the reviewer for highlighting this concern. Due to page limitations, we included the details of our pilot study in the appendix. As stated in the appendix (lines 527 to 530), \"It is noteworthy that we did not observe significant differences when using different templates or different sets of words within each POS. The results were consistent across different setups, and we selected these prompts to try to make these prompts meaningful.\" We will revise the caption for our pilot study (Fig.1) to include this information and to address this concern as follows:\n\n\"We generated a set of prompts (262144 in total) following a fixed\ntemplate, grouping them according to the different parts of speech (POS). These grouped texts are\nthen passed into the CLIP text encoder, and we calculate the sensitivity as the average sentence\ndistance within each group. As shown on the left-hand side, compared to POS representing content,\nCLIP is less sensitive to POS representing motion. (Results are consistent across different templates and different sets of words within each POS. Further details can be found in the appendix.)\"\n\n**Q2**: In Fig.3, it is hard to tell which rows are generated by the base model and which rows are generated by the fine-tuned model.\n\n**Response**: Thank you for pointing out the potential confusion caused by the arrangement of video frames in Fig.3. This complexity arises because our quantitative evaluation involves multiple dimensions. We need to compare our model with other models, show the differences within each video, and evaluate multiple text inputs.\n\nIn Fig.3, we present video frames generated by four different models: LaVie, VideoCrafter2, ModelScopeT2V (base model), and DEMO (fine-tuned model), across three different text prompts. As mentioned in the caption of Fig.3, \"Each video is generated with 16 frames. We display frames 1, 2, 4, 6, 8, 10, 12, 14, 15, and 16, arranged in two rows from left to right. Full videos are available in the supplementary materials.\"\n\nTo improve clarity, we plan to display these videos in a GIF format in our revised paper. This will help illustrate the differences more effectively. Please refer to Fig.1 in the PDF of Global Response.\n\n\n**Q3**: It is hard to evaluate that our DEMO significantly outperforms the base model with the text prompt \"Slow motion flower petals fall from a blossom, landing softly on the ground (the first example in Fig.3)\" as the frames look almost the same.\n\n**Response**: Thank you for pointing out the potential confusion in our presentation of video frames. The motion between different frames, especially those involving many objects such as flower petals, can be difficult to observe with static frames alone. We have provided video files in the supplementary materials, which clearly show significant differences between these two videos.\n\nAdditionally, we plan to revise Fig.3 in our paper to better highlight these differences. Please refer to Q1 and Fig.1 in the Global Response.\n\n\n**Q4**: For qualitative evaluations, the improvements in generated motions in most shown cases are limited.\n\n**Response**: Thanks for raising your concern. This is due to the fact that the improvements of motion are hardly to observe by only showing sequence of static frames. We have provided video files in the supplementary materials to better observe these improvements. Additionally, we plan to revise Fig.3 in our paper to better highlight these differences. Please refer to Q1 and Fig.1 in the Global Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6w1IvmXthm", "reviewer": "Reviewer_gMRr", "summary": "This paper proposes a method for enhancing motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components, called Decomposed Motion (DEMO). To address the issue of inadequate motion representation in text encoding, they decompose text encoding into content encoding and motion encoding processes, focusing on spatial information and temporal information respectively. To solve the problem of reliance on spatial-only text conditioning, they also decompose the text conditioning process into content and motion dimensions to integrate spatial and temporal conditions. Additionally, they incorporate text-motion and video-motion supervision to enhance the model’s understanding and generation of motion.", "questions": "In line 196, “showing marked improvement in individual frame quality compared to the baseline,” can you explain why the quality of a single frame is increased? As I understand, DEMO is designed to increase the model's motion generation capability.", "limitations": "See weakness.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well written overall and it is easy to follow along with the presented concepts and results.\n2. The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding of model design choices.\n3. The motivation for proposing the method is well demonstrated experimentally.\n4. The proposed method is efficient in training, has a small number of parameters and a low inference burden, and does not require additional reference information.", "weaknesses": "1. There is too little content in the related work section.\n2. There is a lack of contrast experiments with other improved methods that rely on external references to exhibit performance differences.\n3. The proposed method is generalized, so why not validate it on other BASE models? From qualitative and quantitative experiments, the performance and various indexes of ModelScope are poor, and while DEMO is useful, it is unclear if there is any improvement when adding DEMO to models with better performance. There is a lack of relevant experimental evidence.\n4. The provided videos are less impressive. The separate files of other methods make it difficult for reviewers to identify the advantages of the proposed method, it would be better to make the comparison in a single file.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. Below are our responses to the concerns raised:\n\n**Q1**: The related work section contains too little content. \n\n**Response**: We acknowledge that the related work section is limited due to page constraints. In the revised paper, we will expand this section and include additional relevant studies (references omitted here due to limits):\n\n**T2V Generation**: T2V has advanced significantly, leveraging T2I success. The first T2V model, VDM, introduced a space-time factorized U-Net for temporal modeling, training on both images and videos. For high-definition videos, models like ImagenVideo, Make-A-Video, LaVie, and Show-1 use cascades of diffusion models with spatial and temporal super-resolution. MagicVideo, Video LDM, and LVDM apply latent diffusion for video, working in a compressed latent space. VideoFusion separates video noise into base and residual components. ModelScopeT2V, the first open-source T2V diffusion model, uses 1D convolutions and attention to approximate 3D operations. Stable Video Diffusion (SVD) divides the process into T2I pre-training, video pre-training, and fine-tuning.\n\n**Q2**: There is a lack of contrast experiments with other methods that rely on external references.\n \n**Response**: Thank you for raising your concern. Here we list the key differences between our method and other methods that rely on external references to address your concern.\n\n1) Different Goal: Our focus is to enhance motion synthesis in general text-to-video generation without relying on external references, which are often infeasible in real-world scenarios. Methods using external references, like video customization or controllable video generation, aim to enable more control to generate videos with specific object appearances, motions, and other details, which is not our goal.\n\n2) Different Information Sources: Our model takes only sparse textual descriptions. It cannot directly take external references without additional training and adaptation, which is beyond the scope of our paper. Methods using external references have access to more detailed visual signals. Comparing our approach with such methods is unfair due to the disparity in information density.\n\n**Q3**: There is a lack of experiments with other base model to show the generalization of the proposed method. Additionally, the performance of the baseline model is poor.\n\n**Response**:\nWe agree that demonstrating the generalization of our method on other base models would be beneficial. To this end, we have applied DEMO to ZeroScope [1] and included the experimental results in the table below. The results are consistent with those obtained using ModelScopeT2V, showing significant improvements in motion quality without loss of visual quality.\n\n|Benchmark|Metric|ZeroScope|DEMO+ZeroScope|\n|---------|------|---------|--------------|\n|MSRVTT|FID(↓)|14.57|13.59|\n||FVD(↓)|812|543|\n||CLIPSIM(↑)|0.2833|0.2945|\n|UCF-101|IS(↑)|37.22|37.01|\n||FVD(↓)|744|601|\n|WebVid-10M|FID(↓)|11.34|10.03|\n||FVD(↓)|615|479|\n||CLIPSIM(↑)|0.2846|0.2903|\n|EvalCrafter|VQA_A(↑)|27.76|33.02|\n||VQA_T(↑)|33.87|37.28|\n||IS(↑)|14.20|15.28|\n||ActionScore(↑)|67.78|72.55|\n||MotionAC-Score(↑)|44|62|\n||FlowScore(↑)|1.10|5.25|\n|Vbench|MotionDynamics(↑)|42.72|70.28|\n||HumanAction(↑)|67.36|88.34|\n||TemporalFlickering(↑)|97.39|94.83|\n||MotionSmoothness(↑)|97.92|95.72|\n\nAdditionally, ModelScopeT2V is the only fully open-source text-to-video model (dataset, model, and code). While its performance may be lower than some closed-source models, it performs reasonably well especially in motion quality. Below are its performance rankings across various benchmarks against both closed-source and open-source models:\n\n|Benchmark|Metric|Ranking|\n|---|---|---|\n|DEVIL [2]|Dynamics Quality|3/10|\n||Dynamics Control|5/10|\n|FETV [3]|Temporal Quality|1/4|\n|T2V-CompBench [4]|Motion|4/20|\n|VBench [5]|Dynamic Degree|6/29|\n||Human Action|13/29|\n||Temporal Style|10/29|\n\n\n**Q4**: The provided videos are less impressive and the separate files make it difficult to compare those methods.\n\n**Response**: Please refer to Q1 in Global Response.\n\n**Q5**: Why DEMO can improve the individual frame quality (FID) since DEMO is designed to enhance the motion generation?\n\n**Response**: Thank you for raising this concern. The confusion arises from the fact that individual frame quality (FID) involves both the realism of the frame itself and the diversity of frames. The FID [6] calculate the Fréchet Distance between generated frames and real frames as follows:\n\\begin{equation}\n    d(P_R, P_G) = \\| \\mu_R - \\mu_G \\|^2 + \\text{Tr}(\\Sigma_R + \\Sigma_G - 2(\\Sigma_R \\Sigma_G)^{\\frac{1}{2}})\n\\end{equation}\nwhere $\\mu$, and $\\Sigma$ are mean, covariance of real ($R$) and generated ($G$) frames. The mean term measures frame realism by comparing the average feature representation of generated and real frames. The covariance term assesses frame diversity by capturing how well generated frames replicate the variability and feature relationships of real frames.\n\nOur base model uses content conditioning for each individual frame and temporal self-attention for smoothing those frames, which improves motion smoothness but limits motion dynamics and frame diversity. In contrast, DEMO introduces a separate motion conditioning module, enhancing motion dynamics and generating more diverse frames. This increased diversity leads to a better FID score, thus improving individual frame quality.\n\n[1] Zeroscope. https://huggingface.co/cerspense/zeroscope_v1_320s. 2023.\n\n[2] Evaluation of Text-to-Video Generation Models: A Dynamics Perspective. 2024.\n\n[3] Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. 2024.\n\n[4] T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation. 2024.\n\n[5] Vbench: Comprehensive benchmark suite for video generative models. 2024.\n\n[6] Gans trained by a two time-scale update rule converge to a local nash equilibrium. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVJIFx2qLo", "reviewer": "Reviewer_WCt2", "summary": "This paper introduces novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Authors investigate sensitivity of CLIP text encoder to motion descriptions and propose to condition the model on content and motion separately. To obtain a motion encoder, authors propose fine-tune separate CLIP text encoder with losses that  encourage its [eot] token to describe motion better. To this end, authors introduce novel text-motion and video-motion losses to encourage the motion conditioning module to generate and render motion dynamics. Experiment results show that proposed method enhances motion of the T2V model.", "questions": "1. I strongly recommend that the authors incorporate a user study in their research to obtain qualitative results. Additionally, conducting ablation studies with a larger sample size, consisting of a minimum of 50 distinct prompts, and involving at least 15 independent assessors, would greatly enhance the robustness and reliability of the findings.\n\n2. On line 127, the authors mention calculating optical flow between attention maps of different frames. However, the methodology employed to perform this operation lacks clarity in the paper. It is essential for the authors to provide a more detailed explanation of the procedures used to compute optical flow between attention maps. Furthermore, considering that attention maps in diffusion models are often noisy, it is crucial to address the reliability and potential limitations associated with this computation.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Importance. The proposed method successfully tackles the an important problem of enhancing motion quality in videos generated by T2V models.\n\n2. Results. Experiment results demonstrate the effectiveness of the proposed method.\n\n3. Clarity. The text of the paper is well written and  easy to follow.", "weaknesses": "1. Lack of user study. Qualitative results are supported only by examples of several videos. Such results are not statistically representative. Please See Question 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: We strongly recommend to incorporate a user study in the research to obtain statistical qualitative results.\n\n**Response**:\nThank you for your excellent suggestion. We conducted a user study to compare our method with other video generation models. We selected 50 prompts from EvalCrafter [1], covering diverse scenes, styles, and objects. For each model comparison, 15 annotators were asked to select their preferred video from three options: method 1, method 2, and comparable results. They evaluated the videos based on visual quality, motion quality, and text-video alignment. The results are shown as follows:\n|Methods|Text-Video Alignment|Visual Quality|Motion Quality|\n|-------|---------------------|--------------|--------------|\n|DEMO vs ModelScopeT2V|62%|66%|74%|\n|DEMO vs LaVie|56%|46%|62%|\n|DEMO vs VideoCrafter2|60%|42%|52%|\n|DEMO vs DEMO w/o $\\mathcal{L}_{\\text{video-motion}}$|58%|56%|72%|\n\nSpecifically, 74\\% of user feedback indicated that DEMO has better motion quality compared to the baseline ModelScopeT2V. Additionally, 62\\% and 66\\% of the user feedback agreed that DEMO has better text-video alignment and visual quality, respectively. These findings align with our quantitative results, demonstrating that DEMO significantly improves motion quality without compromising visual quality and text-video alignment.\n\nWe also performed a user study to test the effect of our video-motion supervision term, $\\mathcal{L}_{\\text{video-motion}}$, which is designed to guide the motion conditioning module to generate better motion dynamics. We achieved win rates of 58\\%, 56\\%, and 72\\% in terms of text-video alignment, visual quality, and motion quality, respectively, compared to DEMO without video-motion supervision. Additionally, we observed that both ModelScopeT2V and our improved version, DEMO, have lower visual quality compared to LaVie and VideoCrafter2. We attribute this to the fact that the training dataset for ModelScopeT2V and DEMO (specifically WebVid10M) is low quality visually. In contrast, LaVie uses the high-quality video dataset Vimeo-25M [2], and VideoCrafter2 adopts the high-quality image dataset JDB [3] for better visual quality.\n\nWe appreciate the recommendation to incorporate this into the revised paper, and we will include a summary of these findings to enhance the robustness and reliability of our qualitative results.\n\n**Q2**: How to calculate the optical flow between attention maps is not clearly described. Furthermore, considering that attention maps in diffusion models are often noisy, it is crucial to address the reliability and potential limitations associated with this computation.\n\n**Response**: Thank you for pointing out the lack of clarity in our explanation. In our approach, we utilize RAFT [4] to calculate the optical flow, as detailed in Table 6 of the appendix. Basically, we treat cross-attention maps as kind of videos but in a different space compared with real world videos. We then directly apply RAFT to extract the optical flows. We recognize that attention maps in diffusion models can be noisy, which poses challenges for reliable optical flow computation. To address this, we tried a simple thresholding strategy that zeroes out regions with little activation. This approach resulted in slight improvements in the stability and accuracy of the optical flow calculations. Nonetheless, we acknowledge that this is a preliminary solution and that more sophisticated strategies could yield better results.\n\nIn the revised paper, we will provide a more detailed explanation of our methodology for calculating optical flow. This will include a step-by-step description of how we preprocess the attention maps, apply the RAFT model, and handle the inherent noise in the attention maps. Additionally, we will discuss the potential limitations of our current approach and outline possible directions for future work to enhance the robustness and reliability of optical flow calculations in noisy conditions.\n\n[1] Teed, Z., & Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 (pp. 402-419). Springer International Publishing.\n\n[2] Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., ... & Liu, Z. (2023). Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103.\n\n[3] Sun, K., Pan, J., Ge, Y., Li, H., Duan, H., Wu, X., ... & Li, H. (2024). Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36.\n\n[4] Teed, Z., & Deng, J. (2020). Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 (pp. 402-419). Springer International Publishing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdY9sRIbyf", "reviewer": "Reviewer_5zwJ", "summary": "This paper introduces DEMO, a text to video diffusion approach that aims at enhancing the movement in the video produced by text to video models. In additional to the classical diffusion loss, DEMO introduces:\n\n* __A text motion loss__: the CLIP text encoder used for the temporal cross attention is fine-tuned - the loss matches the text motion to the motion of the video as computed with the optical flow\n* __A regularization__: to avoid the CLIP text encoder used for the temporal cross attention to diverge too much from the CLIP encoder used for the spatial cross attention\n* __A video motion loss__: to encourage the frame latents output by model to follow the difference between frames in the actual video\n\nThe model is trained on WebVid and compared to other models trained on WebVid on both video quality and motion quality.", "questions": "The author define the U-Net 3D this way: \"the temporal transformer consists of temporal self-attention and feed-forward layers\", while some papers do use cross-attention in the temporal self-attention as well. Is that a mistake in the text, or to which specific papers are you referring to as the \"vanilla\" 3D U-Net ?\n\nIs the temporal attention similar to Make-A-Video and is operating on a single patch ? Or do you use full attention across all patches of all other frames ? That also has an impact on the fairness of comparisons / ablations of the importance of the losses that you introduce.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Technical contribution is strong:\n* The need for motion focused text encoder is well argued\n* The experiments show that this addition contributes into increasing movement\n\nTechnical clarity is also good:\n* The different losses are presented in a clear and detailed way.", "weaknesses": "The comparison with other approaches is not 100% fair in the sense that additional parameters are being trained on the text encoder side (a CLIP encoder).\n\nIt might also be that additional parameters are being trained on the temporal cross attention side. This is actually unclear to me, because at the beginning of the method section (section 3) the author define the U-Net 3D this way: \"the temporal transformer consists of temporal self-attention and feed-forward layers\", while some papers do use cross-attention in the temporal self-attention as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and insightful questions. Below are our responses to the concerns raised:\n\n**Q1**: The comparison with other approaches is not 100\\% fair since additional parameters (a CLIP encoder) are trained.\n\n**Response**: We appreciate your concern about the fairness of comparisons due to additional parameters being trained on the text encoder side (a CLIP encoder). We conducted an experiment to rule out the effect of the additional parameters in the text encoder. Specially, we trained the baseline ModelScopeT2V model with a new CLIP text encoder while keeping its original training loss. We then compared its performance to the model without the additional CLIP encoder parameters (ModelScopeT2V fine-tuned column in the table). From the experimental results, we observed that the performance of the baseline model with the additional text encoder parameters is similar to that of directly fine-tuning the ModelScopeT2V without additional parameters. This indicates that without appropriate supervision, the impact of the additional parameters in the text encoder has only a marginal influence on the overall performance.\n\n|Benchmark|Metric|ModelScopeT2V|ModelScopeT2V fine-tuned|ModelScopeT2V + text encoder|DEMO|\n|---------|------|-------------|-----------------------|----------------------------|----|\n|MSRVTT|FID (↓)|14.89|13.80|13.98|11.77|\n||FVD (↓)|557|536|552|422|\n||CLIPSIM (↑)|0.2941|0.2932|0.2935|0.2965|\n|UCF-101|IS (↑)|37.55|37.21|37.66|36.35|\n||FVD (↓)|628.17|612.53|601.25|547.31|\n|WebVid-10M|FID (↓)|11.14|10.53|10.45|9.86|\n||FVD (↓)|508|461|458|351|\n||CLIPSIM (↑)|0.2986|0.2952|0.2967|0.3083|\n|EvalCrafter|VQA_A (↑)|15.12|15.89|16.21|19.28|\n||VQA_T (↑)|16.88|16.39|16.34|15.65|\n||IS (↑)|14.60|14.92|15.02|17.57|\n||Action Score (↑)|75.88|74.23|75.20|78.22|\n||Motion AC-Score (↑)|44|40|46|58|\n||Flow Score (↑)|2.51|2.72|2.44|4.89|\n|Vbench|Motion Dynamics (↑)|62.50|63.75|63.50|68.90|\n||Human Action (↑)|90.40|90.40|90.20|90.60|\n||Temporal Flickering (↑)|96.02|96.35|95.45|94.63|\n||Motion Smoothness (↑)|96.19|96.38|96.22|96.09|\n\n\n**Q2**: The definition of the 3D U-Net in your paper states: \"the temporal transformer consists of temporal self-attention and feed-forward layers.\" Some papers also use cross-attention in temporal self-attention. Is this a mistake, and which specific papers are you referring to as the \"vanilla\" 3D U-Net?\n\n**Response**:\nThank you for pointing out the ambiguity in our definition of the 3D U-Net. In the methodology section, when we refer to the 3D U-Net, we are specifically referencing the ModelScopeT2V model. This model employs spatial transformers for spatial conditioning and temporal transformers with temporal self-attention for smoothing individual frames along the temporal dimension. We will clarify this in the revised manuscript to avoid further confusion.\n\n\n**Q3**: How your temporal attention is performed? Do you perform the attention on a single patch or full attention across all patches of all other frames?\n\n**Response**:\nThank you for raising your question about the attention mechanism in our model. The temporal cross-attention in our model operates on a single patch across the frame dimension rather than full attention across all patches of all frames.\n\nOur approach involves decomposing the full 3D cross-attention into two separate components: a 2D spatial cross-attention (content conditioning) and a 1D temporal cross-attention (motion conditioning). This design choice enables us to: 1) Preserve the text-to-video generation capability (generate 2D content for each individual frame separately) of the base model. 2) Allow the text instructions to control the temporal dynamics of the video effectively. By separating the spatial and temporal attention mechanisms, we achieve a balance between computational efficiency and performance. This approach ensures fair comparisons and meaningful ablations of the introduced losses, demonstrating the effectiveness of our conditioning mechanism.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Q1**:The presentation of Fig.3 (qualitative results) is not good. The improved motion are hardly to observe and it is difficult to compare DEMO with other methods.

**Response**: We thank all the reviewers for highlighting this concern. Qualitative comparison of videos generated by different models is inherently challenging, especially when focusing on the dynamic aspects of video generation. We acknowledge that displaying only sequence of static frames makes it difficult to compare and directly observe the improvements. To address this issue, we plan to revise Fig.3 by using GIFs for each individual video instead of sequences of static frames. This approach will provide a much clearer and more effective comparison. The revised version can be found in Fig.1 of the attached PDF. For the best viewing experience, please use Adobe Acrobat Reader or another PDF reader that supports animation, as browser support for this feature is limited.

Meta-review: This paper proposes a method to enhance motion dynamics in T2V generation by latent diffusion modeling. In specific, it decomposes both of the text encoder and the text  conditioning module into their respective content and motion components and applies the motion-specific training losses to focus on motion encoding from an input text and motion generation in video. Experimental results show that the proposed model, DEMO, on a number of T2V benchmarks improves performances over baselines in terms of faithful and realist motion generation while maintaining high image quality.

Overall, the motivation is clear and the proposed method seems to be technically sound. In addition, the authors thoroughly address most of concerns and issues raised by the reviewers with more experiments.

However, the description of the proposed method needs to be improved. Especially, it is unclear what modules are optimized by each loss. Mathematical notations and typos also need to be improved (what notation indicates the text conditioning module?) and fixed (Eq. (1), (8)). There are also a number of questions in the proposed method. The main motivation of the proposed decomposition is a lack of motion encoding by CLIP, but L_reg tries to bring the motion encoded text vector and CLIP image vector closer. Also, why does L_video-motion in Eq. (9) not use the optical flow for the motion guidance like L_text-motion in Eq. (5), why do these two motion guidance approaches use different loss formulations?

In empirical validation, it is hard to recognize improved motions by the proposed method from the current examples, as the reviewers said. Moreover, ablation studies are lacking. For example, there is no analysis on the scaling factors of losses, and on performances without L_text-motion or L_reg. 

I think this paper is a borderline paper including above weaknesses. However, based on considerable technical contribution and overall positive ratings by the reviewers, I would recommend the paper to be accepted while I would recommend the authors to faithfully revise the paper reflecting comments from the reviewers and me.
Decision: Accept (poster)