Title: Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-Video Generation

Diffusion Models
Abstract: Despite advancements in Text-to-Video (T2V) generation, producing videos with realistic motion remains challenging. Current models often yield static or minimally dynamic outputs, failing to capture complex motions described by text. This issue stems from the internal biases in text encoding which overlooks motions, and inadequate conditioning mechanisms in T2V generation models. To address this, we propose a novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Our method includes a content encoder for static elements and a motion encoder for temporal dynamics, alongside separate content and motion conditioning mechanisms. Crucially, we introduce text-motion and video-motion supervision to improve the model's understanding and generation of motion. Evaluations on benchmarks such as MSR-VTT, UCF-101, WebVid-10M, EvalCrafter, and VBench demonstrate DEMO's superior ability to produce videos with enhanced motion dynamics while maintaining high visual quality. Our approach significantly advances T2V generation by integrating comprehensive motion understanding directly from textual descriptions. Project page: https://PR-Ryan.github.io/DEMO-project/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DEMO, a text to video diffusion approach that aims at enhancing the movement in the video produced by text to video models. In additional to the classical diffusion loss, DEMO introduces:\n\n* __A text motion loss__: the CLIP text encoder used for the temporal cross attention is fine-tuned - the loss matches the text motion to the motion of the video as computed with the optical flow\n* __A regularization__: to avoid the CLIP text encoder used for the temporal cross attention to diverge too much from the CLIP encoder used for the spatial cross attention\n* __A video motion loss__: to encourage the frame latents output by model to follow the difference between frames in the actual video\n\nThe model is trained on WebVid and compared to other models trained on WebVid on both video quality and motion quality.\n\nSTRENGTHS:\nTechnical contribution is strong:\n* The need for motion focused text encoder is well argued\n* The experiments show that this addition contributes into increasing movement\n\nTechnical clarity is also good:\n* The different losses are presented in a clear and detailed way.\n\nWEAKNESSES:\nThe comparison with other approaches is not 100% fair in the sense that additional parameters are being trained on the text encoder side (a CLIP encoder).\n\nIt might also be that additional parameters are being trained on the temporal cross attention side. This is actually unclear to me, because at the beginning of the method section (section 3) the author define the U-Net 3D this way: \"the temporal transformer consists of temporal self-attention and feed-forward layers\", while some papers do use cross-attention in the temporal self-attention as well.\n\nQUESTIONS:\nThe author define the U-Net 3D this way: \"the temporal transformer consists of temporal self-attention and feed-forward layers\", while some papers do use cross-attention in the temporal self-attention as well. Is that a mistake in the text, or to which specific papers are you referring to as the \"vanilla\" 3D U-Net ?\n\nIs the temporal attention similar to Make-A-Video and is operating on a single patch ? Or do you use full attention across all patches of all other frames ? That also has an impact on the fairness of comparisons / ablations of the importance of the losses that you introduce.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces novel framework called DEcomposed MOtion (DEMO), which enhances motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components. Authors investigate sensitivity of CLIP text encoder to motion descriptions and propose to condition the model on content and motion separately. To obtain a motion encoder, authors propose fine-tune separate CLIP text encoder with losses that  encourage its [eot] token to describe motion better. To this end, authors introduce novel text-motion and video-motion losses to encourage the motion conditioning module to generate and render motion dynamics. Experiment results show that proposed method enhances motion of the T2V model.\n\nSTRENGTHS:\n1. Importance. The proposed method successfully tackles the an important problem of enhancing motion quality in videos generated by T2V models.\n\n2. Results. Experiment results demonstrate the effectiveness of the proposed method.\n\n3. Clarity. The text of the paper is well written and  easy to follow.\n\nWEAKNESSES:\n1. Lack of user study. Qualitative results are supported only by examples of several videos. Such results are not statistically representative. Please See Question 1.\n\nQUESTIONS:\n1. I strongly recommend that the authors incorporate a user study in their research to obtain qualitative results. Additionally, conducting ablation studies with a larger sample size, consisting of a minimum of 50 distinct prompts, and involving at least 15 independent assessors, would greatly enhance the robustness and reliability of the findings.\n\n2. On line 127, the authors mention calculating optical flow between attention maps of different frames. However, the methodology employed to perform this operation lacks clarity in the paper. It is essential for the authors to provide a more detailed explanation of the procedures used to compute optical flow between attention maps. Furthermore, considering that attention maps in diffusion models are often noisy, it is crucial to address the reliability and potential limitations associated with this computation.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for enhancing motion synthesis in T2V generation by decomposing both text encoding and conditioning into content and motion components, called Decomposed Motion (DEMO). To address the issue of inadequate motion representation in text encoding, they decompose text encoding into content encoding and motion encoding processes, focusing on spatial information and temporal information respectively. To solve the problem of reliance on spatial-only text conditioning, they also decompose the text conditioning process into content and motion dimensions to integrate spatial and temporal conditions. Additionally, they incorporate text-motion and video-motion supervision to enhance the model’s understanding and generation of motion.\n\nSTRENGTHS:\n1. The paper is well written overall and it is easy to follow along with the presented concepts and results.\n2. The paper includes experimental results that validate the authors' claims, as well as ablations that offer a better understanding of model design choices.\n3. The motivation for proposing the method is well demonstrated experimentally.\n4. The proposed method is efficient in training, has a small number of parameters and a low inference burden, and does not require additional reference information.\n\nWEAKNESSES:\n1. There is too little content in the related work section.\n2. There is a lack of contrast experiments with other improved methods that rely on external references to exhibit performance differences.\n3. The proposed method is generalized, so why not validate it on other BASE models? From qualitative and quantitative experiments, the performance and various indexes of ModelScope are poor, and while DEMO is useful, it is unclear if there is any improvement when adding DEMO to models with better performance. There is a lack of relevant experimental evidence.\n4. The provided videos are less impressive. The separate files of other methods make it difficult for reviewers to identify the advantages of the proposed method, it would be better to make the comparison in a single file.\n\nQUESTIONS:\nIn line 196, “showing marked improvement in individual frame quality compared to the baseline,” can you explain why the quality of a single frame is increased? As I understand, DEMO is designed to increase the model's motion generation capability.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper aims to improve the motion dynamics generation of the text-to-video generation models. It proposes a framework to decompose both text encoding and conditioning into content and motion components. For text encoding, a CLIP encoder is fine-tuned to encode the motion information in the text prompts better. For text conditioning,  it introduces a motion conditioning module to incorporate motion information in the denoising process.

STRENGTHS:
1. The general idea is interesting, i.e. decompose the text encoding and conditioning into content and motion components to capture the motion prompts and generate motion dynamics better.
2. The pilot study is interesting, which investigates that the CLIP encoder tends to be less sensitive to the motion instructions in the text prompts.
3. This paper is technically clear and easy to follow.

WEAKNESSES:
1. For the pilot study in Fig. 1, only one kind of sentence template is applied. Will the drawn conclusion be the same when the text prompts changes to other formats? This concern is not addressed in the paper.
2. In Fig.3, it is hard to tell which rows are generated by the base model and which rows are generated by the fine-tuned model.
3. From line 179 to line 183, it is hard to evaluate the correctness of the conclusion. Because it is quite hard to distinguish the two generated videos shown in Fig. 3. They look almost the same.
4. For qualitative evaluations. The improvements in generated motions in most shown cases are limited.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
The limitations are discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13