State & Image Guidance: Teaching Old Text-to-Video Diffusion Models New Tricks

Authors: Konstantin Sobolev, Mikhail Zhirnov, Arsen Kuzhamuratov, Denis Dimitrov, Andrey Kuznetsov, Anton Konushin
Primary area: generative models
Keywords: Text-to-Video Generation, Diffusion Models, Diffusion Guidance, Zero-shot Image-to-Video Generation

Abstract:
Current text-to-video (T2V) models have made significant progress in generating high-quality video. However, these models are limited when it comes to generating dynamic video scenes where the description per frame can vary dramatically. Changing the color, shape, position and state of objects in the scene is a challenge that current video models cannot handle. In addition, the lack of a cheap image-based conditioning mechanism limits their creative application. To address these challenges and extend the applicability of T2V models, we propose two innovative approaches: **State Guidance** and **Image Guidance**. **State Guidance** uses advanced guidance mechanisms to control motion dynamics and scene transformation smoothness by navigating the diffusion process between a state triplet <initial state, transition state, final state>. This mechanism enables the generation of dynamic video scenes (Dynamic Scene T2V) and allows to control the speed and the expressiveness of the scene transformation by introducing temporal dynamics via a guidance weight schedule across video frames. **Image Guidance** enables Zero-Shot Image-to-Video generation (Zero-Shot I2V) by injecting reference image into the initial diffusion steps noise predictions. Furthermore, the combination of **State Guidance** and **Image Guidance** allows for zero-shot transitions between two input reference frames of a video (Zero-Shot II2V). Finally, we introduce the novel **Dynamic Scene Benchmark** to evaluate the ability of the models to generate dynamic video scenes. Extensive experiments show that **State Guidance** and **Image Guidance** successfully address the aforementioned challenges and significantly improve the generation capabilities of existing T2V architectures.

Decision: reject
Reviewer scores: 8, 5, 5

Reviews and discussion:
<thread>
creation date:
1730195065925

thread_type:
official_review

<thread_object>
creator: reviewer_nps9

<content>
summary:
The paper describes a method for Text To Video (T2V) generation.
The method proposes two extensions: State and Image guidance.
State Guidance uses state triplets (initial, current, last) to help T2V generate the proper video frame.
Image guidance injects noise in the early stages of the diffusion model to steer it in the right direction.
This is then used to: generate more dynamic video sequences, as well as zero shot video generation from a single image, and a video interpolating between two images.

strengths:
The extensions are reasonable and the various experiments does show nice videos produced by the system.

In addition, a new dataset is introduced that, hopefully, will help future contributions to the field.

weaknesses:
The paper feels rushed. (The caption of the teaser figure misplaced the text for sub-figures (B) and (C)).

I wonder what is the novelty of the proposed method given the "Make Pixels Dance" (CVPR'24) paper.
They, too, use a triplet state representation to encourage better video synthesis. 
Yet, I could not find a direct comparison. Can the authors explain why?

There are many results and one must appreciate the work done by the authors, but it is extremely difficult to follow the experimental results and appreciate the contributions. For example, 

1. Please add a reference to the different methods shown in the various tables.
2. I'm not sure the ablation experiments should appear in the main text. 
3. The supplemental material is difficult to navigate. There are many folders and no easy way to navigate and compare the different results presented there.
4. Table 3: The boldface numbers are confusing as they only refer to the method without SG. Yet, in almost each column there is a better alternative, so it's difficult to judge the overall quality of the results.
5. Table 5: It is confusing to compare VC2+IG with three thresholds to TI2V-Zero and then highlight in bold different measures for different thresholds.

questions:
Can the authors elaborate on the comparison to "Make Pixels Dance"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698535673

thread_type:
official_review

<thread_object>
creator: reviewer_enjp

<content>
summary:
This paper proposes a novel video generation guidance method capable of producing videos with drastically different cross-frame descriptions, such as texture changes, morph transformations, and large motions. The core innovation of the method is the state triplet, which decomposes the video into different phases, each with its own distinct description. The state triplet can be generated either from a large language model (LLM) or manually. The proposed method is training-free and has been evaluated on a new video benchmark, achieving performance comparable to methods that require task-specific training.

strengths:
The proposed method is practically useful given that:

* It enables the transformations that would be very difficult to model with pretrained models, such as morph transformation, drastic texture changes over frames. 

* The method is training-free and does not require text-video pairs with the target motion patterns, which is hard to collect in scale. The training free method achieves comparable performance as the compared training-based method.

weaknesses:
- The T2V limitation mentioned in L202 is not convincing enough. The prompts used in the paper are relatively simple, consisting of only one or two sentences. Some related works, such as CogVideoX [1], utilize DiT-based structures and demonstrate that detailed prompts significantly improve video generation quality, both in appearance and motion. Therefore, if we consider a scenario with both detailed video captions and a DiT-based model that relies less on explicit per-frame modeling due to (1) temporal compression in the tokenizer, and (2) stronger spatial-temporal modeling capabilities (i.e., cross-frame modeling rather than per-frame), the limitation highlighted for the T2V model becomes less relevant. This is because we would not be restricted by a limited T2I model (point 2) and would benefit from enhanced spatial-temporal modeling (point 1).  

- The paper lacks key information on how the transition order is maintained. While Eq. 1 models the joint conditional distribution given the prompt triplet, it does not specify how the generated images are constrained to follow the prompt order: initial -> transition -> final stage.  Ensuring this sequential alignment is crucial for achieving controllability and realism in the generated video.

- As mentioned in the limitation section in the supplementary, the method introduces additional hyper-parameters, such as the guidance scale at for the triplet states. Tweaking those hyper-parameter would be a case-specific effort and paper does not propose a principled approach for estimating/optimizing those hyper-parameters. 

- As mentioned in the first item, there are strong models taking much more descriptive prompt as input for video generation. However, the paper does not include the comparison with those methods. The lack of this comparison makes the claim about the T2V limitation and the proposed method less convincing.

[1] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. Yang, Zhuoyi and Teng, Jiayan and Zheng, Wendi and Ding, Ming and Huang, Shiyu and Xu, Jiazheng and Yang, Yuanming and Hong, Wenyi and Zhang, Xiaohan and Feng, Guanyu and others. arXiv preprint arXiv:2408.06072

questions:
In addition to the key information missing mentioned above, I have several questions related to the details of the paper:

1. How to handle different combinations of text and image prompt? For example, what if we have the triplet description and only the end frame of the generated image? How is this case different from the case where we have the triplet description and only the first frame of the generated image?

2. How is the method able to handle the morph transform even although the pretrained model is rarely trained on videos with morphism since it is not common? More discussion on this would be appreciated.

3. Have the authors try a more detailed caption vs. the simple ones used in the paper? Will that lead to better motion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712381031

thread_type:
official_review

<thread_object>
creator: reviewer_sbej

<content>
summary:
# Summary

The paper proposes a training-free framework for generating better motion dynamics and adding image conditions with existing pre-trained T2V models. Extensive experiments demonstrate the effectiveness of the proposed framework


EDIT:

strengths:
# Strengths

- The proposed framework is training-free, and indeed achieves better motion dynamics for the mentioned types of prompts
- The proposed framework outperforms the mentioned baselines

weaknesses:
# Weaknesses

- In Sec. 3, the definition of diffusion models might be incorrect
- The idea of state guidance seems similar to the deforum-like technique used in the stable diffusion user community
- While the guidance schedule seems reasonable, the paper does not mention how it was designed/selected
- Entries in the proposed dynamic scenes benchmark consist of three states tailored for the proposed framework, resulting in inconsistent experiments settings when compared with other baselines which does not support this type of inputs. In that case, is unclear how reliable the proposed benchmark is
- The paper does not mention how generated results were selected. Considering diffusion models can generate various of results from the same input conditions from different seeds, it would be better to report mean+std for each metric and report the success rate of each generation
- For II2V experiments, the paper does not compare with SEINE
- The scale of user study seems relatively limited and the design of user study seems flawed: "more changes" in question 2 does not necessarily indicate the result is better in terms of visual quality. The results with flickering artifacts and incorrect/unfavourable color changes could also be considered as "more changes"

# Other comments (not weaknesses)

- The paper coined a new term II2V, which is actually frame inbetweening/interpolation

questions:
Please refer to the weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1732300797738

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their constructive feedback and the time and effort spent reviewing our paper. In this rebuttal, we aim to provide clear and evidence-based responses to each reviewer's concerns. Please feel free to request further clarifications during the rebuttal period if needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732780866132

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Revision details

comment:
Dear Reviewers,

Thank you for your insightful comments and suggestions on our manuscript. We have made the following revisions, highlighted in blue in the updated manuscript to improve clarity and address your feedback:

Reviewer sBeJ:
1. Added details on result selection in Section 4 (lines 365-367). Mean and standard deviation results for LaVie and VideoCrafter2 are now reported in Appendix A.3 (lines 863-865, Table 8).
2. Included a comparison with SEINE in the II2V experiment (Table 6).
3. Revised the user study methodology and conducted experiments with a larger user group (Section 4, Table 4, and Appendix A.3, lines 855-860).

Reviewer EnJp:
1. Updated T2V limitations to discuss the scarcity of dynamic video scenes in training datasets (Section 1, lines 060-072; Section 3, lines 200-220).
2. Added experiments with the DiT-based model (CogVideoX) and detailed prompts, incorporating prompt enhancement from the CogVideoX web UI (Section 4, lines 405-413; Tables 3 and 4; and Appendix A.7, lines 1036-1041).

Reviewer NpS9:
1. Fixed misplaced labels for sub-figures (B) and (C) in the teaser figure.
2. Added references to the various methods in multiple tables (Tables 3, 4, 5, 6, 8).
3. Moved the table evaluating different guidance schedules and $\xi$ parameter values to Appendix A.3 (Table 7, lines 830-838).

Best regards,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733265552189

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author-Reviewer Discussion Summary

comment:
Dear Reviewers and Area Chairs,

As the discussion period concludes, we would like to summarize our interactions with the reviewers.

We extend our gratitude to reviewers sBeJ, EnJp, and NpS9 for their time and feedback on our submission. All three initially rated our paper a 5—marginally below the acceptance threshold. However, they positively recognized the proposed extensions to the diffusion-based video generation model, noting their effectiveness across various tasks, the high quality of the generated videos, the training-free nature of our method, and the proposed benchmark.

During the discussion period, we did our best to  address all reviewers' concerns:

### **Reviewer sBeJ:**

1. Refined the definition of the diffusion model.
2. Clarified the differences between our method and Deforum.
3. Provided additional baseline comparisons to ensure experimental fairness.
4. Explained the selection of generated results and conducted further experiments to demonstrate statistical consistency.
5. Added comparisons with SEINE in the II2V experiment.
6. Improved the user study methodology and expanded the participant pool.
7. Explained the design of the guidance schedule.

**Result:** Reviewer sBeJ has committed to increasing their rating.

### **Reviewer EnJp:**

1. Conducted additional experiments with CogVideoX on short and enhanced prompts.
2. Clarified how transition order is maintained in State Guidance.
3. Provided recommended values for new hyperparameters.
4. Explained our method's application in new scenarios and the surprising effectiveness of the morph transform.

**Result:** Reviewer EnJp decided to maintain their original rating.

### **Reviewer NpS9:**

1. We clarified aspects of the paper text and experiments.
2. Explained the differences between our method and "Make Pixels Dance."

**Result:** Reviewer NpS9 accepted our clarifications and indicated they had no further questions but did not provide additional information regarding their rating.

Thank you for your attention.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces two sampling methods with state and image guidance to enhance the generative capabilities of text-to-video diffusion models. The authors demonstrate video generation scenarios such as generating dynamic video scenes, zero-shot generation conditioned on the input image, and zero-shot generation conditioned on the first and last frames.

The paper initially received ratings of 5, 5, and 5. After the rebuttal, one of the reviewers increased the rating to 8, and the final ratings became 8, 5, and 5. During the rebuttal phase, the authors presented additional experimental results to address the reviewers' concerns. However, the other reviewers maintained their ratings despite having no further questions. The final ratings showed that both reviewers were not persuaded by the authors' responses. The area chair valued the expertise of these two experienced reviewers and found no strong reason to disagree with their recommendations. Consequently, the area chair regretfully recommended rejecting the paper.