Compositional Video Generation as Flow Equalization

Authors: Xingyi Yang, Xinchao Wang
Primary area: generative models
Keywords: video generation; compositionality

Abstract:
Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated unprecedented capability to transform natural language descriptions into stunning and photorealistic videos. Despite these promising results, a significant challenge remains: these models struggle to fully grasp complex compositional interactions between multiple concepts and actions. This issue arises when some words dominantly influence the final video, overshadowing other concepts.
    
To tackle this problem, we introduce \textbf{Vico}, a generic framework for compositional video generation that explicitly ensures all concepts are represented properly. At its core, Vico analyzes how input tokens influence the generated video, and adjusts the model to prevent any single concept from dominating. Specifically, Vico extracts attention weights from all layers to build a spatial-temporal attention graph, and then estimates the influence as the \emph{max-flow} from the source text token to the video target token. Although the direct computation of attention flow in diffusion models is typically infeasible, we devise an efficient approximation based on subgraph flows and employ a fast and vectorized implementation, which in turn makes the flow computation manageable and differentiable. By updating the noisy latent to balance these flows, Vico captures complex interactions and consequently produces videos that closely adhere to textual descriptions. We apply our method to multiple diffusion-based video models for compositional T2V and video editing. Empirical results demonstrate that our framework significantly enhances the compositional richness and accuracy of the generated videos.

Decision: reject
Reviewer scores: 8, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730431509662

thread_type:
official_review

<thread_object>
creator: reviewer_kasb

<content>
summary:
This paper presented a new method named Vico, a framework designed for compositional video generation. Different from existing methods that may not reflect the intended composition of elements, the proposed Vico tries to adjust the model to ensure that no single concept dominates. Specifically, Vico calculate the contribution of each text token using max-flow and leverage a sub-graph flow to propagate information. The proposed method can be implemented by inserting into diffusion-based methods. Extensive experimental results verify the effectiveness of the proposed method.

strengths:
1.	A new method named Vico for compositional video generation is proposed.
2.	The proposed Vico leverage max-flow to ensure that no single concept dominates.
3.	Extensive results verify the effectiveness of the proposed method in generation high-quality videos.

weaknesses:
1.	This paper argues that the proposed method explicitly constrains the contribution of each token to be the equal, but in fact there are some words with no contribution or words with outstanding contribution in a sentence, that is, the importance of each word should be different. For example, "On a certain day, a boy is running", where "On a certain day" is not informative, only the following "the boy is running" is informative. The authors should discuss the problem.
2.	The proposed method needs to be optimized at test time, and the resource consumption of optimization such as time and computational overhead should be discussed.
3.	This paper said “While the compositional text-to-image sythesis Liu et al. (2022); Chefer et al. (2023); Kumari et al. (2023); Feng et al. (2023); Huang et al. (2023) has been more studied, the challenge of compositional video generation has received less attention.” My question is that whether these image methods can be extended to compositional video generation task? If not, the authors should give a explanation. If yes,  please give a comparison with them.
4.	Can Vico be used in compositional motion scenarios? Such as "In a room, a cat is running from left to right, a dog is running from right to left", this complex movement scene is very interesting.

--------------------------------
Post review:
Thank the authors for the responses. Given ICLR's high standards, I decided to keep the score at 5. The motivation of this paper is good, but in the current version, the different processing of prompt words may ignore some cases, such as when some words other than verbs and nouns have a certain amount of information. And the proposed method requires test-time optimization, which greatly increases the inference cost, considering that the model is getting larger and larger now.

questions:
Please refer to Weaknesses for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730450770221

thread_type:
official_review

<thread_object>
creator: reviewer_y7gd

<content>
summary:
This paper introduces Vico, a novel framework specifically designed for compositional video generation. Vico employs a maximum flow approach to ensure fair contributions from each input token, integrating sub-graphs, a soft flow strategy, and vectorized path flow computation for efficient inference. Extensive experiments demonstrate that Vico significantly surpasses existing baselines in compositional generation tasks.

strengths:
1. This paper presents a highly innovative solution to the problem, utilizing traditional max flow to address token-level response balancing, thereby achieving effective compositional generation.
2. I appreciate that extensive effort has been put into designing feasible experiments. The authors introduce practical techniques such as subgraph, soft min, and vectorized flow strategies, which significantly enhance inference speed.
3. The experiments are thorough and well-executed, including comprehensive comparative studies and a detailed user study, which provides strong validation for the proposed approach.
4. The visual quality of the generated videos is satisfying, despite the limited number of examples presented.

weaknesses:
1. The primary concern is the lack of comparisons or discussions involving recent text-to-video (T2V) methods. The baseline model, VideoCrafter2, was released over a year ago. To convincingly demonstrate the relevance of the compositional generation problem, the paper should ideally compare against more advanced, recent baselines like OpenSora[1],  CogVideoX[2], or more.
2. Additionally, the paper lacks comparisons to existing compositional video generation models. For instance, methods like LVD[3], which uses LLMs for prompt decomposition with gradient-based layout optimization during inference, VideoTetris[4], which employs spatiotemporal diffusion for multi-object generation, and VideoDirectorGPT[5], which combines an LLM director with spatial convolutions for layout learning, are not discussed. Including comparative studies with these related works would strengthen the evaluation and clarify the advantages of the proposed approach.
3. The paper lacks a theoretical justification for the assumption that each token should have an equal impact on the generated result. In typical prompts, not all tokens necessarily require equal influence; for instance, non-descriptive tokens or function words might logically play a reduced role in the attention process. Given the strength of this assumption, a more rigorous rationale or proof would help validate its applicability.
4. Another strong assumption is made regarding cross-attention in T2V models. The paper’s approach relies on cross-attention being embedded uniformly across all layers and frames, yet many contemporary T2V models, such as CogVideoX[2], Mochi-1[6], and commercial models like Kling[7], utilize 3D attention mechanisms similar to MMDiT. This raises concerns about the applicability of the paper’s assumptions to modern architectures, which may limit its generalizability.
5. The paper contains a few typographical errors that affect readability. For example, in the introduction, the prompt "a bird looks like a cat" is used as an example, but the actual effect of this prompt is not demonstrated.

[1] Open-Sora: Democratizing Efficient Video Production for All, https://github.com/hpcaitech/Open-Sora

[2] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[3] Lian, Long, et al. "Llm-grounded video diffusion models." arXiv preprint arXiv:2309.17444 (2023).

[4] Tian, Ye, et al. "VideoTetris: Towards Compositional Text-to-Video Generation." arXiv preprint arXiv:2406.04277 (2024).

[5] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." arXiv preprint arXiv:2309.15091 (2023).

[6] Mochi-1: https://www.genmo.ai/blog

[7] Kling: https://www.klingai.com/

questions:
1. My primary question is,  do the paper's assumptions hold true for modern 3D attention models, such as the open-source CogVideoX. Can the proposed method balance token contributions effectively when applied to an attention graph in such models?
2. In the case of 3D attention, how much additional computational overhead does the proposed approach introduce? Cross-attention calculations are relatively lightweight, but if full attention is employed, as in MMDiT, would the method’s computations lead to significant cost increases?
3. The paper mainly showcases relatively short prompts. I would like to ask if the proposed method can handle modern, highly descriptive prompts that may reach up to 100 words. Would the method remain feasible for inference, and what would be the associated computational cost in such cases?
4. As a fair comparison, could the paper utilize T2V CompBench[1], a benchmark specifically designed to evaluate compositional video generation, to better assess and compare the performance of the proposed model against other existing models? I wonder how well this method performs under a more comprehensive benchmark.

[1] Sun, Kaiyue, et al. "T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation." arXiv preprint arXiv:2407.14505 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706303623

thread_type:
official_review

<thread_object>
creator: reviewer_juy3

<content>
summary:
This manuscript tackles the problem of video generation, focusing on improving the compositional and complex interactions in the final output. The proposed approach utilizes a flow equalization formulation to ensure that different tokens of input get a fair amount of attention throughout the self-attention layers.

strengths:
Capturing compositional relationships in the final generated output is a very important problem and, to the best of my knowledge, is one of the biggest issues with current SOA in GenAI.
This paper correctly identifies one of the main issues throughout the attention mechanism and tries to improve the contribution of different tokens in attention layers as a test-time optimization. The authors model the information flow of the generative model as a graph, which is a smart and (semi-)novel strategy (for the videos) in my opinion.

weaknesses:
The biggest weakness of the solution is the readability of this paper. It was very hard for me to read through the text and jump from text to mathematical notations and back. I will ask for a few clarifications in the questions block.

questions:
1-  In line 247, the capacity matrix W, what are each row and column? Why does the first row have the `t` index while other rows don't?
2- Lines 241, and 243. What is the difference between e_(i,j), i=j and e_(i,i) difference?
3- Section 3.1: At first glance, it is not clear what is x_t (definition) and how A_i is extracted out of it. 
4- What is token-reweight in Table 1? 
5- There are some valid categories of issues in Video Generation shown in Figure 1. My question is if the lack of fair attribution and attention is the only problem that causes these issues. If not, what are the other factors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717123491

thread_type:
official_review

<thread_object>
creator: reviewer_6lse

<content>
summary:
This paper introduces a novel approach for compositional video generation. The proposed method begins by analyzing the impact of input tokens on the video output, ensuring that no single concept predominates the generated content. The core concept involves computing each text token's contribution to the video generation process through maximum flow. The paper introduces an efficient computational technique by approximating the subgraph flow with a vectorized implementation.Finally, the method has been tested across various diffusion-based video models, and the experimental results confirm its effectiveness in enhancing both visual fidelity and semantic accuracy.

strengths:
1. The proposed method is innovative and can be integrated with existing video generation techniques. The experiments demonstrate the effectiveness of applying the "Vico" method on current models such as AnimaDiff, ZeroScore V2, and VideoCrafter V2, showing notable improvements in results.

2. The paper is well-articulated, featuring thorough theoretical analysis and proof.

weaknesses:
1. The user study (Table 2) is limited to only ten video clips, which is insufficient to conclusively prove the effectiveness of the method.

questions:
n/a
</content>
</thread_object>

</thread>
<thread>
creation date:
1732106793429

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank all reviewers for their constructive feedback

comment:
We sincerely thank all reviewers for their constructive feedback. We deeply appreciate the following positive comments:

- The paper addresses an important problem: `Reviewer JUy3`
- The method is innovative, leveraging max-flow for token balancing: `Reviewer JUy3, Reviewer Y7gd, Reviewer 6LSE`.
- The experiments are thorough and well-executed: `Reviewer 6LSE, Reviewer Y7gd, Reviewer kAsb`.
- The video quality is impressive and compositionally accurate: `Reviewer Y7gd, Reviewer kAsb`.
- The writing is clear and supported by strong theoretical analysis: `Reviewer 6LSE`.

We will address the specific questions and concerns raised by the reviewers in the subsequent sections of this rebuttal.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Vico, an inference-time method designed to enhance compositional video generation. Vico ensures equal influence for each textual token (verbs and nouns only) on the final video output through test-time optimization, dynamically assessing and rebalancing token impacts at each diffusion step. The reviewers recognize the significance of the problem, the (semi-)novel approach, and the extensive results that demonstrate the method's effectiveness.

However, the reviewers also highlight several weaknesses. While many concerns were addressed following the rebuttal, the remaining issues are:

* Test-time optimization overhead: Two reviewers raised concerns about the significant increase in computation time. For instance, Vico requires 200 seconds for CogVideoX-2B, more than doubling the 90 seconds needed by the baseline CogVideoX. The authors responded by outlining plans to integrate the Vico framework into the training phase to mitigate this overhead.

* Balancing strategy: Reviewer kAsb questioned the approach of equalizing only nouns and verbs, suggesting it may oversimplify the process. 

Finally, this paper receives mixed evaluations: accept (low confidence), marginally above the acceptance threshold, and marginally below the acceptance threshold. The opposing Reviewer kAsb remains unwilling to adjust their score after reading the rebuttal, making it a borderline case.

The AC finds Reviewer kAsb's arguments persuasive and agrees with their concerns. While this work shows potential for improvement, the significant overhead introduced by test-time optimization diminishes its practical significance. Regarding the technical design, the AC agrees with Reviewer kAsb's critique that balancing only nouns and verbs may overlook other important word types. This limitation could hinder the model's ability to effectively handle long and complex prompts. This limitation is particularly relevant as recent video generation models often use rephrasers to expand short user prompts into complex structures, incorporating multiple nouns, verbs, and other word types (e.g., attributes, geometric details, relationships) that convey subtle yet essential information.

In light of the above, the AC recommends that accepting this submission may be premature until the concerns are fully resolved.