Compositional Video Generation as Flow Equalization

## Abstract

Abstract Large-scale Text-to-Video (T2V) diffusion models
have recently demonstrated unprecedented
capability to transform natural language descriptions into stunning and photorealistic videos.
Despite the promising results, a significant challenge remains:
these models struggle to fully grasp complex compositional
interactions between multiple concepts and actions. This issue arises when some words dominantly influence the final video, overshadowing other concepts.
To tackle this problem,
we introduce Vico , a generic framework for compositional video generation
that explicitly ensures all concepts are represented properly.
At its core, Vico analyzes how input tokens influence the generated video,
and adjusts the model to prevent any single concept from dominating.
Specifically, Vico extracts attention weights from all layers to build a spatial-temporal attention graph, and then estimates the influence
as the max-flow from the source text token to the video target token.
Although the direct computation of attention flow in diffusion models is typically infeasible,
we devise an efficient approximation based on
subgraph flows and employ
a fast and vectorized implementation,
which in turn makes the flow computation
manageable and differentiable.
By updating the noisy latent to balance these flows, Vico captures complex interactions and consequently produces videos that closely adhere to textual descriptions. We apply our method to multiple diffusion-based video models for compositional T2V and video editing. Empirical results demonstrate that our framework significantly enhances the compositional richness and accuracy of the generated videos. Visit our website at https://adamdad.github.io/vico/ .

## 1 Introduction

Humans recognize the world compositionally. That is to say, we perceive and understand the world by identifying parts of objects and assembling them into a whole. This ability to recognize and recombine elements—making “infinite use of finite mean”—is crucial for understanding and modeling our environment. Similarly, in the realm of generative AI, particularly in video generation, it is crucial to replicate this compositional approach.

Despite advancements in generative models, current diffusion models fail to capture the true compositional nature of inputs. Typically, some words disproportionately influence the generative process, leading to visual content that does not reflect the intended composition of elements. While the compositional text-to-image sythesis [, , , , ] has been more studied, the challenge of compositional video generation has received less attention. This oversight is largely due to the high-dimensional nature of video and the complex interplay between concepts and motion.

As an illustration, we highlight some failure cases in Figure 1 (Left), where *certain words dominate* while others are underrepresented. Common issues include *missing subject* and *spatial confusion*, where some concepts do not appear in the video. Even with all concepts present, *semantic leakage* can occur, causing attributes amplified incorrectly, for example, *a bird looks like a cat*. A challenge specific to T2V is *Motion Mixing*, where the action intended for one object mistakenly interacts with another, such as generating a flying wale instead of flying balloon.

To address these challenges, we present Vico, a novel framework for compositional video generation
that ensures all concepts are represented equally.
Vico operates on the principle that,
each textual token should have an equal opportunity
to influence the final video output. At our core,
Vico first assesses and then rebalances the influence of these tokens.
This is achieved through test-time optimization, where we assess and adjust the impact of each token at every reverse time step of our video diffusion model. As shown in 1, Vico resolves the above questions and provides better results.

One significant challenge
is accurately attributing text influence.
While cross-attention [, , , ] provides faithful attribution
in text-to-image diffusion models, it is not well-suited for video models. It is because such cross-attention
is only applied on spatial modules along,
treating each frame independently,
without directly influencing temporal dynamics.

To surmount this, we develop
a new attribution method for T2V model, termed *Spatial-Temporal Attention Flow* (ST-flow). ST-flow considers all attention layers of the diffusion model, and views it as a spatiotemporal flow graph. Using the maximum flow algorithm, it computes the flow values, from input tokens (sources) to video tokens (target). These values serve as our estimated contributions.

Unfortunately, this naive attention max-flow computation is, in fact, both computationally expensive and non-differentiable.
We thus derive an efficient and differentiable approximation for the ST-Flow. Rather than computing flow values on the full graph,
we instead compute the flow on all subgraphs.
The ST-Flow is then
estimated as the maximum subgraph flow.
Additionally, we have develop a special matrix operation to compute this subgraph flow in a fully vectorized manner, making it approximately 100$\times$ faster than the exact ST-flow.

Once we obtain these attribution scores, we proceed to optimize the model to balance such contributions. We do this as a min-max optimization, where we update the latent code, in the direct that, the least represented token should increase its influence.

We implement Vico on multiple video applications, including text-to-video generation and video editing. These applications highlight the framework’s flexibility and effectiveness in managing complex prompt compositions, demonstrating significant improvements
over traditional methods in both the accuracy of generated video. Our contributions can be summarized below:

- •
We introduce Vico, a framework for compositional video generation.
It optimizes the model to ensure each input token fairly influences the final video output.
- •
We develop ST-flow, a new attribution method that uses attention max-flow to evaluate the influence of each input token in video diffusion models.
- •
We derive a differentiable method to approximate ST-flow by calculating flows within subgraphs. It greatly speed up computations with a fully vectorized implementation.
- •
Extensive evaluation of Vico in diverse settings has proven its robust capability, with substantial improvements in video quality and semantic accuracy.

## 2 Preliminaries

Denoising Diffusion Probabilistic Models. Diffusion model reverses a progressive noise process based on latent variables. Given data $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$ sampled from the real distribution, we consider perturbing data with Gaussian noise of zero mean and $\beta_{t}$ variance for $T$ steps/
At the end of day, $\mathbf{x}_{T}\to\mathcal{N}(0,\mathbf{I})$ converge to isometric Gaussian noise. The choice of Gaussian provides a close-form solution to generate arbitrary time-step $\mathbf{x}_{t}$ through

$$ $\displaystyle\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar {\alpha}_{t}}\bm{\epsilon},\quad\text{where}\quad\bm{\epsilon}\sim\mathcal{N}( 0,\mathbf{I})$ (1) $$

where $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$. A variational Markov chain in the reverse process is parameterized as a time-conditioned denoising neural network $\bm{\epsilon}_{\bm{\theta}}(\mathbf{x},t)$ with $p_{\bm{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};
\frac{1}{\sqrt{1-\beta_{t}}}(\mathbf{x}_{t}+\beta_{t}\bm{\epsilon}_{\bm{\theta
}}(\mathbf{x},t)),\beta_{t}\mathbf{I})$. The denoiser is trained to minimize a re-weighted evidence lower bound (ELBO) that fits the noise

$$ $\displaystyle\mathcal{L}_{\text{DDPM}}$ $\displaystyle=\mathbb{E}_{t,\mathbf{x}_{0},\bm{\epsilon}}\Big{[}||\bm{\epsilon }+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\bm{\theta}}(\mathbf{x},t)||_{2}^{2} \Big{]}$ (2) $$

Training with denoising loss, $\bm{\epsilon}_{\bm{\theta}}$ equivalently learns to recover the derivative that maximize the data log-likelihood [, , ].
With a trained $\bm{\epsilon}_{\bm{\theta}^{*}}(\mathbf{x},t)\approx\nabla_{\mathbf{x}_{t}}
\log p(\mathbf{x}_{t})$, we generate the data by reversing the Markov chain

$$ $\displaystyle\mathbf{x}_{t-1}\leftarrow\frac{1}{\sqrt{1-\beta_{t}}}(\mathbf{x} _{t}+\beta_{t}\bm{\epsilon}_{\bm{\theta}^{*}}(\mathbf{x},t))+\sqrt{\beta_{t}} \bm{\epsilon}_{t};$ (3) $$

The reverse process could be understood as going along $\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t})$ to maximize the likelihood.

Text-to-Video (T2V) Diffusion Models. Given a text prompt $y$, T2V diffusion models progressively generate a video from Gaussian noise. This generation typically occurs within the latent space of an autoencoder [] to reduce the complexity. The architecture design of T2V models often follows either a 3D-UNet [, , , , ] or diffusion transformer [, , ]. For computational efficiency, these architectures commonly utilize separate self-attention [] for spatial and temporal tokens. Moreover, cross-attentions is applied on each frame separately, thereby injecting conditions into the model. More related work is in Appendix C.

Maximum-Flow Problem. [, , ] Consider a directed graph $G(V,E)$ with a source node $s$ and a target node $t$. A flow is function on edge $f:E\to\mathbb{R}$ that satisfies both *conservation constraint* and *capacity constraint* at every vertex $v\in V\backslash\{s,t\}$. This means the total inflow into any node $v$ must equals its total outflow, and the flow on any edge cannot exceed its capacity. The flow value $|f|=\sum_{e_{s,v}\in E}f(s,u)$ is defined as the total flow out of the source $s$, which is equal to the total inflow into the target $t$, $|f|=\sum_{e_{u,t}\in E}f(u,t)$. The maximum flow problem is to find a flow $f^{*}$ that maximizes this value.

## 3 Vico: Compositional Video Generation as Flow Equalization

In this paper, we solve the problem of compositional video generation by equalizing influence among tokens. We calculate this influence using max-flow within the attention graph of the T2V model and ensure efficient computation. We define our problem and optimization scheme in Sec 3.1. The definition of ST-Flow and its efficient computation are discussed in Sections 3.2 and 3.3.

Figure: Figure 2: Overall pipeline of our Vico. Before each denoising step, Vico extracts attention maps from each layer to build a spatiotemporal graph. We calculate the attribution scores as max-flow in the graph and adjust the noisy latent code to balance this flows.
Refer to caption: x2.png

### 3.1 Overall Pipeline and Optimization

Our objective is to generate a video from an input prompt $P$, with $K$ text tokens of interest $\mathcal{V}=\{v_{1},\dots,v_{K}\}$. We aim to ensure that each token fairly contributes to the final video. This process is detailed in Figure 2.

Objectives. To achieve this, we define an attribution function $A_{i}=A(v_{i})\in\mathbb{R}$ for each token $v_{i}$, quantifying its impact on the video. We optimize the attribution scores to ensure fairness:

$$ $\displaystyle\max_{\mathbf{x}_{t}}\mathcal{L}_{\text{fair}}(A_{1},\dots,A_{K}) =\max_{\mathbf{x}_{t}}\min_{v_{i}}\{A_{1},\dots,A_{K}\};$ (4) $$

Here, $\mathcal{L}_{\text{fair}}=\min_{v_{i}}\{A_{1},\dots,A_{K}\}$ serves as the fairness function, focusing on the least represented token. By updating the noisy latent $\mathbf{x}_{t}$ to maximize $\mathcal{L}_{\text{fair}}$, we ensure equal contributions across all tokens. Specifically, we estimate $A_{i}$ as flow in attention graph, which will be discussed later.

Optimization. To implement Eq 4, we perform test-time optimization. Before each denoising step, we first feed $\mathbf{x}_{t}$ into the model, extract the $A_{i}$, and update $\mathbf{x}_{t}$ through gradient ascent: $\hat{\mathbf{x}}_{t}\leftarrow\mathbf{x}_{t}+\eta\nabla_{\mathbf{x}_{t}}
\mathcal{L}_{\text{fair}}(A_{1},\dots,A_{K})$. $\eta$ is the step size. Then, $\hat{\mathbf{x}}_{t}$ is going through a denoising step to get $\mathbf{x}_{t-1}$ according to Eq 3. We repeat these steps until the video is generated.

### 3.2 Attention Flow Across Space and Time

With above formulation, our focus is to develop an efficient and precise attribution $A_{i}$. Recognizing issues with cross-attention, we instead calculate $A_{i}$ as the flow through the entire attention graph.

Flawed Cross-Attention in Text-to-Video Models. Cross-attention score has been instrumental in attributing [] and controlling layout and concept composition in text-to-image models [, , ]. However, applying it to T2V diffusion model introduces new problem.

This problem arises because T2V models typically employ cross-attention on spatial tokens only [, , ]. It treats the video as a sequence of independent images, and temporal self-attention mixes tokens across different frames. Consequently, this separation hinders cross-attention’s ability to capture video dynamics, making it challenging to manage actions across frames.

For example, applying the cross-attention-based DAAM attribution [] on VideoCrafterv2 reveals significant issues in visualization. As shown in Figure 3 (Left), cross-attention leads to a flickering pattern in the attention maps, failing to consistently highlight the target object across frames.

Recognizing these limitations, we propose a new measurement termed *Spatial-Temporal Flow (ST-Flow)*, which estimates the influence throughout the entire spatiotemporal attention graph in the video diffusion model. As seen in Figure 3 (Right), ST-Flow gets heatmap with improved consistency.

Figure: Figure 3: Attribution heatmap comparison between DAAM and ST-Flow.
Refer to caption: x3.png

Attention as a Graph Over Space and Time. In our approach, we conceptualize the stacked attention layers as a directed graph $G=(V,E)$, where nodes represent tokens and edges weighted by the influence between tokens. A 4-layer example is illustrated in Figure 2 (Right).

Its adjacency matrix is built using attention weights and skip connections []. Suppose $w^{att}_{i,j}$ is the $i$-th row $j$-th column element of attention matrix averaged across heads.
For self-attention, the edge weight $e_{i,j}$ between any two tokens, $i$ and $j$, is $e_{i,j}=w^{att}_{i,j}+1$ if $i=j$, indicating a skip-connection, and $e_{i,j}=w^{att}_{i,j}$ if $i\neq j$. In the case of cross-attention, edge $e_{i,j}=w^{att}_{i,j}$ connects text to video, and $e_{i,i}=1$ for connections within video tokens due to skip connections. Given that connections only exist from one layer to the next, the resulting matrix exhibits block-wise sparsity pattern. This is expressed as $\bm{W}=\begin{bmatrix}\bm{0}&E_{t,1}&\bm{0}&\dots&E_{t,l}\\
\bm{0}&\bm{0}&E_{2}&\dots&\bm{0}\\
\bm{0}&\bm{0}&\bm{0}&\dots&\vdots\\
\bm{0}&\bm{0}&\bm{0}&\dots&E_{l}\\
\end{bmatrix}$. Here, $E_{i}$ denotes the edge weights within video, and $E_{t,i}$ indicates the influence from text to video at each cross-attention layer.

Attribution as Flow on Graph. Given graph $G$, we compute the attribution $A_{i}$ by analyzing all paths from a text token $v_{i}$ to video tokens at the output layer. As such, we formulate it as a *max-flow problem* with capacity matrix $\bm{W}$. To facilitate this, we add an auxiliary target node $\bm{v}_{t}$ to $G$, connecting it to all output video tokens with inflow edges $e_{\bm{v}_{t}^{+}}=1$(^1^11The maximum inflow is 1 for each node due to softmax normalization in the attention.). We treat each text token $v_{i}$ as the source, and $\bm{v}_{t}$ as the sink. The max-flow from source to sink quantifies the influence of $v_{i}$, termed *ST-Flow*.

Definition 1 (ST-Flow). In attention graph $G$ with capacity matrix $\bm{W}$, a input token $v_{i}$ as source and sink node $\bm{v}_{t}$, the attribution value of $A_{i}=|f|^{*}$ is computed as the maximum flow from $v_{i}$ to $\bm{v}_{t}$.

Our ST-Flow can be considered as an extension of Attention Flow [], incorporating all attention layers in diffusion model. It is proved to be a kind of Shapley Value [], which is an ideal contribution allocation in game theory [, , ] and interpretable AI model [, ].

Exact ST-Flow Computation is infeasible. While theoretically possible, calculating the ST-Flow in T2V diffusion models faces practical issues that render it infeasible:

- •
Non-Differentiable. The max-flow algorithm, by its nature, is non-differentiable. This is a problem when we do gradient-based optimization in Eq 4.
- •
Efficiency Issue. Solving max-flow for each input token is slow. Even with the Dinic’s algorithm [] (^2^22Given that the attentions has more edge than tokens, Dinic is best choice in theory. However, our implementation shows that max-flow on each token takes $\sim$8s.), the time complexity is $O(K|V|^{2}|E|)$ for large attention graphs extracted from video diffusion model.

Despite these obstacles, in Sec 3.3, we derive a min-max approximation to circumvent these issues.

### 3.3 Differentiable ST-Flow with Min-Max Path Flow

As discussed above, exact computation of ST-Flow is challenging.
Instead of directly estimating the ST-Flow, we approach this by focusing on approximating its lower bound, which is computationally feasible. This is made possible, since any sub-graph has max-flow smaller than that of full graph.

Theorem 1 (Sub-Graph Flow)(^3^33Proof in Appendix A). For any sub-graph $g$ of a graph $G$, $g\subseteq G$, the maximum flow $f^{*}_{g}$ in $g$ is less than or equal to the maximum flow $f^{*}_{G}$ in $G$, $|f^{*}_{g}|\leq|f^{*}_{G}|$.

Based on this theorem, we need not compute the ST-Flow directly. Instead, we sample multiple subgraphs $g$ from $G$, calculate the maximum flow for each, and take the highest value among these:

$$ $\displaystyle|f_{G}|\geq A_{i}=\max_{\forall g\subseteq G}|f_{g}|;$ (5) $$

This approach allows for a more efficient calculation by focusing on a manageable number of subgraphs, solving the max-flow for each, and identifying the maximum flow.

In this work, we focus on the simplest type of subgraph in graph $G$: a path from a $v_{i}$ to target $\bm{v}_{t}$. We efficiently approximate the ST-Flow by computing the *max path flow* for each path. We propose two min-max strategies to achieve this:

- •
Hard Flow Strategy. For each text token $v$, we sample all paths $v_{i}$ to $v_{t}$. The max-flow on each path is calculated as the minimum edge capacity along the path, $|f|=\min_{j}e_{j}$. And the best approximated $A_{i}=\max|f|$ is the maximum of these minimums across all paths.
- •
Soft Flow Strategy. Instead of get the hard min-max flow, we use *soft-min* and *soft-max* operations using the log-sum-exp trick. This approach provides a smoother approximation of flow values, which can be especially useful in our gradients-based optimization. The soft-min/max is computed as below, with $\tau$ as a temperature
$\displaystyle\text{softmax}(e_{1},e_{2},\dots;\tau)=\tau\log\Big{(}\sum_{j}
\exp{(\frac{e_{j}}{\tau})}\Big{)};$
(6)
$\displaystyle\text{softmin}(e_{1},e_{2},\dots;\tau)=-\text{softmax}(-e_{1},-e_
{2},\dots;\tau),$
(7)

Vectorized Path Flow Computation. While depth-first and breadth-first searches can identify all paths for above min-max optimization, these methods are slow and cannot be parallelized.
Instead, we define a special operation called *min-max multiplication* on the capacity matrix to calculate the maximum flow for each path in a vectorized manner.

Definition 2 (Min-max Multiplication). Given two matrices $A\in\mathbb{R}^{m\times k}$ and $B\in\mathbb{R}^{k\times n}$, min-max multiplication $C=A\odot B\in\mathbb{R}^{m\times n}$ is defined where each element $C_{i,j}=\max_{r}(\min(A_{i,r},B_{r,j}))$.

This operation computes the minimum value across all $r$ for the $i$-th row of $A$ and the $j$-th column of $B$, and $\max_{r}$ selects the maximum of these minimum values for each $C_{i,j}$.
We call it a *multiplication* because it resembles matrix multiplication but replaces element-wise multiplication with a minimum operation and summation with maximization.

A very good property is that, the min-max multiplication of capacity matrix $\bm{W}^{k}=\bm{W}^{k-1}\odot\bm{W}$ can be interpreted as the max path flow for all $k$-hop paths.

Proposition 1 (Max Path Flow using Min-max Multiplication)(^4^44Proof in Appendix B). For min-max power of capacity $\bm{W}^{k}=\bm{W}^{k-1}\odot\bm{W}$, element $\bm{W}^{k}_{i,j}$ equals the max path flow for all $k$-hop path from $v_{i}$ to $v_{j}$.

For attention graph that current layer’s node is only connect to the next layer, all path from text token to output video token has exactly the length of $l$. In this way, what we do is just to extract the attention graph $G$, do $l$ times Min-max Multiplication on its flow matrix, and we consider the value as a approximation of ST-Flow. A tine complexity analysis is prepared in Appendix F.

In this way, we get all pieces to build Vico. We first compute attribution using the approximated ST-Flow, then using Eq 4 to update the latent to equalize such flow.

## 4 Experiments

In our experiments section, we evaluate Vico through a series of tests. We start by assessing its performance on generating videos from compositional text prompts. Next, we demonstrate ST-Flow accurately attributes token influence through video segmentation and human study. We also conduct an ablation study to validate our key designs. More application results are provide in Appendix D.

### 4.1 Experiment Setup

Baselines. We build our method on several open-sourced video diffusion model, including VideoCrafterv2 [], AnimateDiff [] and Zeroscopev2 (^5^55[https://huggingface.co/cerspense/zeroscope_v2_576w](https://huggingface.co/cerspense/zeroscope_v2_576w)). Since no current video diffusion models specifically focus on generation with compositional conditions, we re-implement several methods designed for text-to-image diffusion models and compare with them. These methods include:

- •
Original Model. We directly ask the original base model to produce video based on prompts.
- •
Token Re-weight. We use the compel (^6^66[https://github.com/damian0815/compel](https://github.com/damian0815/compel)) package to directly up-lift the weight of specific concept token, with a fixed weight of 1.5.
- •
Compositional Diffusion []. This method directly make multiple noise predictions on different text, and sum the noise prediction as the compositional direction for latent update. In our paper, given a prompt, we first split into short phrases. For example “a dog and a cat” is splitted into “a dog” and “a cat”, make individual denoising, and added up.
- •
Attend-and-Excite []. A&E refines the noisy latents to excite cross-attention units to attend to all subject tokens in the text prompt.

**Table 1: Quantitative results for different methods on compositional text-to-video generation.**
| Name | Spatial Relation$\uparrow$ | Multiple Object$\uparrow$ | Motion Composition$\uparrow$ | Overall Consistency$\uparrow$ |
| --- | --- | --- | --- | --- |
| AnimateDiff [] | 24.80% | 33.44% | 33.90% | 27.75% |
| +Compositional Diffusion [] | 19.43% | 7.27% | 23.58% | 24.07% |
| +Attend-and-Excite [] | 20.88% | 31.25% | 34.78% | 28.05% |
| +Token-Reweight | 28.11% | 36.89% | 37.45% | 26.77% |
| +Vico (hard) | 24.22% | 29.95% | 37.23% | 28.85% |
| +Vico (soft) | 31.47% | 37.20% | 37.95% | 28.89% |
| ZeroScopev2 | 59.52% | 52.52% | 45.51% | 25.83% |
| +Compositional Diffusion [] | 31.77% | 8.23% | 33.13% | 23.02% |
| +Token-Reweight | 57.48% | 50.00% | 40.42% | 25.74% |
| +Attend-and-Excite [] | 59.02% | 62.27% | 45.82% | 25.84% |
| +Vico (hard) | 63.60% | 63.34% | 46.32% | 24.89% |
| +Vico (soft) | 62.28% | 69.05% | 45.31% | 26.15% |
| VideoCrafterv2 [] | 35.86% | 40.66% | 43.82% | 28.06% |
| +Compositional Diffusion [] | 23.61% | 10.59% | 35.49% | 24.49% |
| +Token-Reweight | 46.08% | 49.16% | 44.33% | 28.29% |
| +Attend-and-Excite [] | 48.11% | 66.62% | 43.48% | 28.33% |
| +Vico (hard) | 49.85% | 67.84% | 44.46% | 28.41% |
| +Vico (soft) | 50.40% | 73.55% | 44.98% | 28.52% |

Evaluation and Metrics. We evaluate compositional generation using VBench []. Specifically, we focus on evaluating compositional quality in terms of *Spatial Relation*, *Multiple Object Composition*. For both metrics, the model processes text containing multiple concepts, generates a video. Then a caption model verifies the accuracy of the concept representations within the generated video.

Additionally, we design a new metric, *Motion Composition*. This metric evaluates the generated video based on the presence and accuracy of multiple objects performing different motions. We collect 70 prompts of the form "obj1 is motion1 and obj2 is motion2". Using GRiT [], we generate dense captions on video for each object and verify if each (object, motion) pair appears in the captions. The score is computed as $\frac{\sum_{1,2}(\mathbb{I}(\texttt{obj}_{i})+\mathbb{I}(\texttt{obj}_{i},
\texttt{motion}_{i}))}{4}$. Here, $\mathbb{I}(x)$ is an indicator function that returns 1 if $x$ is present in the generated captions, and 0 otherwise.

The overall video quality is measured using ViCLIP [] to compute a score based on text and video alignment, denoted as *Overall Consistency*.

Implementation Details. We use the implementations on diffusers for video generation. All videos are generated by a A6000 GPU. We sample videos from Zeroscopev2 and VideoCrafterv2 using 50-step DPM-Solver++ []. AnimateDiff is sampled with 50-step DDIM []. We optimize the latent at each sampling steps, and update the latent with Adam [] optimizer at the learning rate of $1e-5$. We test both the soft and hard-min/max versions of Vico, setting the temperature $\tau=0.01$ for the soft version. The NLTK package identify all nouns and verbs for equalization.

### 4.2 Compositional Video Generation

Quantitative Results. In Table 1, we present the scores achieved by Vico compared to other methods across various base models on compositional text-to-video generation. Vico consistently surpasses all baselines on every metric. Notably, our ST-flow based method surpasses cross-attention based techniques like Attend&Excite, thanks to its ability to incorporating influences across full attention graph. Additionally, the soft min-max version of Vico generally achieves better fidelity than the hard version, as it is better suited for gradient optimization.

Specifically, Vico demonstrates its most significant improvements in multi-subject generation tasks. For instance, on VideoCrafterv2, it shows a remarkable increase, improving scores from $40.66\%\to 73.55\%$. This suggests that our attention mechanism in T2V is more adept at managing object arrangement. In contrast, compositional diffusion models [] often fail, as they assume conditions to be independent, which is problematic for realistic condition compositions.

Qualitative Results. We compare the videos generated by different methods in Figure 4. Attend&Excite receive slightly improvements, but still mixes semantics of different subject. For example, on the “a dog and a horse” example (Top Left), both Attend&Excite and the baseline incorrectly combine a dog’s face with a horse’s body. Vico addresses this issue by ensuring each token contributes equally, effectively separating their relationships.

Additionally, cross-attention often leads to temporal inconsistencies in the modified videos. For instance, in the “spider panda” case (Bottom Left), Attend&Excite initially displays a Spider-Man logo but it disappears abruptly in subsequent frames. In contrast, Vico captures dynamics across both spatial and temporal attention, leading to better results. More results is in Appendix D and E.

Figure: Figure 4: Qualitative comparison of the videos generated by VideoCrafterv2 baseline, Attribute&Excite and our Vico with compositional textual descriptions.
Refer to caption: x4.png

### 4.3 Attribution on Video Diffusion Model

In this section, we aim to demonstrate that our ST-Flow (hard) provides a more accurate measure of token contribution compared to other attention-based indicators.

Objective Evaluation: Zero-shot Video Segmentation. We test several attribution methods using the VideoCrafterv2 model for zero-shot video segmentation on the Ref-DAVIS2017 [] dataset. To create these maps, we first performed a 25-step DDIM inversion [] to extract noise patterns, followed by sampling to generate the attribution maps. We specifically use maps from from *end of text* ([EOT]) token [] for segmentation. We used the mean value of the map as a threshold for binary segmentation. We compare with cross-attention [] and Attention Rollout []. The more accurate the segmentation is, the attribution is more reasonable for human.

Results are presented in Table 5. Our method outperform others, providing the highest segmentation metrics in zero-shot setting. As visualized in Figure 5, cross-attention maps show inconsistent highlighting and flickering. Attention Rollout also conciders the full attention graph, but overly smoothed weights, resulting in less precise object focus.

Subjective Evaluation: User Study. Besides, segmentation-based validation, we conducted a subjective user study to evaluate the quality of attribution maps generated by various methods. 20 participants rated maps from three different approaches across ten video clips. The evaluation focused on *Temporal Consistency*, assessing the presence of flickering, and *Reasonability*, determining alignment with human interpretations. Ratings ranged from 1 to 5, with 5 as the highest. As summarized in Table 5, Our ST-Flow outperforms other attention-based explanation, achieving the highest scores in both Temporal Consistency (4.19) and Reasonability (3.78).

**Table 2: User study on attribution method.**
| Attribution Method | Temporal Consistency$\uparrow$ | Reasonability$\uparrow$ |
| --- | --- | --- |
| Cross-Attention [] | 2.56 | 2.84 |
| Attention Rollout [] | 3.88 | 3.45 |
| ST-Flow (Ours) | 4.19 | 3.78 |

### 4.4 Ablation Study

In this section, we ablate our two key designs: the loss function and the proposed ST-Flow. We do experiments on VideoCrafterv2 and compare the performance.

Loss Function. We modify the loss function used, from using the “min” as a fairness indicator (as described in Sec 3.1) to a variance loss, defined as $\mathcal{L}_{\text{fair}}=-\sum_{i}(A_{i}-\bar{A})^{2}$. It minimizes the differences between each $A_{i}$ and the average attribution value $\bar{A}$, making it fair. The results are shown in Table 5, row 3 and 4. We notice while the variance loss ensures uniformity across all tokens, it overly restricts them, often degrading video quality. Conversely, our min-loss focuses on the least represented token, enhancing object composition accuracy without significantly affecting overall quality.

ST-Flow $v.s.$ Cross-Attention. A major contribution of our work is the development of ST-Flow and its efficient computation. We compare it against a variant using cross-attention as attribution. In this method, cross-attention maps are extracted and a mean score is computed for each token as $A_{i}$. As demostrated in Table 5, row 2 and 4, using ST-Flow (soft) largely outperforms cross-attention. We also provide the running speed analysis in Appendix F, confirming the efficiency of our approach.

## 5 Conclusion

In this paper, we present Vico, a framework designed for compositional video generation. Vico starts by analyzing how input tokens influence the generated video. It then adjusts the model to ensure that no single concept dominates. To implement Vico practically, we calculate each text token’s contribution to the video token using max flow. This computation is made feasible by approximating the subgraph flow with a vectorized implementation. We have applied our method across various diffusion-based video models, which has enhanced both the visual fidelity and semantic accuracy of the generated videos.

## Appendix A Proof of Theorem 1: Sub-graph Flow

In a network $G=(V,E)$ with a capacity function $c:E\rightarrow\mathbb{R}^{+}$, and a subgraph $g$ of $G$, the maximum flow $f_{g}$ in $g$ is less than or equal to the maximum flow $f_{G}$ in $G$.

### Proof

- 1.
Definition of a Subgraph: A subgraph $g$ of $G$ can be defined as $g=(V^{\prime},E^{\prime})$ where $V^{\prime}\subseteq V$ and $E^{\prime}\subseteq E$. All capacities in $g$ are inherited from $G$, i.e., $c^{\prime}(e)=c(e)$ for all $e\in E^{\prime}$.
- 2.
Flow Conservation: Both $G$ and $g$ must satisfy the flow conservation law at all intermediate nodes. That is, the sum of the flow entering any node must equal the sum of the flow exiting that node, except for the source (where flow is generated) and the sink (where flow is absorbed).
- 3.
Reduced Set of Paths: Since $E^{\prime}\subseteq E$, every path through $g$ is also a path through $G$, but not every path through $G$ is necessarily a path through $g$. This reduction in the number of paths (or edges) in $g$ implies that some routes available for flow in $G$ are not available in $g$.
- 4.
Capacity Limitations: For any edge $e$ in $E^{\prime}$, the capacity in $g$ (i.e., $c^{\prime}(e)$) equals the capacity in $G$ (i.e., $c(e)$). Therefore, no edge in $g$ can support more flow than it can in $G$. Additionally, since some edges might be missing in $g$, the overall capacity of pathways from the source to the sink might be less in $g$ than in $G$.
- 5.
Maximum Flow Reduction: Given the reduction in paths and capacities, any flow that is feasible in $g$ is also feasible in $G$, but not vice versa. Hence, the maximum flow $f_{g}$ that can be pushed from the source to the sink in $g$ must be less than or equal to the maximum flow $f_{G}$ that can be pushed in $G$.

Conclusion: From these points, it follows directly that the maximum flow in a subgraph $g$ cannot exceed the maximum flow in the original graph $G$. This proves that $f_{g}\leq f_{G}$.

## Appendix B Proof of Proposition 1: Max Path Flow using Min-max Multiplication

Definitions and Proposition:
Let $\mathbf{W}$ be a capacity matrix of a graph where $\mathbf{W}_{i,j}$ is the capacity of the edge from vertex $i$ to vertex $j$. If there is no edge between $i$ and $j$, $\mathbf{W}_{i,j}=0$ or some representation of non-connectivity. A $k$-hop path between two vertices $i$ and $j$ is a path that uses exactly $k$ edges.

Proposition: The $k$-th min-max power of $\mathbf{W}$, denoted $\mathbf{W}^{k}$, calculated as $\mathbf{W}^{k}=\mathbf{W}^{k-1}\odot\mathbf{W}$, has elements $\mathbf{W}^{k}_{i,j}$ that represent the maximum flow possible on any $k$-hop path from vertex $i$ to $j$.

Min-max Multiplication:
Given matrices $\mathbf{A}$ and $\mathbf{B}$, $\mathbf{C}=\mathbf{A}\odot\mathbf{B}$ is defined such that:

$$ $\mathbf{C}_{i,j}=\max_{r}(\min(\mathbf{A}_{i,r},\mathbf{B}_{r,j}))$ $$

Proof by Induction:

Base Case ($k=1$):

- •
Claim: $\mathbf{W}^{1}_{i,j}$ represents the capacity of the edge from $i$ to $j$, which is the maximum flow on a 1-hop path.
- •
Proof: By definition, $\mathbf{W}^{1}=\mathbf{W}$, and $\mathbf{W}^{1}_{i,j}=\mathbf{W}_{i,j}$, which directly corresponds to the edge capacity between $i$ and $j$. Hence, the base case holds.

Inductive Step:

- •
Assumption: Assume that for $k-1$, $\mathbf{W}^{k-1}_{i,j}$ correctly represents the maximum flow on any $k-1$-hop path from $i$ to $j$.
- •
To Prove: $\mathbf{W}^{k}_{i,j}$ represents the maximum flow on any $k$-hop path from $i$ to $j$.

Proof:
From the definition of min-max multiplication,

$$ $\mathbf{W}^{k}_{i,j}=\max_{r}(\min(\mathbf{W}^{k-1}_{i,r},\mathbf{W}_{r,j}))$ $$

- •
$\mathbf{W}^{k-1}_{i,r}$ is the maximum flow from $i$ to $r$ using $k-1$ hops.
- •
$\mathbf{W}_{r,j}$ is the capacity of the edge from $r$ to $j$ (1-hop).

Interpretation: $\min(\mathbf{W}^{k-1}_{i,r},\mathbf{W}_{r,j})$ finds the bottleneck flow for the path from $i$ to $j$ through $r$ using $k$ hops. The minimum function ensures the path’s flow is constrained by its weakest segment.

Maximization Step: $\max_{r}$ over all possible intermediate vertices $r$ selects the path with the highest bottleneck value, thus ensuring the selected path is the most capable among all possible $k$-hop paths.

Conclusion:
The inductive step confirms that the flow represented by $\mathbf{W}^{k}_{i,j}$ is indeed the maximum possible flow across any $k$-hop path from $i$ to $j$. Hence, by induction, the proposition holds for all $k$.

## Appendix C Related Work

Video Diffusion Models. Video diffusion models generate video frames by gradually denoising a noisy latent space []. One of the main challenges with these models is their high computational complexity. Typically, the denoising process is performed in the latent space [, , ]. The architectural commonly adopt either a 3D-UNet [, , , , ] or diffusion transformer [, , ]. To enhance computational efficiency, these architectures often employ separate self-attention mechanisms for managing spatial and temporal tokens. Conventionally, training these models involves fine-tuning an image-based model for video data [, , ]. This process includes adding a temporal module while striving to preserve the original visual quality.

Despite their ability to generate photorealistic videos, these models frequently struggle with understanding the complex interactions between elements in a scene. This shortcoming can result in the generation of nonsensical videos when responding to complex prompts.

Compositional Generation. Current generative models often face challenges in creating data from a combination of conditions, with most developments primarily in the image domain. Energy-based models [, , ], for example, are mathematically inclined to be compositionally friendly, yet they require the conditions to be independent.
In practice, many image-based methods utilize cross-attention to effectively manage the composition of concepts [, , , ]. However, when it comes to video, compositional generation introduces additional complexities. Some video-focused approaches concentrate specific form of composition, including object-motion composition [], subject-composition [], utilize explicit graphs to control content elements [], or integrate multi-modal conditions []. Despite these efforts, a generic solution for accurately generating videos from text descriptions involving multiple concepts is still lacking. We present the first solution for compositional video generation using complex text prompts, an area that remains largely underexplored.

Attribution Methods. Attribution methods clarify how specific input features influence a model’s decisions. gradient-based methods [, , ] identify influential image regions by back-propagating gradients to the input. Attention-based methods [, ] that utilize attention scores to emphasize important inputs. Ablation methods[, ] modify data parts to assess their impact. Shapley values [] distribute the contribution of each feature based on cooperative game theory. In our paper, we extend existing techniques of attention flow to video diffusion models. We develop an efficient approximation to solve the max-flow problem. This improvement helps us more accurately identify and balance the impact of each textual elements on synthesized video.

## Appendix D Compositional Video Editing

Our system, Vico, can be integrated into video editing workflows to accommodate text prompts that describe a composition of concepts.

Setup. We begin by performing a 50-step DDIM inversion on the input video. Following this, we generate a new video based on the given prompt.

Results. An example of this process is illustrated in Figure 5. The original video demonstrates a strong bias towards a single presented object, making editing with a composition of concepts challenging. However, by applying Vico, we successfully enhance the video to accurately represent the intended compositional concepts.

Figure: Figure 5: Video edit results with compositional prompts.
Refer to caption: x6.png

## Appendix E More Visualizations

Here we provide more example for compositional T2V in Figure 6

Figure: Figure 6: Video visualization for compositional video generation
Refer to caption: x7.png

## Appendix F Speed Analysis

Attribution Speed. In this section, we assess the running speed of our ST-flow. To assess its computational efficiency, we compare ST-flow with cross-attention and Attention Rollout [] computation, by reporting the theoretical complexity and empirical running time. We assume we have 1 cross attention map of $mxn$ and $L$ self-attention map of $n\times n$, and demonstrated the theoretical results. Specifically, we measure the average running time required for each diffusion model inference, focusing solely on the time taken for attribution computation, excluding the overall model inference time. We use the VideoCrafterv2 as the base model.

As detailed in Table 6, the cross-attention computation is fast, as it processes only a single layer. Both Attention Rollout and our approximated ST-Flow involve matrix multiplications and consequently share a similar time complexity. However, our ST-Flow approximation benefits from the relatively faster speed of element-wise min-max operations compared to the floating-point multiplications used in Attention Rollout, leading to slightly quicker execution times.

In contrast, the exact ST-Flow method is much slower. This is because it requires independently estimating the flow for each sink-source pair, a process that takes considerable time.

**Table 6: Speed comparison for attribution method.**
| Method | Complexity | sec/inference |
| --- | --- | --- |
| Cross-Attn. | O(1) | 0.002s |
| Attention Rollout | $O(Lmn^{2})$ | 0.042s |
| Exact-ST-Flow | $O(L^{3}mn^{4})$ | 8s |
| ST-Flow (soft) | $O(Lmn^{2})$ | 0.037s |

Diffusion Inference Speed. Our Vico framework includes a iterative optimization process alongside with the denoising. As expected, it should results in longer inference time. We evaluated this using a 50-step DPM denoising process on the VideoCrafterv2 model, at a resolution of $512\times 320$ for 16 frames, on a single A6000 GPU.

The results, shown in Table 7, reveal that the baseline VideoCrafterv2 completed in 23 seconds. Adding the Attend&Excite increased the duration to 48 seconds. In comparison, our Vico framework finished in a comparable time of 50 seconds. Despite its additional complexity, Vico’s efficient design keeps the inference time within a reasonable range.

**Table 7: Text-to video model inference time comparison.**
| Method | Time |
| --- | --- |
| VideoCrafterv2 | 23s |
| + Attend&Excite | 48s |
| + Vico (soft&hard) | 45s |

## Appendix G Implementation details of Vico

ST-Flow Computation. To compute the ST-Flow, we begin by extracting attention weights from all layers. These weights are averaged across all heads and then upscaled to the image size using bicubic interpolation. Due to the block-wise sparse pattern of the connections, min-max matrix multiplication is applied to the capacity matrix for connected layers. Furthermore, given that cross-attention layers include skip connections from previous layers, we divide the network into multiple groups. Within each group, min-max matrix Multiplication is performed. Finally, we aggregate the scores across all groups to obtain the results. The pseudocode for the min-max multiplication is in Algorithm 1.

Figure: Algorithm 1 Batched Min-Max Matrix Multiplication

Latent Step. During the first half of the sampling process, we update the latent variables. We establish a loss threshold of 0.2; once this threshold is reached, no further updates are made.

## Appendix H Limitations

Although Vico effectively allocates attribution across different tokens, it does not explicitly bind attributes to subjects. Moreover, there is a critical balance to maintain between latent updates and semantic coherence. Excessive updating can lead to the generation of nonsensical videos.

## Appendix I Broader Applications

Technically, the computation of attention flow proposed in our system is versatile and can be efficiently applied to a variety of other applications like erase certain concept in diffusion models. Additionally, the principle of fairly distributing the contribution of different input parts can be extended to other domains, such as language modeling.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: n/a
soundness: 3
strengths: 1. The proposed method is innovative and can be integrated with existing video generation techniques. The experiments demonstrate the effectiveness of applying the "Vico" method on current models such as AnimaDiff, ZeroScore V2, and VideoCrafter V2, showing notable improvements in results.

2. The paper is well-articulated, featuring thorough theoretical analysis and proof.
summary: This paper introduces a novel approach for compositional video generation. The proposed method begins by analyzing the impact of input tokens on the video output, ensuring that no single concept predominates the generated content. The core concept involves computing each text token's contribution to the video generation process through maximum flow. The paper introduces an efficient computational technique by approximating the subgraph flow with a vectorized implementation.Finally, the method has been tested across various diffusion-based video models, and the experimental results confirm its effectiveness in enhancing both visual fidelity and semantic accuracy.
weaknesses: 1. The user study (Table 2) is limited to only ten video clips, which is insufficient to conclusively prove the effectiveness of the method.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: 1-  In line 247, the capacity matrix W, what are each row and column? Why does the first row have the `t` index while other rows don't?
2- Lines 241, and 243. What is the difference between e_(i,j), i=j and e_(i,i) difference?
3- Section 3.1: At first glance, it is not clear what is x_t (definition) and how A_i is extracted out of it. 
4- What is token-reweight in Table 1? 
5- There are some valid categories of issues in Video Generation shown in Figure 1. My question is if the lack of fair attribution and attention is the only problem that causes these issues. If not, what are the other factors?
soundness: 3
strengths: Capturing compositional relationships in the final generated output is a very important problem and, to the best of my knowledge, is one of the biggest issues with current SOA in GenAI.
This paper correctly identifies one of the main issues throughout the attention mechanism and tries to improve the contribution of different tokens in attention layers as a test-time optimization. The authors model the information flow of the generative model as a graph, which is a smart and (semi-)novel strategy (for the videos) in my opinion.
summary: This manuscript tackles the problem of video generation, focusing on improving the compositional and complex interactions in the final output. The proposed approach utilizes a flow equalization formulation to ensure that different tokens of input get a fair amount of attention throughout the self-attention layers.
weaknesses: The biggest weakness of the solution is the readability of this paper. It was very hard for me to read through the text and jump from text to mathematical notations and back. I will ask for a few clarifications in the questions block.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. My primary question is,  do the paper's assumptions hold true for modern 3D attention models, such as the open-source CogVideoX. Can the proposed method balance token contributions effectively when applied to an attention graph in such models?
2. In the case of 3D attention, how much additional computational overhead does the proposed approach introduce? Cross-attention calculations are relatively lightweight, but if full attention is employed, as in MMDiT, would the method’s computations lead to significant cost increases?
3. The paper mainly showcases relatively short prompts. I would like to ask if the proposed method can handle modern, highly descriptive prompts that may reach up to 100 words. Would the method remain feasible for inference, and what would be the associated computational cost in such cases?
4. As a fair comparison, could the paper utilize T2V CompBench[1], a benchmark specifically designed to evaluate compositional video generation, to better assess and compare the performance of the proposed model against other existing models? I wonder how well this method performs under a more comprehensive benchmark.

[1] Sun, Kaiyue, et al. "T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation." arXiv preprint arXiv:2407.14505 (2024).
soundness: 3
strengths: 1. This paper presents a highly innovative solution to the problem, utilizing traditional max flow to address token-level response balancing, thereby achieving effective compositional generation.
2. I appreciate that extensive effort has been put into designing feasible experiments. The authors introduce practical techniques such as subgraph, soft min, and vectorized flow strategies, which significantly enhance inference speed.
3. The experiments are thorough and well-executed, including comprehensive comparative studies and a detailed user study, which provides strong validation for the proposed approach.
4. The visual quality of the generated videos is satisfying, despite the limited number of examples presented.
summary: This paper introduces Vico, a novel framework specifically designed for compositional video generation. Vico employs a maximum flow approach to ensure fair contributions from each input token, integrating sub-graphs, a soft flow strategy, and vectorized path flow computation for efficient inference. Extensive experiments demonstrate that Vico significantly surpasses existing baselines in compositional generation tasks.
weaknesses: 1. The primary concern is the lack of comparisons or discussions involving recent text-to-video (T2V) methods. The baseline model, VideoCrafter2, was released over a year ago. To convincingly demonstrate the relevance of the compositional generation problem, the paper should ideally compare against more advanced, recent baselines like OpenSora[1],  CogVideoX[2], or more.
2. Additionally, the paper lacks comparisons to existing compositional video generation models. For instance, methods like LVD[3], which uses LLMs for prompt decomposition with gradient-based layout optimization during inference, VideoTetris[4], which employs spatiotemporal diffusion for multi-object generation, and VideoDirectorGPT[5], which combines an LLM director with spatial convolutions for layout learning, are not discussed. Including comparative studies with these related works would strengthen the evaluation and clarify the advantages of the proposed approach.
3. The paper lacks a theoretical justification for the assumption that each token should have an equal impact on the generated result. In typical prompts, not all tokens necessarily require equal influence; for instance, non-descriptive tokens or function words might logically play a reduced role in the attention process. Given the strength of this assumption, a more rigorous rationale or proof would help validate its applicability.
4. Another strong assumption is made regarding cross-attention in T2V models. The paper’s approach relies on cross-attention being embedded uniformly across all layers and frames, yet many contemporary T2V models, such as CogVideoX[2], Mochi-1[6], and commercial models like Kling[7], utilize 3D attention mechanisms similar to MMDiT. This raises concerns about the applicability of the paper’s assumptions to modern architectures, which may limit its generalizability.
5. The paper contains a few typographical errors that affect readability. For example, in the introduction, the prompt "a bird looks like a cat" is used as an example, but the actual effect of this prompt is not demonstrated.

[1] Open-Sora: Democratizing Efficient Video Production for All, https://github.com/hpcaitech/Open-Sora

[2] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[3] Lian, Long, et al. "Llm-grounded video diffusion models." arXiv preprint arXiv:2309.17444 (2023).

[4] Tian, Ye, et al. "VideoTetris: Towards Compositional Text-to-Video Generation." arXiv preprint arXiv:2406.04277 (2024).

[5] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." arXiv preprint arXiv:2309.15091 (2023).

[6] Mochi-1: https://www.genmo.ai/blog

[7] Kling: https://www.klingai.com/

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to Weaknesses for more details.
soundness: 3
strengths: 1.	A new method named Vico for compositional video generation is proposed.
2.	The proposed Vico leverage max-flow to ensure that no single concept dominates.
3.	Extensive results verify the effectiveness of the proposed method in generation high-quality videos.
summary: This paper presented a new method named Vico, a framework designed for compositional video generation. Different from existing methods that may not reflect the intended composition of elements, the proposed Vico tries to adjust the model to ensure that no single concept dominates. Specifically, Vico calculate the contribution of each text token using max-flow and leverage a sub-graph flow to propagate information. The proposed method can be implemented by inserting into diffusion-based methods. Extensive experimental results verify the effectiveness of the proposed method.
weaknesses: 1.	This paper argues that the proposed method explicitly constrains the contribution of each token to be the equal, but in fact there are some words with no contribution or words with outstanding contribution in a sentence, that is, the importance of each word should be different. For example, "On a certain day, a boy is running", where "On a certain day" is not informative, only the following "the boy is running" is informative. The authors should discuss the problem.
2.	The proposed method needs to be optimized at test time, and the resource consumption of optimization such as time and computational overhead should be discussed.
3.	This paper said “While the compositional text-to-image sythesis Liu et al. (2022); Chefer et al. (2023); Kumari et al. (2023); Feng et al. (2023); Huang et al. (2023) has been more studied, the challenge of compositional video generation has received less attention.” My question is that whether these image methods can be extended to compositional video generation task? If not, the authors should give a explanation. If yes,  please give a comparison with them.
4.	Can Vico be used in compositional motion scenarios? Such as "In a room, a cat is running from left to right, a dog is running from right to left", this complex movement scene is very interesting.

--------------------------------
Post review:
Thank the authors for the responses. Given ICLR's high standards, I decided to keep the score at 5. The motivation of this paper is good, but in the current version, the different processing of prompt words may ignore some cases, such as when some words other than verbs and nouns have a certain amount of information. And the proposed method requires test-time optimization, which greatly increases the inference cost, considering that the model is getting larger and larger now.

Meta-review: This paper introduces Vico, an inference-time method designed to enhance compositional video generation. Vico ensures equal influence for each textual token (verbs and nouns only) on the final video output through test-time optimization, dynamically assessing and rebalancing token impacts at each diffusion step. The reviewers recognize the significance of the problem, the (semi-)novel approach, and the extensive results that demonstrate the method's effectiveness.

However, the reviewers also highlight several weaknesses. While many concerns were addressed following the rebuttal, the remaining issues are:

* Test-time optimization overhead: Two reviewers raised concerns about the significant increase in computation time. For instance, Vico requires 200 seconds for CogVideoX-2B, more than doubling the 90 seconds needed by the baseline CogVideoX. The authors responded by outlining plans to integrate the Vico framework into the training phase to mitigate this overhead.

* Balancing strategy: Reviewer kAsb questioned the approach of equalizing only nouns and verbs, suggesting it may oversimplify the process. 

Finally, this paper receives mixed evaluations: accept (low confidence), marginally above the acceptance threshold, and marginally below the acceptance threshold. The opposing Reviewer kAsb remains unwilling to adjust their score after reading the rebuttal, making it a borderline case.

The AC finds Reviewer kAsb's arguments persuasive and agrees with their concerns. While this work shows potential for improvement, the significant overhead introduced by test-time optimization diminishes its practical significance. Regarding the technical design, the AC agrees with Reviewer kAsb's critique that balancing only nouns and verbs may overlook other important word types. This limitation could hinder the model's ability to effectively handle long and complex prompts. This limitation is particularly relevant as recent video generation models often use rephrasers to expand short user prompts into complex structures, incorporating multiple nouns, verbs, and other word types (e.g., attributes, geometric details, relationships) that convey subtle yet essential information.

In light of the above, the AC recommends that accepting this submission may be premature until the concerns are fully resolved.

Decision: Reject