VideoTetris: Towards Compositional Text-to-Video Generation
Authors: Ye Tian
Ling Yang
Haotian Yang
Yuan Gao
Yufan Deng
Xintao Wang
Zhaochen Yu
Xin Tao
Pengfei Wan
Di ZHANG
Bin CUI
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Video Generation
Video Diffusion Models
Primary area: generative_models

Abstract: Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose a new dynamic-aware data processing pipeline and a consistency regularization method to enhance the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

VideoTetriss Towards Compositional Text-to-Vider Generation
Ye Tian
Ling Yang Haotian Yang² Yuan Gao? Yufan Deng' Jingmin Chen'
Xintao Wang? Zhaochen Yu' Xin Tao?
Pengfei Wan
Di Zhang? Bin Cui!
Peking University KKuaishou Technology
Project: silcotetris.iihuub..
Code: http:///iiuubb.oooo/aggnnggg TeViis
Abstract
Diffusion models have demonstrated great success in text-to-video (T2V) gener- ation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual mantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, processing to enhance the training data regarding motion dynamics and prompt understanding, equipped with new reference frame atlention mechanism im- prove the consistency of auto-regressive video generation. Extensive experiments demonstrate that Video'Tetris achieves impressive qualitative and quantitative results in compositional T2V generation.
propose enhanced video data pre-
Introduction
With the significant development of diffusion models [1-3] recently, advanced text-to-video models [4-7] have emerged and demonstrated impressive results. However, these models often struggle with generating complex scenes following compositional prompts, such as "A man on the left walking his dog on the right", which requires the model to compose various objects spatially and temporally. Moreover, with growing interest in generating long videos, existing methods [8-10] try to explore multi pullipprrmpt long video generation, which is typically limited simple single-object scene changes. These methods fail to manage scenarios where the number of objects changes dynamically, often resulting in bizarre transformations that do not accurately follow the input text.
To overcome these challenges, we introduce Video'Tetris, a novel and effective diffusion- based framework enable compositional text-to-video generation. Firstly, we define compositional video generation as encompassing two primary tasks: (i) Video Generation with Compositional Prompts, which involves integrating objects with various attributes and relationships into complex and co- herent video; and (ii) Long Video Generation with Progressive Compositional Prompts, where 'progressive' refers to the continuous changes in the position, quantity, and presence of objects with different attributes and relationships. Then, we introduce a novel Spatio- Tentio-eempora Compositional Diffusion, which manipulates the cross-attention value of denoising network temporally and spa- tially, synthesizing videos that faithfully follow complex or progressive instructions. Subsequently, to enhance the ability of long video generation models to grasp complex semantics and generate intricate scenes encompassing various attributes and relationships, we propose an Enhanced Video
"Contributed equally. Contact: yangling081 @ 163.com 'Corresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Video Generation with Compositional Prompts
AnimsteDiff
Gen-2
Pika
VidooCin
LVD
Ours
heroic robot on the left magical girl
cute brown dog and sleepy cat
on the right are saving day.
are napping in sun.
(b) Long Video Generation with Progressive Compositional Prompts
FreeNoixe
Streaming T2V
Ours
A handsome young man is drinking coffee on wooden table.
.......>>> (transitions to) A handsome young man and beautiful young lady on his left are drinking coffee on a wooden table.
Free FraeNNise Noise
Streaming T2V
Ours
A cute brown squirrel in Antarctica, on pile of hazeinuts cinematic.
-....>>> (transitions to)
A cute brown squirrel and cute white squirre! in Antarctica, on pile of hazelnuts cinematic
Figure 1: (a): Comparison in Video Generation with Compositoinal Prompts. (b): Comparision in Long Video Generation with Progressive Compositional Prompts. Video'eetris demonstrates superior performance, exhibiting precise adherence position information, diverse attributes, interaction, consistent seene transitions, and high motion dynamics in compositional video generation.

---

Data Preprocessing pipeline, augmenting the training data with enhanced motion dynamics and prompt semantics, enabling the model to perform more effectively long video generation with progressive compositional generation. Finally, we propose a consistency regularization method, namely Reference Frame Attention, that maintains content consistency space with latent noise while being capable of accepting arbitrary image inputs, ensuring the consis- tency of multiple objects across different frames and positions. fig. I(a) showcases our VideoTetris's superior performance in compositional short video generation. We accurately compose two distinct objects with their own attributes while maintaining their respective "left" and "right" positions and ensuring natural interaction between multiple objects. As for long video generation comparisons in fig. 1(b), FreeNoise [10] either depicts characters appearing abruptly and inexplicably transforming a man into woman or depicts a squirrel transforming from hazelnut. StreamingT2V [11] fails to incorporate information about new characters altogether, ignoring quantity information and exhibits severe color distortion in later stages. In contrast, our VideoTetrii excels in generating long videos with progressive compositional prompts, seamlessly integrating new characters into the video scenes while maintaining consistent and accurate positional and quantity information. Notably, in generating long videos of the same length, FreeNoise produces only minor variations within the same scene, whereas VideoTetris demonstrates significantly higher motion dynamics, resulting in outputs that more closely resemble long narrative videos.
coherent representation
Our contributions are summarized follows: 1). We introduce Temporal Compositional Diffusion method for handling scenes with multiple objects and following progressive complex prompts. 2). We develop Enhanced Video Data Preprocessing pipeline to enhance neg--regressive long video generation through motion dynamics and prompt semantics 3). We propose a consistency regularization method with Reference Frame Attention that maintains content coherence in composi- tional video generation. 4). Extensive experiments show that Vetro can generate state-of-the-ar quality compositional videos, as well as produce high-quality long videos that align with progressive compositional prompts while maintaining the best consistency.
2 Related Work
Text-to- Video Diffusion Models The field of text-to- video generation has seen significant advance- ments with the progress of diffusion models [1, 12, 13] and the development of large-scale video-text paired datasets [14, 15]. Early works such as LVDM [16] and ModelScope [7], adapted 2D image diffusion models by flattening the U-Net architecture to 3D U-Net and training on extensive video datasets. Subsequently, methods like AnimatedDiff [5] have incorporated temporal attention modules into the existing 2D latent diffusion models, preserving the established efficacy of T2I models. More recently, several transformers-bart based diffusion methods [17, 18, 6] have enabled large-scale joint training of videos and images, leading to significant improvements in generation quality.
Long Video Generation Most existing text-to-video diffusion models have been trained on fixed- size video datasets due to the increased computational complexity and resource constraints. Conse- quently, these models are often limited to generating a relatively small number of frames, leading to significant degradation in quality when tasked with generating longer videos. Several advancements [19, 8, 10, 20] have sought to overcome this limitation through various strategies. More recently, Vlogger [9] and SparseCtrl [21] employ masked diffusion model for conditional frame input. Although these masked diffusion approaches facilitate longer video generation, they often encounter model inconsistencies and quality degradation due domain shifts in input. Streaming T2V [11] pro- poses a new paradigm, utilizing a ControlNet [22]-like conditioning scheme to enable auto-regressive video generation. However, due to the low quality of the training data, the final video outputs often exhibit inconsistent and low-quality artifacts.
Compositional Video Generation While current video generation models can synthesize text- guided videos, they often face challenges in generating videos featuring multiple objects or adhering to multiple complex instructions, which requires the model to compose objects with diverse temporal and spatial relationships. In the realm of text-to-vides diffusion models, exploration of such scenarios remains incomplete. Several text-to-image methods like RPG [23] leverage additional layout or regional information to facilitate more intricate image generation [24-27, 23). Within video diffusion techniques, approaches like LVD [28] and VideoDirectorGPT [29] employ a layout-to- vo-ieeo generator to produce videos based on spatial configurations. However, these based methods often offer
3

---

(Optional) Auto-fegrresivv Long Video Generation
VideoTetris
ControiNet
Branch
Trainable Weights
Frazer Weights
Pretrained Model
Spatto-Temporel Compositional Cross Attention
Reference Attention
VAE
Zt
Text Prompt
K
Reference Frame(s)
ocby dospiin the lyl swinninn ulinpe oevon freely with her jonher the right conals and tartlss under.
03-4 6:440
12446
VAE
2.168
Figure 2: The overall pipeline of Vetro We introduce Opatio-Tempone Compositional module for compositional video generation and Reference Frame Attention for consistency regularization. For longer video generation, a ControlNet [22]-like branch can be adopted for auto-regressive generation.
only rudimentary and suboptimal spatial guidance, struggling particularly with overlapping objects, thereby resulting in videos with unnatural content. In contrast, our method adopts compositional region diffusion approach. By explicitly modeling the spatial positions of objects with cross attention maps, our approach allows the objects naturally integrate and blend during the denoising process, resulting in more realistic and coherent video output.
3 Method
Overview In this section, introduce method VideoTetris for compositional text-to-vider generation. Our goal is to develop an efficient approach that enables text-to-video models to handle scenes with multiple objects and follow sequential complex instructions. We first introduce Spatio- Temporal Compositional Region Diffusion in section 3.1, which allows different objects to naturally integrate and blend during the denoising process in training-free manner. Furthermore, for the task of generating long videos with progressive complex prompts, we construct an auto-regressive model based on the ControlNet [22] architecture and introduced Enhanced Video Data Preprocessing pipeline in section 3.2 to collect a high-quality video-text pair dataset to train our auto-regressive model for enhanced motion dynamics and prompt understanding. Combined with Compositional Region Diffusion, our auto-regressive model can generate long videos with seamless transitions between diverse target scenes. Finally, we propose a consistency regularization with Reference Frame Attention in section 3.3 for better object appearance preserving.
3.1 Spatio-Temporal Compositional Diffusion
Motivation To achieve natural compositional generation, a straightforward approach is to use the layout as a condition to guide the generation process. However, this method presents several challenges: (i) Requiring large-scale training. Given the significant potential for improvement in layout-to-image models, training a layout-to-video model or training temporal convolution and attention layers for layout-to-iman and may struggle keep pace with the latest advancements in text-to-video models. (ii) Layout- based generation models impose significant constraints on object bounding boxes. For long video duration, the need to continuously adjust the positions and sizes of these boxes to maintain coherent video content introduces complex planning process, which adds complexity to the overall method. Therefore, instead of training layout-to-video model, we utilize cross-attention for precise generation [30-35] and propose training-fit free approach that directly adjusts the cross aattenttttiion of different targets [23, 36-39], as is shown in fig. 3. This approach aims to overcome the limitations of layout-based methods and leverage the potential of more flexible and efficient generation techniques.
model would require substantial computational resources
Localizing Subobjects with Prompt Decomposition For given prompt p. we first decompose temporally into contents at different frames: p = {p-, p2,... ,r),, where denotes the total number of frames and p' denotes the given text prompt i-th frame. Subsequently, for the i-th

---

Video Timeline
Denoising
Denoising
Denoising
Text Prompt:
A little dolphin
M
ME
storts exploring an old city under the sea, she first found a green turtle at the bottom, then her huge father comes along to accompany her at the right side.
Spatto-Temporal Composing
Spatio-Temporal
Spatio-Temporal
Composing
Composing
Crass
Crass
Cross
Attention V
Attention
Attention
K
V
V
Text Prompt #1 A little dolphin is exploring an old city under the sea
Text Prompt little dolphin
Text Prompt #
A little dolphin with her huge father and exploring an old city under the sea with a green cute turtle
exploring old city under the sea with green cute turtle
Temporal Decomposing
Compositional Diffusion. For a given story "A little dolphin starts exploring an old city under the sea, she first found green turtle at the bottom, then her huge father comes along to accompany her at the right side.", we first decompose it temporally to Text Prompt #1. #2 and #3, then decompose each of them spatially to compute each sub-region's cross attention maps. Finally, we compose them
Figure 3: Illustration of
to form a natural story.
frame, decompose the original p¹ spatially into different sub-objects: {pb.pí.". pi. P/. } with their corresponding region masks M = {Ma Mi, M. where n denotes the number of different objects. In this way, we decompose prompt list temporally and spatially acquire each sub-object's corresponding region information in the video timeline. We then calculate the cross attention value for the j-th sub-object at i-th frame as follows:
CrossAuth = Softmax QUKII)
Vd VU K= H(P)), V = Wv * h(p))
where Q' represents the query for the latent frame features, W WK, , Wy are linear projections, & denotes the text encoder, and d is the latent projection dimension of the latent frame features.
LLM-based Automatic Spatio-Temport Decomposer (Optional) Alternatively, the spatio- temporal decomposition process can directly utilize a Large Language Model (LLM) to automate tasks, given the robust performance of LLMs in language comprehension, reasoning, summarization and region generation ablilities [23, 28, 27, 26]. We employ the in-context learning (ICL) capability of LLMs and guide the model to use chain-of thouuht (CoT) [40] reasoming. Concretely, we first guide the LLM decompose the story temporally, generating frame-wise prompts, and reception each one of them with LLM for better semantic richness. Then we use another LLM to decompose each prompt spatially into multiple prompts corresponding to different objects, assigning a region mask to each sub-prompt. The specific prompt templates that include task rules (instructions), in-context examples (demonstrations) are detailed in table 4, table 5 and table 6 of appendix A.
Tentoollemmpaa Subobjects Compasition After we decompose the original prompt list tempo- rally and spatially, we then compose them together from spatial temporal. To this end, we first compute the cross-attention value of all sub-objects CrossAttn'rgiion at i-th frame with:
CrossAttt'rgionn
CrossAmt
Subsequently, to ensure a cohesive transition across the boundaries of distinct regions and seamless integration between the background and the entities within each region, employ the weighted sum
5

---

of the CrossAtthregion and the ortrriaa for the original compositional prompt p with
Q'(Ki)T V
CrossAttn original = Softmax(
K = WK p(p'), V Wv * ø(p")
vd
CrossAttn' = a * CrossAttmrrige original + (1 - a) * Cross
Here parameter is utilized to adjust the balance between global information and individual charac- teristics, aiming to achieve video content more aligned with human aesthetic perception. Finally, we naturally concatenate all the cross-attention values computed along the temporal dimension:
CrossAttn = Concat Concat(CrossAAtt CrrssAAttn CrossAttm", ,.... .rrossAttn)
(4)
In this way, either for a pre-trained text-to-video model such as Modelscope [7], Animatediff' [5], VideoCrafter2 [4] and Latte [6], an auto-regressive model for longer video generation like Atreaming/2V111 this approach can be directly applied in a training-free manner to obtain composi- tional, consistent and aesthetically pleasing results.
3.2 Enhanced Video Data Preprocessing
Enhancement of Motion Dynamics For auto-regressive video generation, we empirically find StreamingT22 [11] is the most effective in producing consistent content. However, there is notable tendency for the occurrence of poor-quality cases and color degradation the later stages of video generation. We attribute this issue to the suboptimal quality of the original training data. To enhance the motion consistency and stability of long video generation, it is imperative to filter the video data to retain high-quality content with consistent motion dynamics. Inspired by Stable Video Diffusion [41], we empirically observed a significant correlation between a video's 's optical flow [42] score its motion magnitude. Excessively low optical flow often corresponds to static video frames, while excessively high optical flow typically indicates frames with intense changes. To ensure the generation of smooth and suitable video data, filter Panda-70M [15] by selecting videos with average optical flow scores computed by RAFT [43] falling within a specified range (81 to 82).
Enhancedment of Prompt Semanties While the Panda-70M's videos exhibit the best visual quality, the paired prompts tend to be relatively brief, which conflicts with our objective of generating videos that adhere to intricate, detailed, and compositional prompts. Directly using such data for training can result in a video generation model that inadequately comprehends complex compositional prompts. Inspired by recent text-to- eext-to-image research [23, 44, 45], it has been demonstrated that high-quality prompts significantly enhance the output quality of visual content. Therefore, after filtering the initial set of videos, we perform a recaptioning process on the selected samples to ensure they better aligned with our objectives. We employ three multimodal LLMs to generate spatio-temporally intricate and detailed descriptions of each video, followed by local LLM to consolidate these descriptions, extract common elements, and add further information. More details on this process can be found in appendix A.2.
3.3 Consistency Regularization with Reference Frame Attention
Given our approach involves the addition and removal of different objects in long videos, maintaining the consistency of each object throughout the video is crucial for final outputs. Most consistent ID control methods, such as IP-Adapter [46], StreamingT2V [11], InstantID [47], and Vlogger [9], typically encode reference images using an image encoder, often CLIP [48], and then integrate the results into the cross-attention block. However, since CLIP is pre-trained on image-text pairs, its image embeddings are designed to align with text. Consistency control, on the other hand, focuses on ensuring that the feature information of the same object in different frames is similar, which does not involve text. We hypothesize that using CLIP for this purpose is indirect approach and propose Reference Frame Attention to maintain the inter-frame consistency of object features.
Formally, we first directly encode the reference images, which are usually the initial frames where the object appears, using the same autoencoder as the pre-trained T2V model. This ensures that the computational target during latent denoising is spatially consistent with the reference target within the hidden representation space. We then train a 2D convolutional layer and projection layer that are structurally identical to those in the original pipeline. This process can be represented as:
Zzef
W WCConv(Coouotto (AutoEncodeer((fee....
(5)

---

LND
AnimoteDif
Piaa
VideoC
Ours
Text prompt: brave knight and a wlse wizard are journeying through a forest.
Text prompt: A talking sponge on the left and a superhero baby on the right are having adventure.
Figure 4: Qualitative Results of Video Generation with Compositional Prompts in Comparision with SOTA Text-to- Video Models
where W, Conv denote the projection layer and the 2D convolutional layer, fkek41 denotes the l frames from index & that are chosen for refernce. After encoding, we insert a Reference Frame Attention block in each attention block that calculates the cross-attention between the current object and the reference object, supplementing the existing attention blocks:
QKT Vd VV,
RefAttn Softmmx
K= + Freff V = Wv oref
(6)
It is noteworthy that to ensure the consistency of different objects across various regions, we need to separately multiply the corresponding object's region mask with Q, K, and V during this computation process, and in practical applications, when a new object emerges in the auto-regressive long video, we precompute corresponding Fref in the relevant regions for further process.
Experiments
4.1 Experimental Setups
We conducted our experiments in two specific scenarios: Video Generation with Compositional Prompts and Long Video Generation for progressive Compositional Prompts. For the first scenario, we directly applied our Spatio-Temporal Compositional Diffusion on VideoCrafter2 [4] to generate videos with F 16 frames. For the second scenario, employed the ControlNet [22]- like branch from Streaming T2V [11] as the backbone and processed the Panda-70M [15] dataset using the Enhanced Video Data Preprocessing methods in section 3.2 the training set. For both scenarios, we used C to generate 100 different prompts/prompt lists as input to the models, generated 6 videos for each prompt, and randomly selected one for comparison. Additional model hyperparameters and implementation details of VideoTetris are provided in appendix A.5.
chat.oppenai.com

---

Table 1: Quantitative Results of Video Generation with Compositional Prompts
Method
VBLIP-VQA
VUnidet
CLIP-SIM
Animatediff [5]
0.3834
0.1921
0.8676
VideoCrafter2 [4]
0,4510
0.1719
0,9249
Gen-2 [53] (Commercial)
0,4427
0.1503
0.9421
Pika [54] (Commercial)
0,4219
0.1782
0.9736
LVD [28] VideoTetris (Ours)
0,4820
0.1934
0.8873
0.5563
0.2350
0.9312
4.2 Metrics
To evaluate compositinal video generation, existing metrics, such as CLIPScore [48] and Fréchet Video Distance (FVD) [49], assess coarse text-video and video-video similarity but do not capture detailed correspondences in object-level attributes and spatial relationships. Instead, extended the T21-CompBench [50] to the video domain and introduced the following metrics for compositional text-to-video evaluation: VBLIP-VQA: the average BLIP [51]-VQA score averaged across all frames and VUnidet: the average Unidet [52] score averaged across all frames. In addition, we followed previous work [10] and used CLIP-SIM [48] to measure the content consistency of generated videos by calculating the CLIP [48] similarity among adjacent frames of generated videos.
4.3 Video Generation with Compositional Prompts
Qualitative Results We compare our VideoTetris with several state-of-the-art text-to-video (T2V) models on their ability to generate videos based on complex compositional prompts. These models include open-source options like LVD [28], VideoCrafter2 [4], and Animatediff [5], well as commercial models Gen-2 [53] and Pika [54]. Using VideoCrafter2 as backbone, we directly evaluate our Opatto-Temporal Compositional Diffusion module's 's training- free performance. In fig. we show text-to-video synthesis results. For the prompt. "A brave knight and wise wizard are journeying through forest," " most models generate two similar characters, blending features and losing individual distinctions. This highlights challenges in semantic alignment and compositional modeling for open-source models. In contrast, our VideoTetris preserves the distinct characteristics of each object and integrates them seamlessly with the background without confining them to fixed regions. For the prompt, "A talking sponge on the left and a superhero baby on the right are having an adventure," models like AnimateDiff split the image, while Runaway Gen-2, Pika, and VideoCrafter2 produce misaligned characters. LVD produces entangled features, resulting in disordered representations. In contrast, our method accurately aligns objects their specified positions while maintaining high video quality, outperforming other methods. Additional examples in fig. demonstrate our model's capability to handle complex prompts with multiple objects, maintaining high quality and adherence to compositional semantics.
Quantitative Results We report our quantitative results in table Our VedeoTeetris achieves the best VBLIP-VQA and VUnidet scores across all models, demonstrating our superiority for complex compositional generation. We also achieved CLIP-SIM higher than the original backbone VideoCratfer 2[4) and comparable to commercial models thanks to accurate semantic understanding. This proves that better text-video alignment can benefit overall consistency.
User Study For further evaluation, we conducted a user study comparing our method with other video generation models, reported in appendix A.4. Using GPT-4, we collected 100 compositional prompts and generated 100 video samples across diverse scenes, styles, and objects. Users compared model pairs by selecting their preferred video from three options: method 1, method and comparable results.
4.4 Long Video Generation for Progressive Compositional Prompts
Qualitative Results We compared our VideoTetris with state-of-the-art long video generation models FreeNoise [10] and Streaming [11]. FreeNoise inherently supports we provide StreamingT2V with different prompts at various frame indexes for multi-prompt video generation. We present our qualitative experimental results in fig. 1 and fig. 5. For the multi-
and

---

Siramiig FieeNoise
Progressive Prompts:
brave young knight journeying through forest (transitions to) brave young knight and a wise wizard are journeying through a forest
Figure 5: Qualitative Results of Long Video Generation for Progressive Compositional Prompts.
prompt sequence from "A brave young knight is journeying through forest" to "A brave young knight and wise wizard are journeying through a forest," FreeNoise generates consistent content; however, the new character appears abruptly, and the two characters switch identities by the end. Additionally, FreeNoise consistently produces near-statee pear-static global motion, with neither the background nor character positions changing. Conversely, Streaming produces bizarre videos in which the knight disappears for half the duration and a merged character appears. successfully models stable and consistent changes in long videos. The new character appears naturally and integrates seamlessly with the existing background throughout the video. Moreover, our approach achieves significantly more dynamic motion compared to FreeNoise. This further demonstrates our method's capability of generating long videos that fully adhere to the evolving semantics while maintaining overall consistency.
contrast, our method
Quantitative Results We report our quantitative results table 2. We achieve the best VBLIP- VQA and VUnidet scores across all models, demonstrating the robust generation capability of our model in compositional video generation. FreeNoise archives better CLIP-SIM score due to its unique noise scheduling method, but this empirically damages transitions in natural story.
Table 2: Quantitative Results of Long Video Generation for Progressive Compositional Prompts
Method
VBLIP-VQA VUnidet CLIP-SIM
FreeNoise [10]
0.4372
0.1823
0.9706
Streaming [11]
0.2412
0.1367
0.6720
VideoTetris (Ours)
0.4839
0.2137
0.9521
4.5
Ablation Study
Effect of Enhanced Video Data Preprocessing We conducted an ablation study about the En- hanced Video Data Preprocessing pipeline, and show the results in fig. 6 and table 3. We directly compare our auto-regressive generation results with the original StreamingT22 [11] using the original prompts and comparison methods. fig. 6 demonstrates the significant improvements we achieved. For the given prompt our model better captures the semantics of "early morning sunlight." In addi- tion, we generate long videos with all test prompts in StreamingT21 and report MAWE [11], CLIP(image-text alignment), AE [55] and CLIP-SIM scores, which further proves our effectiveness.
Table Quantitative Comparison of Ablation Study.
Method
MAWE J
CLIP t
AE f
CLIP-SIM
FreeNoise [10]
49.53
32.14
4.79
0.91
Streaming T2V [11]
10.26
31.30
5,07
0.93
Tetri w/o Reference Frame Attention
10.21
33.50
7.21
0.92
VideoTetris (Ours)
9.98
34.80
8.07
0.96

---

wio Stecomingg
Text prompt:
Close flyover over a large wheat field the early morning sunlight.
Figure 6: Ablation Study. Comparison Between the original StreamingT2V [11], VideoTetris w/o Reference Frame Attention and VideoTetris.
Effeet of Reference Frame Attention We also conducted an ablation study about the Reference Frame Attention, as demonstrated in fig. 6 and table 3. We observe from the result that our Reference Frame Attention achieves more consistent outputs, and the frequency of color artifacts significantly decreases, resulting in a more uniform overall color. This highlights the benefit of aligning reference and noise information semantically the latent space. We provide more ablation studies about compositional approaches in appendix A.3.
5 Conclusion and Discussion
Conclusion In this study, we addressed the limitations of current video generation models incapable of generating compositional video content and introduced novel VideoTetris framework that enables high-quality compositional generation. We propose an efficient Spatio-Tempond Compositional module that decomposes and composes semantic information temporally and spatially in the cross- attention space. Additionally, to further enhance consistency auto-regressive long video generation, introduced an Enhanced Video Data Preprocessing pipeline and designed brand new Reference Frame Attention module. Extensive experiment results confirmed the superiority of our paradigm in extending the generative capabilities of video diffusion models.
Limitations Our proposed method can generate both short and long compositional videos. For fixed text-to-video generation, models. However, for long videos, due to the current performance limitations of long video generation models, method inevitably encounters some bad cases. Additionally, using ControlNet [22] for auto-regressive long video generation results in huge computation cost and overly strong control information, leading an excessive number of transition frames.
directly enhance the compositional performance of existing
Broader Impact Recent notable progress in text-to-video diffusion models has opened up new possibilities in creative design, autonomous media, and other fields. However, the dual-use nature of this technology raises concerns about its socictal impact. There is significant risk of misuse of video diffusion models, particularly the impersonation of individuals. It essential emphasize that our algorithm is designed to enhance the quality of video generation, and we do not support or provide means for malicious uses.
Acknowledgement
This work is supported by National Natural Science Foundation of China (U23B2048, U22B2037), Beijing Municipal Science and Technology Project (Z231 123310001033 1001033300,) research grant No. SH- 2024JK29 and High- performarfm Computing Platform of Peking University.
10

---

References
[1] J. Ho, A. Jain, and P. Abbeel, "Denoising difflusion probabilistic models." Advances in neural information processing systems, vol. 33. pp. 1840-6851,.20 2020.
[2] Y. Song, Sohl-Dickstein, D. P. Kingma, Kumar, Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," arXiv preprint arXiv:201 13456,, 2020.
[3] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Canguli. "Deep unsupervised learning using nonequilibrium thermodynamiies"" in International conferecce on machine learning, pp. 2256-2265, PMLR, 2015.
[4] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang. C. Weng, and Y. Shan, "Videocrafter2: Overcoming data limitations for high-quality video diffusion models," arXiv preprint S.11:22001000447 2024.
[5] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai, "Animatediff: An- imate your personalized text-to-image diffusion models without specific tuning," International Conference on Learning Representations, 2024.
[6] X. Ma, Y. Wang, Jia, X. Chen, Z. Liu, Y-EE Li, C. Chen, and Y. Qiao, "Latte: Latent diffusion transformer for video generation," arXiv preprint 24XY::2000.03044 2024.
[7] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang, "Modelscope text-to-video technical report," arXiv preprint arXiv:2308. 06571, 2023.
[8] G. Oh, J. Jeong, S. Kim, W. Byeon, J. Kim, S. Kim, H. Kwon, and S. Kim, "Mtvg: Multi-text video generation with text-to-video models," arXiv preprint arXiv:2312.04086, 2023.
[9] S. Zhuang, Li,X. X. Chen, Wang. Liu, Qiao, and Y. Wang, "Vlogger: Make your dream a vlog." " arXiv preprint 1xXi:2440.009444 2024.
[10] H. Qiu, M. Xia, Y. Zhang. Y. He, X. Wang. Y. Shan, and Z. Liu, "Freenoise: Tuning-fice free longer video diffusion via noise rescheduling," arXiv preprint arXiv:2310.1 15169, 2023.
[11] R. Henschel, L. Khachatryan, D. Hayrapetyan, H. Poghosyan, V. Tadevosyan, Z. Wang, S. Navasardyan, and H. Shi, "Streaamigt2v:: Consistent, dynamic, and extendable long video generation from text," arXiv preprint 24X7:24033.17773.. 2024.
[12] L. Yang. Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang. "Diffusioo models: A comprehensive survey of methods and applications." " ACM Computing Surveys, vol. 56, no. 4, pp. 1- 11399,200 2023.
[13] L. Yang, Z. Zhang. Z. Yu, J. Liu, M. Xu, S. Ermon, and B. CUI, "Cross-modal contextualized diffu- sion models for text-guided visual generation and editing," in International Conference on Learning Representations, 2024.
[14] Y. Wang, Y. He, Y. Li. K. J. Yu, X. Ma. X. Li, G. Chen, X. Chen, Y. Wang. P. Luo, Z. Liu, Y. Wang. L. Wang, and Y. Qiao, "Internvid: A large-scale video-text dataset for multimodal understanding and generation," " in The Twelfth International Conference on Learning Representations, 2024.
[15] T.-S. Chen, A. Siarohin, W. Menapace, E. Deyneka, H.-w. Chao, B. E. Jeon, Y. Fang, H.-Y. Lec, Ren, M.-H. Yang, et al., "Panda-70m: Captioning 70m videos with multiple cross-modality teachers," arXiv preprint 24X2::2402. 19479, 2024.
[16] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and J. Fleet, "Video diffusion models," Advances in Neural Information Processing Systems, vol. 35, pp. 8633 1633.866,222 2022.
[17] O. Bar-Tal, H. Chefer, O. Tov, C. Herrmann, R. Paiss, S. Zada, A. Ephrat, J. Hur, Y., T. Michaeli, al., "Lumiere: space-time diffuusio model for video generation." " arXiv preprint arXiv:240I.12945 2024.
[18] A. Gupta, L. Yu, K. Sohn. X. Gu. M. Hahn, L. Fei-Fei, I. Essa, L. Jiang, and Lezama, "Photorealistic video generation with diffusion models." arXiv preprint arXiv:2312.06662, 2023.
[19] F.Y.. Wang, W. Chen, G. Song, H.-J Ye, Y. Liu, and H. Li, "Gen-I-video: Multi-text to long video generation via temporal co-denoising," arXiv preprint arXiv:2305. 18264, 2023.
[20] W. Weng, R. Feng, Y. Wang, Q. Dai, C. Wang, D. Yin, Z. Zhao, K. Qiu. J. Bao, Y. Yuan, C. Luo, Y. Zhang, and Z. Xiong, "Art-v; Auto-regressive text-to-video generation with diffusion models." 2023.
11

---

[21] Y. Guo, C. Yang, A. Rao, M. Agrawala, D. Lin, and B. Dai, "Sparsectrl: Adding sparse controls to text-to-video diffusion models." arXiv preprint arXiv:2311.16933, 2023.
[22] L. Zhang, A. Rao, and M. Agrawala, "Adding conditional control text-to-image diffusion models," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023.
[23] L. Yang, Yu, C. Meng, M. Xu, Ermon, and B. Cui, "Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal Ilms," International Conference on Machine Learning, 2024.
[24] Y. Li, H. Liu. Q. Wu, F. Mu, J. Yang. Gao. Li, J. Lee, "Gligen: Open-set grounded text-to-imne -mmae generation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2251 1- 1-22211, 2023.
[25] J. Xie, Y. Li, Y. Huang, H. Liu, W. Zhang, Y. Zheng, and M. Z. Shou, "Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion," in Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 7452-7461, 2023.
[26] L. Lian, B. Li, A. Yala, and T. Darrell, "Lim-grounded difffusio:: Enhancing prompt understanding of text-to-image diffusion models with large language models," arXiv preprint arXiv:2305 2023.
[27] L. Qu, S. Wu, H. Fei. L. Nie, and T.-S. Chua, "Layoutllm- 12i: Eliciting layout guidance from Ilm for text-lo-image generation," " Proceedings of the 31st ACM International Conference on Multimedia, pp. 643-654, 2023.
[28] L. Lian, B. Shi, A. Yala, T. Darrell, and B. Li, "LIm-grounded video diffusion models," in The Twelfth International Conference on Leurning Representations, 2023.
[29] H. Lin, A. Zala, J. Cho, and M. Bansal, "Videodirectorgpt: Consistent multi-scene video generation via Ilm oinggdded planning," arXiv preprint arXiv:2309 15091, 2023.
[30] Z. Tu, W. Pan, Y. Duan, J. Tang, and C. Li, "Rght tracking via reliable feature configuration"" Science China Information Sciences, vol. 65, no. p. 142101, 2022.
[31] C. Xu, Z. Cui, C. Wang, C. Zhou, and J. Yang. "Learning cross- modal interaction for rgb-t tracking." Science China Information Sciences, vol. 38. p. 44, 2023.
[32] C. Wang, S.JJ. Cai, B.-X. Shi, and Z. Chong, "Visual topic semantic enhanced machine translation for multi-modal data efficiency," Journal of Computer Science and Technology, vol. 38, no. 6, pp. 1223-1236, 2023.
[33] Y.Y. Zhang. L. Niu, Y. Makihara, J.-F. Zhang, W.-J. Zhao, Y. Yagi, and L.-Q. Zhang. "Natural image matting with attended global context," Journal of Computer Science and Technology, vol. 38, no. 3, pp. 659-673, 2023.
[34] A. Wang. S. Wen, Y. Gao, Q. Li, K. Deng, and C. Pang. "Toward enhancing room layout estimation by feature pyramid networks," Data Science and Engineering, vol. 7, no. pp. 213 213-224, 2022.
[35] 7.. Lei,, G. Zhang, I.. Wu, K. Zhang, and R. Liang, "A multi-level mesh mutual attention model for visual question answering." Data Science and Engineering, vol. no. 4. pp. 339-353, 2022.
[36] N. Liu, S. Li, Y. Du, A. Torralba, and Tenenbaum, "Compositional visual generation with composable diffusion models," in European Conference on Computer Vision, pp. 423-444-44 439, Springer, 2022.
[37] R. Wang, 7. Chen, C. Chen, Ma, H. Lu, and X. Lin, "Compositional text-to-image synthesis with attention map control of diffusion models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, 2024.
pp.
[38] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or, "Attend-am -nd--addd--daa excieexiie Attention baaned semantic guidance for text-to-image diffusion models," ACM Transactions Graphics (TOG), vol. 42, no. 4, pp. 1 1-00, 2023.
[39] W. Feng, X. He, T.-. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang, "Training-free frre structured diffusion guidance for compositional text-to-image synthesis." arXiv preprint uxXiv:2212.05032,20000 2022.
[40] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E, Chi, Q V. Le, D. Zhou, et al. "Chain-of thought prompting elicits reasoning in large language models," Advances in neural information processing systems. vol. 35. pp. 24824 24837, 2022.
12

---

[41] A. Blattmann, T. Dockhom, S. Kulal, D. Mendelevitch. M. Kilian, D. Lorenz. Y. Levi, Z. English, V. Voleti, A. Letts, et al., "Stable video diffusion: Scaling latent video diffusion models to large datasets," " arXiv preprint 1r11::2311.1512, 2023.
[42] G. Farnebäck, "Twm-frame motion estimation based on polynomial expansion." in Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29 29-July 2. 2003 Proceedings 13, pp. 363- 370, Springer. 2003.
[43] Z. Teed and Deng, "Raft: Recurrent all pairs field transforms for optical flow." in Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pp. 402 419, Springer, 2020.
[44] E. Segalis, D. Valevski, D. Lumen, Y. Matias, and Y. Leviathan, "A picture is worth a thousand words: Principled recaptioning improves image generation," arXiv preprint arXiv:2310. 16656, 2023.
[45] W. Zheng, Teng. Z. Yang. W. Wang. J. Chen, X. Gu, Y. Dong. M. Ding, and J. Tang. "Cogview3: Finer and faster text-to-image generation via relay diffusion." arXiv preprint arXiv:2403.0512) 2024.
[46] H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang. "Ip-adapter: Text compatible image prompt adapter for text-to-imase diffusion models," arXiv preprint wxXiv:230.00672 2023.
[47] Q. Wang. X. Bai, H. Wang. Z. Qin, and A. Chen, "Instantid: Zero-shot identity-preseer preserving generation in seconds," arXiv preprint
2024.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al.. "Learning transferable visual models from natural language supervision," in International conference machine learning, pp. 8748 8763, PMLR, 2021.
[49] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. "Towards accurate generative models of video: A new metric & challenges," arXiv preprint frXiv:1812.01711) 2018.
[50] K. Huang. K. Sun, E. Xie, Z. Li. and X. Liu, "T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation." Advances in Neural Information Processing Systems, vol. 36, pp. 78723-78747, 2023.
[51] J. Li, D. Li. C. Xiong, and S. Hoi, "Blip: Bootstrapping ergguage-image pre-training for unified vision- language understanding and generation," International conference on machine learning. pp. 12888 12900, PMLR, 2022.
[52] X. Zhou, V. Koltun, and P. Krähenbühl, "Simple multi-dataset detection," in Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7571-7580, 2022.
[53] "Gen-2." http:tttse////eerrd wwwwwyyl..wcoomo. genn/ggen,.... 2024.
[54] "Pika labs." https://wwwwt pika. art/ 2023.
[55] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta,
C. Mullis, M. Wortsman, al., "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances in Neural Information Processing Systems, vol. 35, pp. 25278-25294, 2022.
[56] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., "Gpt-4 technical report," arXiv preprint arXiv:2303. 087707744 2023.
[57] H. Liu, W. Yan, M. Zaharia, and P. Abbeel, "World model on million-length video and language with ringattention," arXiv preprint 20X::2202208268,, 2024.
[58] H. Zhang, X. Li, and I.. Bing, "Video-llama: An instruction-tuned audio-visual language model for video understanding." arXiv preprint arXiv:2336600885. 2023.
[59] M. Maaz, H. Rasheed, S. Khan, and S. Khan, "Video-chatgpt: Towards detailed video understanding via large vision and language models," arXiv preprint arXiv:2306.05424 2023.
[60] B. Lin, B. Zhu, Y. Ye, M. Ning, Jin, and L. Yuan, "Vidco-llava: Learning united visual representation by alignment before projection," " arXiv preprint 1rX12:2311.10122. 2023.
[61] M. Chen, I. Laina, and A. Vedaldi, "Training-free layout control with cross-attention guidance," " 2023.
13

---

A Appendix
A.1 LLM-based Automatic Spatio-Tempora Decomposer
Large Language Models (LLMs) have showcased impressive language comprehension, reasoning, and summarization abilities. Using LLMs for specific region generation has been proven efficient and effective in previous works [23, 28, 27, 26]. In our work, we employ the in-context learning (ICL) capability of LLMMs generate reasonable and natural temporal information and spatial regions, eliminating the need for manual spatio-temporal prompt decomposing for each prompt. We construct prompt templates that include task rules (instructions), in-context examples (demonstrations), and the user's s input prompt (test). The specific prompts used for decomposing prompts spatio-temporally and recaptioning sub-prompts are detailed in table 4, table 5 and table 6. In designing our LLM prompts, we focused on clearly defining the roles and guidelines for the LLM, building on insights from previous research [26, 23, 28). Moreover, by guiding the model to use chain-of chain-of-thouggiee tooohuhhh((oo)) [40] reasoning, where it articulates its reasoning process during generation, we empirically achieved better outcomes. This CoT method produced more accurate suggestions compared to when the model's reasoning process was not explicitly detailed. In our experiment, we choose GPT-4 [56] as our L.L..
Your Role: Excellent Story Planner
## Objective: Analyze your input descriptions and plan reasonable story
by providing the specific prompt.
## Process Steps
1. Read the user input story with his given total frames.
Analyze the story, and specify every object and its attribute.
3. Crafting video timeline with prompt and its corresponding frame index, Keep the frame index an integer multiple
4. Explain your understanding (reasoning) and then format your result as examples.
## Examples
Example 1
User prompt: would like to create story about man in cafe. He first drinks coffee alone on a wooden table, and then a young lady with blonde hair comes company. They started chatting joyfully at the end. Total frames: 80
Reasoning: This story contains three main objects: man, wooden table, and young lady with blonde hair. We can split the 80 frames into 3 different parts construct story.
Output: [*0*; "A man is drinking coffee on a wooden table", "32": "A drinking coffee
man and young lady with blonde hair wooden table", "64": "A man and a young lady with blonde hair are drinking coffee and chatting joyfully on a wooden table"]
Example 2:
Your Current Task: Follow the steps closely and accurately identify frame index specific sub-prompts based the given story and total frames. Ensure adherence to the above output format.
User prompt:
{the input user prompt}
Reasoning:
Table 4: Our full prompt for Decomposing Prompts Temporally.
14

---

Your Role: Excellent Prompt Recaptioner
You are an excellent recaptioning bot. Your task is to recaption each given prompt with a more descriptive prompt while maintaining the original meaning with at least 40 words.
You will given with an caption a video, this caption is very short and simple, only containing the main entity and perhaps the simplest description of the background.
Please take the provided caption and expand to at least 40 words upon it by providing additional details. You can start this procedure by following these rules:
## Objective: Recaption each given prompt with a more descriptive prompt while maintaining the original meaning with at least 40 words.
## Process Steps
1. If the original prompt contains words about the camera view, such "top view of" or "camera clockwise", remember to also contain them in the recaption.
2. Describe each entity appearing in this original caption with at least more than two adjectives, making every entity as detailed possible. any other thing that
3. Using your knowledge to fulfill the background should or should not appear in this frames. Adding as much details as you can to enrich the caption, but you shouldn't t change the original meaning of the prompt or any main entity.
4. Your recaption should contain at least 40 words, and you should keep it within words.
5. Your answer should strictly follow the form "Recaption:
6. Your answer must not contain words like "video" or "frame". Only enrich the given prompt.
## Examples
Example 1:
Original Caption: man and woman are walking down hallway Recaption: a man and woman in business attire are seen walking down hallway in a professional building, engaged in a serious discussion with the man holding a book and the woman holding a clipboard, reflecting professional or academic setting.
Example 2:
Your Current Task:
given a caption video, this caption is very short and simple, only containing the main entity and perhaps the simplest description of the background. Please take the provided caption and expand it to at least 40 words by providing additional details.
You
Original Caption: (the input user prompt} Recaption:
Table 5; Our full prompt for Prompt Recaptioning.
15

---

Your Role: Excellent Region Planner
## Objective:
Analyze your input prompts and plan every object's reasonable region in the frame with bounding boxes.
## Process Steps
1. Analyze the given multi-ojject prompt, consider reasonable layout.
2. Define square images with top-left at [o, 0] and bottom-right at [1,
1], and the output Box Format: [Top-left x, Top-left y, Width, Height]
3. Assign each sub-object to specific region. You can start by splitting the original image square.
4. The corresponding regions do not need to be very specific long the region includes the sub-object and all regions never overlap
5. Output the result, and present every object and its region with bounding box. ## Examples
Example
User prompt: A handsome young man and lady with blonde hair are drinking coffee on a wooden table.
Output: ["a handsome young man": "[0.5, O, 0.5, 0.8]", "a lady with blonde hair"; "[O, 0, 0.5, 0.8]", "a wooden table": " "[o, 0.8, 1, 0.
Example 2:
Your Current Task: Follow the steps closely and accurately output each sub-object's bounding box. Ensure adherence to the above output format.
User prompt: (the input user prompt}
Output:
Table 6: Our full prompt for the LLM Spatial Decomposer
A.2 Dataset Prompt Recaptioning
In this section, we detailed our dataset prompt recaptioning process. We first select the top three caption models ranked highest in [57], namely Video-ILaMA [58], Video-ChatGPTT [59] and Video- llava [60], and have them generate captions of 40-50 words for each filtered video. We assume that different caption models may be suitable for different types of video input, so we collect outputs from various models to ensure comprehensive effect. We then append these captions to the original prompt caption provided by Panda-70M. All collected prompts experiment, LLama-3*), to consolidate the captions, extract common elements, and add details. The final unified caption, around 40-50 words in length, used for training each filtered video.
fed into local LLM (in our
A.3 Ablations about Effect of Spatio-Temporal Compositionl Diffusion
In this section, we provided detailed explanations and ablations compared with similar prompt decomposing and object composing diffusion methods, LVD [28], Training-Free Layout Control with Guidance [61], and VideoDirectorGPT [29] in table 7.
The decomposition methods in [28] and [29] only isolate specific tokens from the original prompt. This approach struggles with complex attributes or multiple identical objects, making it difficult for the video generation model understand numeracy and attribute binding, leading significant per- formance degradation these scenarios. In contrast, our decomposition method extracts subobjects and then uses global information for recaptioning, resulting in richer descriptions. Our generated frames are more natural, detailed, and semantically accurate.
hhtps://llaaa.c http://llaaammtta.cmmmlaaaaa
16

---

Next, we have conducted ablation studies about the composition method from [61] and reported the results in the table below. The backward guidance approach [61] tends to restrict objects within specified boxes, offering low flexibility and poor responsiveness to multiple or overlapping objects. In contrast, our model's local-global information fusion ensures that the final generated images are more harmonious and visually appealing, performing better in compositional generation, even in overlapping regions.
Table 7: Ablation Studies for Spatio-Temporal Compositional Diffusion
Method
VBLIP-VQA
VUnidet
CLIP-SIM
Ours w/ Decomposing in LVD [28]
0.5203
0.2139
0,9303
Ours w/ Decomposing in [29]
0,4982
0.2237
0.9178
Ours Composing VideoDirectorGGPT [61]
0.5112
0.1857
0,9073
Viteo'eetris (Ours)
0.5563
0.2350
0.9312
A.4
User Study
To verify the effectiveness of our proposed VideoTetris, we conduct an extensive user study across various scenes and models. Users compared model pairs by selecting their preferred video from three options: method method 2, and comparable results. As presented in fig. 7, our method (orange in left) obtains more preferences than others (blue in right), which further proving its effectiveness.
Preference Rate Comparison
Webektrre
4.%%
57%
Widerkrr
70.3%
23.0%
9.%
Wedeckerss
32.8%
G0
200
Figure 7: User Study about Comparision with Other Video Generation Methods
A.5 Model Hyperparameters and Implementation Details
In this section, we further detailed model hyperparameters and our implementation details. The hyperparameters section 3.2 and section 3.3 are shown table 8. In training process, we randomly drop out 5% of text prompts for Jlassifier-rres guidance training. We trained our model with batch size = and learning rate 1e-5 on A800 GPUs for 16k steps in total.
A.6 More examples
We provided more examples in the figures below.
17

---

Table 8: Hyperparameters of VedeoTetris
Dynamic-Aware Data Filtering
optical flow score threshold $1
0.25
optical flow score threshold
0.75
Diffusion Training
Parametrization
Diffusion steps
1000
Noise scheduler
Linear
3o
0.0085
Sr
0.0120
Sampler
DDIM
Steps
50
!)
1.0
Reference Frame Attention
2D Conv input dim
4
2D Conv output dim
320
2D Conv kernel size
2D Conv padding
MLP hidden layers
MLP inner dim
320
MLP output dim
1024
18

---

Text prompt: curious detective and a sneaky thief are solving a mystery.
Text prompt: A cheerful farmer and hardworking blacksmith building barn.
Text prompt: A speedy train and a leisurely boat are traveling across the country.
Text prompt: A dolphin, her father dolphine and turtle explore underwater city.
Text prompt: A wizard, hot girl, pirate, and a robot walk into bar.
Text prompt: A mother fox, a baby fox and a father fox on camping trip.
Figure 8: More qualitative results of VetroTetris.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope: a novel framework for compositional video genration.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, disscuss the limitications of the work in section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes. We give detailed instructions about experiment setup in section 4.1 and appendix A.5
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We provide open access to our code and data are available at https://gihub..om/Yeae httpgghhwwb.oooo/mmiinggggg. V/iiio
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/uuides/ htwp://oooeSuuuuennnnnen
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.cccpubiic//eecc / ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have also give detailed instructions about experiment setup section 4.1 and appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Error bars are not reported because it would be computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have give detailed information about experiment setup in section 4.1 and appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / httppps///aarisss..cupi///t/t/ee
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We disscuss the limitications of the work in section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of code used in the paper are properly credited, and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper well documented. We provide them as supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "dRELS1BCEf", "reviewer": "Reviewer_Qhp4", "summary": "The paper presents a novel framework designed to improve text-to-video (T2V) generation, especially for complex scenarios involving multiple objects and the composition of different objects.\nThe proposed VideoTetris introduces spatio-temporal compositional diffusion, a dynamic-aware data processing pipeline, and a consistency regularization method to enhance the quality and coherence of generated videos.\nExtensive experiments demonstrate the framework's superior performance in generating both short and long videos with complex and multi-object prompts.", "questions": "- Does the proposed framework have the scalability limitations in terms of the length and complexity of the generated videos?\n- How does the framework perform in real-world scenarios where the input text prompts may be less structured and more diverse?", "limitations": "- Scalability: as mentioned in the weakness, the model highly relies on the performance of LLM which has unstable performance and sometimes has hallucination issue. Hence, it requires manual check and limits the scalability.\n- Limited length of long video generation: also mentioned in the weakness, autoregressive strategy of long video synthesis limits the length of generated video. Could the author provide more or even longer video to verify this limitation?", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I like the idea that the model first splits a story prompt into frame-wise short prompt with object composition. This approach supports sharing global information among long synthesic video (which is usually computationally expensive and hard to achieve).\n- The visual quality provided in the supplementary looks great (although the video number is limited)\n- The experiments include the quantitative comparison with the state-of-the-arts in the tasks of short and long video generation with multi-object prompts, and also ablation study, which can provide better understanding about the performance of the proposed VideoTetris.", "weaknesses": "- My major concern is that the model heavily relies on the performance of LLM. In the proposed method, LLM acts as a director to split an input story prompt into spatiotemporal prompts. I am wondering whether LLM is robust enough to generate temporal-consistent prompt with spatial-consistent composition.\n- Following the concern above, the supplementary only provides one example of long video generation. The authors should more detailed and more samples. For example, provide 100 story prompts and the corresponding frame-wise prompts with object composition. For what we have now, it is hard to fairly justify the model performance.\n- Another concern is the autoregressive strategy of long video synthesis. While the long video shows good consistency of motion and subject identity, I found the visual quality gradually decreases in the video. For example, in the sample \"fig5-ours.mp4\", the contour of leaves is visible at beginning; however, it gets more and more blurry until the end of the video. Is this also happens on other long synthesis videos?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely appreciate the time and effort you have dedicated to reviewing our paper and providing valuable feedback. We are pleased that the spatial-temporal decomposing method, visual quality, and the adequacy of our quantitative and ablation experiments were well-received. Below, we address your comments.*\n\n**Q1: Concerns about LLM's decomposing capabilities. My major concern is that the model heavily relies on the performance of LLM. In the proposed method, LLM acts as a director to split an input story prompt into spatiotemporal prompts. I am wondering whether LLM is robust enough to generate temporal-consistent prompts with spatial-consistent composition.**\n**Q2: Following the concern above, the supplementary only provides one example of long video generation. The authors should more detailed and more samples. For example, provide 100 story prompts and the corresponding frame-wise prompts with object composition. For what we have now, it is hard to fairly justify the model performance.**\n\nA1&A2: \nThank you for raising this concern regarding the robustness of the LLM in our proposed method. We have addressed these concerns in several ways:\n1. **Robustness and Capability of GPT-4**: As detailed in Appendix A.1, we employed GPT-4, a large-scale multimodal model. During its pre-training phase, it leveraged a vast amount of vision-language pairs, which inherently equips it with the capability to establish connections between visual and linguistic elements, thereby enabling accurate spatiotemporal decomposition.\n2. **In-Context Learning (ICL)**: We leveraged in-context learning (ICL) to enhance the model's performance, as outlined in Appendix A.1. By providing the LLM with pre-designed examples, we simplified the learning task, allowing the model to combine its visual knowledge with the ICL examples to produce consistent outputs. This approach is well within the capabilities of GPT-4 and strengthens its ability to handle complex spatiotemporal prompts.\n3. **Validation Through Similar Works**: Similar works, such as VideoDirectorGPT [1], VideoDrafter [2], Vlogger [3], and LVD [4], have adopted the same approach, using LLMs for spatiotemporal planning. Their experimental results have validated the efficacy of this method. Our own quantitative and qualitative experiments, as shown in Section 4, further validate the efficacy of using GPT-4 for this purpose. The consistency and reliability of the outputs in our experiments affirm the model's capability to manage the tasks at hand.\n\n\n\n**Q3: Another concern is the autoregressive strategy of long video synthesis. While the long video shows good consistency of motion and subject identity, I found the visual quality gradually decreases in the video. For example, in the sample \"fig5-ours.mp4\", the contour of leaves is visible at the beginning; however, it gets more and more blurry until the end of the video. Is this also happens on other long synthesis videos?**\n\nA3: The sample video \"fig5-ours.mp4\" you referred to is an isolated case. In our selection process, we did not cherry-pick the results. To address your concern, we have included an additional long video **in the uploaded PDF**, where the quality remains consistent. Compared to the baseline StreamingT2V and FreeNoise, our videos exhibit more accurate and vibrant colors. Due to rebuttal length constraints, we cannot include more video samples here, but we will open-source all code for independent testing. Our method shows significant color enhancements over the baseline StreamingT2V.\n\n**Q4: Does the proposed framework have the scalability limitations in terms of the length and complexity of the generated videos?**\n\nA4: As illustrated in Figure 2, our framework leverages a ControlNet branch for autoregressive generation, similar to StreamingT2V. Taking a T2V UNet that generates 16 frames as a base model, it uses the first 8 frames of a video as the condition to predict the entire 16-frame video. In the real-world long video generation process, this module continuously uses the last 8 frames of the previous segment as the condition to predict the subsequent frames in an autoregressive manner. Consequently, this framework **naturally supports the generation of infinitely long videos** without scalability limitations. We have provided videos with hundreds of frames in the uploaded PDF, which demonstrate stable visual quality throughout, thereby validating the effectiveness of this autoregressive framework.\n\n**Q4: How does the framework perform in real-world scenarios where the input text prompts may be less structured and more diverse?** \n\nA4: As shown in Appendix A.1, Table 5, in real-world scenarios, notably, our framework incorporates a **recaptioning** step after spatiotemporal decomposition. Inspired by approaches like DALL-E 3 and RPG, and various industry practices, this step optimizes irregular prompts into more detailed and coherent descriptions, aligning with the training patterns to generate more visually appealing content. Therefore, for less structured and more diverse input text prompts, the robust understanding capabilities of LLMs ensure that the quality of the generated videos does not significantly decline.\n\n[1] Lin, Han, et al. \"Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.\" arXiv preprint arXiv:2309.15091 (2023).\n\n[2] Long, Fuchen, et al. \"Videodrafter: Content-consistent multi-scene video generation with llm.\" arXiv preprint arXiv:2401.01256 (2024).\n\n[3] Zhuang, Shaobin, et al. \"Vlogger: Make your dream a vlog.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Lian, Long, et al. \"Llm-grounded video diffusion models.\" arXiv preprint arXiv:2309.17444 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gqf8pRGlXi", "reviewer": "Reviewer_pWwG", "summary": "The paper proposes VideoTetris, a novel framework for compositional T2V generation. It introduces Spatio-Temporal Compositional Diffusion method for handling scenes with multiple objects and by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, authors propose a novel Dynamic-Aware Data Processing pipeline to enhance auto-regressive long video generation and a consistency regularization method with Reference Frame Attention that maintains coherence in multi-object generation.", "questions": "1. Novelty Clarification: It is crucial for the authors to provide a clear explanation of the novelty introduced in their paper compared to existing works. This clarification should highlight the unique contributions and advancements made in the proposed approach, distinguishing it from prior works in the field.\n\n2. Baseline Comparison and Ablations: The authors should compare their method with stronger baselines specifically designed for Short Video Generation with Single Multi-Object Prompts. This comparison will provide a more comprehensive evaluation and demonstrate the superiority of their approach. Additionally, conducting ablations for this specific component of the proposed method will further enhance the understanding of its effectiveness and showcase its individual contributions.\n\n\n[1] Lian, Long, et al. \"LLM-grounded Video Diffusion Models.\" *The Twelfth International Conference on Learning Representations.*\n\n[2] Chen M., Laina I., Vedaldi A. Training-Free Layout Control with Cross-Attention Guidance //2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). – IEEE, 2024. – С. 5331-5341.\n\n[3] Lin, Han, et al. \"Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.\" *arXiv preprint arXiv:2309.15091* (2023).", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clarity: The paper exhibits well-written text that is clear and easy to follow. This quality contributes to the overall readability and accessibility of the research.\n\n2. Method: The proposed framework presents a powerful solution for enhancing multi-object T2V generation, while also enabling the generation of long videos. This method offers significant advancements in the field and addresses the challenges associated with generating videos containing multiple objects.\n\n3. Resource-Friendly: Remarkably, the proposed method is resource-friendly, as it only requires 4 A800 GPUs for fine-tuning the model. This efficient utilization of resources makes the method more accessible and practical for implementation in real-world scenarios.", "weaknesses": "1. Novelty\n1.1. The paper lacks clarity regarding the specific novelty introduced by the Spatio-Temporal Compositional Diffusion method. As per my understanding, this method comprises two components: Localizing Subobjects with Prompt Decomposition and Spatio-Temporal Subobjects Composition. However, similar components can be found in existing literature [1, 2, 3]. It is recommended to emphasize the unique aspects and novelty of the paper to distinguish it from prior works.\n\n1.2. The authors highlight the proposed Dynamic-Aware Video Data Processing pipeline as one of the paper's major contributions. However, it appears that this pipeline involves filtering videos with an optical flow score, which has already been proposed in Stable Video Diffusion. If this statement is accurate, the pipeline cannot be considered a contribution of the paper since it lacks novelty.\n\n2. Experiments:\n2.1. The paper's qualitative results are supported by only three examples for Short Video Generation with Single Multi-Object Prompts and two examples for Long Video Generation with Progressing Multi-Object Prompts (as shown in figures 1, 4, and 5). This limited number of examples is not statistically representative. It is recommended to provide more samples for visual comparison and conduct a user study with at least 50 samples and 15 users to ensure a more comprehensive evaluation.\n\n2.2. For Short Video Generation with Single Multi-Object Prompts, the authors compare their method with standard T2V models that are not specifically designed for Compositional Video Generation. It is recommended to include a comparison with LVD 1, as mentioned in Section 2, to provide a more relevant baseline for evaluation.\n\n2.3. The paper lacks ablations for the Spatio-Temporal Compositional Diffusion method. It is recommended to compare the proposed Localizing Subobjects with Prompt Decomposition pipeline with relevant prior works, such as those mentioned in [1] and [3]. Similarly, the Spatio-Temporal Subobjects Composition should be compared with the approach described in [2] to provide a more comprehensive analysis of the proposed method's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback.  We are grateful for your acknowledgment of the clarity, methodological strengths, and resource efficiency of our paper. Please see below for our responses.*\n\n**Q1: Novelty Clarification & Caparison with [1, 2, 3]**\n\nA1: Thank you for your suggestion. Our method fundamentally differs from these three methods in the following ways:\n\n1. **Interactivity, Flexibility and Capacity**: While [1, 2, 3] generate masks for each entity using a standard cross-attention mechanism, our approach innovatively generates linguistic descriptions for each region. It composes cross-attention for sub-prompts in parallel and fuses them from the initial denoising step. Unlike conventional techniques that strictly confine object generation within corresponding masks—thereby limiting expressiveness and fidelity—our method offers greater flexibility. By providing coarse-grained initial positions and sizes, each object can adjust its position and size based on global information, significantly enhancing **the interactivity among objects and resulting in a richer expressive capacity**.\n2. **Compositional Generation**: Our method **for the first time focuses on compositional generation tasks**. In contrast, LVD primarily aims to generate LLM-grounded smooth movements of objects within a box. We empirically find that this backward guidance based methods like [1] and [2] have limitations in compositional generation. As demonstrated in our uploaded PDF, when dealing with multiple objects' attributes or numeracy tasks, the paradigms in [1] and [2] result in inferior outcomes.\n3. **Training-Free Spatio-Temporal Compositional Diffusion**: Compared to VideoDirectorGPT [3], our method is training-free for short video generation. Given the rapid advancements in T2V models, training-based methods struggle to keep pace. Our efficient, **training-free spatio-temporal comspositional diffusion** delivers comparable or superior visual quality, and can generalize to existing video diffusion backbones **seamlessly**.\n\n**Q2:Concerns about Contribution and Novelty of Dynamic-Aware Video Data Processing Pipeline**\n\nA2: Our Dynamic-Aware Video Data Processing pipeline comprises two main components: **Video and Text**, which extends **beyond merely filtering videos**. We first select videos with moderate motion using an optical flow score, then optimize dynamic semantic information for these videos by generating captions with multiple Multimodal Large Language Models (MLLMs) and summarizing them with a local LLM. This results in prompts that capture complex cross-modal semantic consistency between  video contents and text captions.\n\nIn contrast, Stable Video Diffusion (SVD) only filters videos and uses traditional annotation methods, which struggle to maintain **cross-modal semantic consistency**. Our approach achieves this consistency by using MLLMs to match high-quality dynamic texts with corresponding videos.\n\n**Q3: More Samples and missing User Study**\n\nA3: Thanks for your constructive suggestion. We provided six examples for short video generation in Appendix A.6, Figure 8, and included additional short and long video samples **in the uploaded PDF.** For further evaluation, we conducted a user study comparing our method with other video models. Using GPT-4, we generated 100 short prompts and 20 long story prompts, totaling 120 video samples across diverse scenes, styles, and objects. Users compared model pairs by selecting their preferred video from three options: method 1, method 2, and comparable results. The user study results are also reported **in our uploaded PDF**.\n\nThese supplementary materials and the user study demonstrate that our model has significant advantages in qualitative experiments, proving its effectiveness. And we will include these materials and the user study in our final version.\n\n\n**Q4: Comparision with LVD**\n\nA4: We have now included a comparison with LVD **in our uploaded PDF**. The results indicate that while LVD is capable of maintaining the natural motion trajectories of objects, it exhibits more errors in compositional tasks. Additionally, we tested LVD using our metrics and reported the results as in the table. \n\nLVD outperforms modelscope and animatediff but still lags behind VideoTetris, highlighting our method's advantages in compositional tasks. \nIn the final version of our paper, we will include LVD’s results in all comparative experiments and quantitative results to provide a more relevant baseline.\n\n\n**Q5: Ablation studies about comparisons with method [1,2,3]**\n\nA5: Thank you for your insightful comment. We have conducted the ablation studies accordingly. First, we evaluated the decomposition methods from [1] and [3], generated final long videos,  and reported all results **in the uploaded table.** These methods only **isolate specific tokens** from the original prompt, struggling with complex attributes or multiple identical objects. This leads to difficulties in understanding numeracy and attribute binding, causing performance degradation. In contrast, our method extracts subobjects and uses global information for recaptioning, resulting in richer, more natural, detailed, and semantically accurate frames.\n\nNext, we have conducted ablation studies about the composition method from [2] and reported the results **in the uploaded table.** The backward guidance approach in [2] tends to restrict objects within specified boxes, offering low flexibility, interactivity and poor responsiveness to multiple attribute bindings or repeated objects. In contrast, our model's local-global information fusion ensures that the final generated images are more harmonious and visually appealing, performing better in compositional generation.\n\nIn the final version of our paper, we will include all of the above ablation studies to provide a more comprehensive analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAYreCzjpq", "reviewer": "Reviewer_uyep", "summary": "The paper presents \"VideoTetris,\" a new framework designed to improve text-to-video generation in complex scenarios with dynamic changes and multiple objects. It introduces spatio-temporal compositional diffusion techniques for better alignment with textual semantics and integrates a dynamic-aware data processing and consistency regularization to enhance video consistency. The results from extensive experiments demonstrate significant qualitative and quantitative improvements in T2V generation.", "questions": "How is continuity ensured between different shots?", "limitations": "The manuscript includes discussions on limitations and social impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand.\n2. The motivations for spatio-temporal compositional diffusion and Dynamic-Aware Video Data Processing are clearly explained, with supportive experimental evidence provided.\n\n3. The experiments are sufficiently thorough, demonstrating that VideoTetris achieves impressive results in both short and long video production scenarios.", "weaknesses": "1.Although Spatio-Temporal Compositional Diffusion directly adjusts cross-attention, it still segments videos into different shots, each with a specific layout, suggesting it's essentially a layout-based method. This approach contradicts the motivation outlined in section 3.1.\n\n2.The paper presents limited technical innovations, focusing more on engineering implementation. In Spatio-Temporal Compositional Diffusion, it uses LLMs to pre-process prompts and region masks, which are then sequentially applied during generation. Meanwhile, Dynamic-Aware Video Data Processing enhances data preprocessing for higher quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback. We are glad to see that the paper is well written the motivations are clearly explained, and the experiments are sufficiently thorough. Please see below for our responses to your comments.*\n\n**Q1: Although Spatio-Temporal Compositional Diffusion directly adjusts cross-attention, it still segments videos into different shots, each with a specific layout, suggesting it's essentially a layout-based method. This approach contradicts the motivation outlined in section 3.1.**\n\nA1: Our approach fundamentally differs from layout-based methods in that we adopt an initial region-based method. Layout-based methods **strictly confine** each object's generation within its corresponding mask, resulting in limited expressiveness and fidelity. In contrast, we provide both local and global semantic information within specific regions at the outset. This allows us to fuse local and global semantics, enabling the content within each region to adjust its position and size based on global information, significantly enhancing **the interactivity among objects** and resulting in a richer expressive capacity. Our initial positions and sizes of objects can be adaptively adjusted during the generation process; we do not restrict each object to its planned subregion. This region-based division **is more flexible and has a richer expressive capacity.**\n\nWe have included a comparison with LVD [1], a layout-based method, in our uploaded PDF. The quantitative results reported therein highlight our method's advantages over layout-based methods. Our approach significantly outperforms in terms of attribute handling, numeracy, naturalness, and harmony of the generated frames.\n\n**Q2:The paper presents limited technical innovations, focusing more on engineering implementation. In Spatio-Temporal Compositional Diffusion, it uses LLMs to pre-process prompts and region masks, which are then sequentially applied during generation. Meanwhile, Dynamic-Aware Video Data Processing enhances data preprocessing for higher quality.**\n\nA2: To clarify, our **core innovation** is the **Spatio-Temporal Compositional Diffusion**. In real-world video generation tasks, compositional generation is a common and practical scenario. Our method is **the first to define and effectively address this task**, extending beyond single video generation to progressive long video generation. This marks a significant advancement over previous long video generation tasks, involving dynamic changes in objects, which is unprecedented.\n\nOur Dynamic-Aware Video Data Processing and Reference Frame Attention processes further enhance video visual quality by improving both video dynamics and continuity. These processes support our Spatio-Temporal Compositional Diffusion, making it more suitable for industrial production. The task definition, method, and actual results of this combination represent a novel and significant innovation in the field.\n\n\n\n**Q3: How is continuity ensured between different shots?**\n\nA3: As mentioned in Section 3.3, we proposed **Reference Frame Attention** to enhance continuity between different shots. Innovatively, we use VAE instead of CLIP to encode reference frames from the previous video clip, capturing characters and background information. This encoded data is input into cross-attention, interacting with the initial latent of the current denoising step, ensuring continuity across different shots.\n\nAdditionally, our trained **ControlNet-like branch** uses the first 8 frames of a video as a condition to predict the entire 16-frame sequence. Through Dynamic-Aware Video Data Processing, this model ensures smooth autoregressive generation, maintaining consistent base information and natural motion transitions across shots.\n\n[1] Lian, Long, et al. \"Llm-grounded video diffusion models.\" arXiv preprint arXiv:2309.17444 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OUKxD4vXa", "reviewer": "Reviewer_9Z5k", "summary": "This paper proposes VideoTetris, a novel framework for compositional text-to-video generation. It addresses the limitations of existing methods in handling complex scenes with multiple objects and dynamic changes. VideoTetris achieves this through several key innovations: I) Spatio-Temporal Compositional Diffusion: Manipulates the cross-attention of denoising networks to synthesize videos that follow complex instructions. II) Dynamic-Aware Video Data Processing: Filters and recaptions video-text pairs to enhance consistency in auto-regressive long video generation. III) Consistency Regularization with Reference Frame Attention: Maintains coherence in multi-object generation by aligning object features across frames. Extensive experiments demonstrate that VideoTetris significantly outperforms state-of-the-art methods in both short and long video generation tasks, showcasing its ability to generate high-quality, coherent, and compositional videos.", "questions": "N/A. See weakness for more details.", "limitations": "The authors have discussed both limitations and potential negative social impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is easy to follow.\n2.\tThis paper introduces a new approach for compositional text-to-video generation, addressing the limitations of existing methods in handling complex scenes and dynamic changes.\n3.\tExtensive experiments demonstrating the superior performance of VideoTetris compared to state-of-the-art methods in both short and long video generation tasks.", "weaknesses": "1.\tThis paradigm introduces more computational cost by the decomposition of input text using LLMs and the computation of cross-attention for multiple sub-objects and frames. These operations require more computation, especially in scenarios with numerous objects or a high number of frames, potentially impacting the efficiency of the overall video generation process. Besides, the use of ControlNet for auto-regressive generation leads to a high computational cost, which may limit the practical applicability of the method. \n2.\tThe Reference Frame Attention module relies on the assumption that object features remain consistent across frames. However, this may not always hold true, especially for dynamic scenes with fast movements or significant changes in lighting. Investigating more robust consistency regularization methods that can handle these challenges would be beneficial.\n3.\tThe generated videos exhibit relatively subtle variations in content, which could result in static or less dynamic scenes. This may limit their practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for your time and efforts in reviewing our paper and your valuable feedback. We are glad to see that the paper is easy to follow, the approach is novel and the experiment are comprehensive. Please see below for our responses to your comments.*\n\n**Q1: Concerns about the computational cost of LLMs, cross attention, and ControlNet.**\n\nA1：\n1. Using LLMs for prompt enhancement is a widely adopted technique. Similar works like VideoDirectorGPT [1], Vlogger [2], and industrial models such as DALL-E 3 [3] use LLMs for prompt enhancement or decomposition, proving its effectiveness. Our model uses LLMs only once for spatiotemporal decomposition, aligning with most advanced or commercial models.\n2. The computational complexity of the cross-attention mechanism is $O(N\\times L1\\times L2\\times d)$, where $N$ is the number of sub-prompts and $L1, L2$ are the sequence lengths of Q and K/V. This complexity **is negligible compared to the base model (UNet)** and doesn't significantly increase computational cost. Additionally, our cross-attention calculations are **batch-level**, allowing multiple sub-prompts to be processed simultaneously, leveraging GPU parallelism. In practical tests using VideoCrafter2 as the backbone, a single inference on an A800 takes approximately 57 seconds. Adding one sub-prompt increases the total time by around **0.9 seconds.** This additional cost is minimal and acceptable.\n3. ControlNet is employed to balance video quality, generating long videos with superior visual effects, dynamics, and flexibility. Please kindly note that our Spatio-Temporal Compositional Diffusion method can be applied to any video generation framework. We plan to explore more cost-effective autoregressive methods in the future to further improve efficiency and quality.\n\n\n**Q2: Investigating more robust consistency regularization methods that can handle dynamic scenes with fast movements or significant changes in lighting**\n\nA2: Thank you for your insightful comment. We clarify as below:\n1. **Suitability for Autoregressive Long Video Generation**: Given that our research focuses on autoregressive long video generation, our approach necessitates smooth and gradual movements without rapid acceleration or drastic changes. This allows our autoregressive framework to generate coherent videos effectively. Hence, the assumption that object features remain consistent across frames is appropriate for this specific scenario.\n2. **Dynamic Scene Improving**: We have extensively explored dynamic scene handling. Our proposed Dynamic-Aware Video Data Processing method elevates video data to a highly dynamic and trainable level. The Reference Frame Attention module ensures consistency across multiple frames, **significantly improving the dynamics of our videos compared to previous models.** For detailed comparisons, please refer to Section 4.4 and the uploaded PDF, where we showcase long video comparisons.\n3. **Future Work for Extremely Dynamic Scenes**: For scenarios involving extremely dynamic scenes with fast movements or significant changes in lighting, we recommend training separate models tailored to different motion intensities, such as LoRA, motion modules, or specific base models. Alternatively, encoding the magnitude of motion and incorporating it as a condition during training could facilitate the generation of highly dynamic videos. In such cases, setting up multi-frame attention mechanisms like the Reference Frame Attention module, along with additional ID and **motion magnitude conditions**, could prove beneficial. We plan to explore these possibilities in our future work.\n\n\n\n**Q3: The generated videos exhibit relatively subtle variations in content, which could result in static or less dynamic scenes. This may limit their practical applicability.**\n\nA3: Thank you for your observation, we explain below:\n1. **Base Model Limitations for Short Video Generation**: For short video generation, we used VideoCrafter2 as the base model without retraining it. The subtle variations in content are primarily due to the inherent limitations of VideoCrafter2, which was not predominantly trained for dynamic scenes. Therefore, some subtle changes are expected.\n2. **Enhanced Dynamics in Long Video Generation**: It is important to note that for long video generation, we **trained our own model** using a unique Dynamic-Aware Video Data Processing method. This significantly enhances the dynamics of the generated videos. For instance, in Section 4.4 and the uploaded PDF, there is a detailed comparison showcasing the dynamic behavior of a squirrel. The squirrel dynamically changes its position, eats a hazelnut, and interacts with another squirrel throughout a 30-second video. These processes are both highly dynamic and natural, demonstrating the model's capability to produce vibrant and engaging scenes.\n3. **Future Enhancements and Retraining**: With our training process, if we were to retrain the base model for short video generation, we could undoubtedly enhance the dynamic range of the videos, thereby improving their practical applicability. Due to resource constraints, we used a pre-trained model for our experiments. However, we are committed to exploring further possibilities. With ample resources, we plan to retrain the base model to enhance video dynamics. **Increasing dynamics is one of our primary goals**, and we will continue to work towards achieving it.\n\n[1] Lin, Han, et al. \"Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.\" arXiv preprint arXiv:2309.15091 (2023).\n\n[2] Zhuang, Shaobin, et al. \"Vlogger: Make your dream a vlog.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Betker, J., et al. \"Improving\nimage generation with better captions. \"Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their thorough reviews and valuable feedback. We are pleased to hear that our method offers significant advancements in the field (Reviewer pWwG), the paper is well-written and easy to follow (Reviewers pWwG, uyep, and 9Z5K), the visual quality is satisfying (Reviewer Qhp4), and the experiments are sufficiently thorough, demonstrating superior performance (Reviewers Qhp4, uyep, and 9Z5K).

We summarize and highlight our responses to the reviewers as follows:

* We clarify the novelty of our method and the major differences compared to other layout-based generation models (Reviewers pWwG and 9Z5K), providing detailed quantitative and qualitative comparisons (Reviewer pWwG).

* We provide more qualitative results for short and long video generation  **in the uploaded pdf**  (Reviewers Qhp4, pWwG, and 9Z5K).

* We conducted a comprehensive user study and an additional ablation study to further evaluate our method and provide a more comprehensive analysis, reporting all results **in the table of the uploaded PDF** (Reviewer pWwG).

We address each reviewer's concerns in detail below their respective reviews. Please kindly review them. We will include all additional experimental results, including new comparisons, more samples, and detailed ablation studies, **in our final version**. Thank you, and please feel free to ask any further questions.

Meta-review: The paper received 1 Borderline Accept and 3 Weak Accepts. The reviewers praised the presentation, the presented qualitative results, and general method. However, there were some concerns regarding the breadth of experiments (particularly wrt long videos) and limited technical novelty. In general, I agree with the reviewers and recommend Accept as Poster.
Decision: Accept (poster)