VideoTetris: Towards Compositional Text-to-Video Generation

Authors: ye  tian
ling  yang
haotian  yang
yuan  gao
yufan  deng
jingmin  chen
xintao  wang
zhaochen  yu
xin  tao
pengfei  wan
di  zhang
bin  cui
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:31:19

Abstract: Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose a new dynamic-aware data processing pipeline and a consistency regularization method to enhance the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

Main paper: 
 introduction: 
 With the significant development of diffusion models [1] [2] [3] recently, advanced text-to-video models [4] [5] [6] [7] have emerged and demonstrated impressive results. However, these models often struggle with generating complex scenes following compositional prompts, such as "A man on the left walking his dog on the right", which requires the model to compose various objects spatially and temporally. Moreover, with a growing interest in generating long videos, existing methods [8] [9] [10] try to explore multi-prompt long video generation, which is typically limited to simple single-object scene changes. These methods fail to manage scenarios where the number of objects changes dynamically, often resulting in bizarre transformations that do not accurately follow the input text.To overcome these challenges, we introduce VideoTetris, a novel and effective diffusion-based framework to enable compositional text-to-video generation. Firstly, we define compositional video generation as encompassing two primary tasks: (i) Video Generation with Compositional Prompts, which involves integrating objects with various attributes and relationships into a complex and coherent video; and (ii) Long Video Generation with Progressive Compositional Prompts, where 'progressive' refers to the continuous changes in the position, quantity, and presence of objects with different attributes and relationships. Then, we introduce a novel Spatio-Temporal Compositional Diffusion, which manipulates the cross-attention value of denoising network temporally and spatially, synthesizing videos that faithfully follow complex or progressive instructions. Subsequently, to enhance the ability of long video generation models to grasp complex semantics and generate intricate scenes encompassing various attributes and relationships, we propose an Enhanced Video VideoCrafter2 Ours AnimateDiff Pika Gen-2FreeNoise StreamingT2V Data Preprocessing pipeline, augmenting the training data with enhanced motion dynamics and prompt semantics, enabling the model to perform more effectively in long video generation with progressive compositional generation. Finally, we propose a consistency regularization method, namely Reference Frame Attention, that maintains content consistency in a coherent representation space with latent noise while being capable of accepting arbitrary image inputs, ensuring the consistency of multiple objects across different frames and positions. fig. 1(a) showcases our VideoTetris's superior performance in compositional short video generation. We accurately compose two distinct objects with their own attributes while maintaining their respective "left" and "right" positions and ensuring natural interaction between multiple objects. As for long video generation comparisons in fig. 1(b) , FreeNoise [10] either depicts characters appearing abruptly and inexplicably transforming a man into a woman or depicts a squirrel transforming from a hazelnut. StreamingT2V [11] fails to incorporate information about new characters altogether, ignoring quantity information and exhibits severe color distortion in later stages. In contrast, our VideoTetris excels in generating long videos with progressive compositional prompts, seamlessly integrating new characters into the video scenes while maintaining consistent and accurate positional and quantity information. Notably, in generating long videos of the same length, FreeNoise produces only minor variations within the same scene, whereas VideoTetris demonstrates significantly higher motion dynamics, resulting in outputs that more closely resemble long narrative videos.Our contributions are summarized as follows: 1). We introduce a Spatio-Temporal Compositional Diffusion method for handling scenes with multiple objects and following progressive complex prompts. 2). We develop an Enhanced Video Data Preprocessing pipeline to enhance auto-regressive long video generation through motion dynamics and prompt semantics 3). We propose a consistency regularization method with Reference Frame Attention that maintains content coherence in compositional video generation. 4). Extensive experiments show that VideoTetris can generate state-of-the-art quality compositional videos, as well as produce high-quality long videos that align with progressive compositional prompts while maintaining the best consistency. 
related work: 
 Text-to-Video Diffusion Models The field of text-to-video generation has seen significant advancements with the progress of diffusion models [1, 12, 13] and the development of large-scale video-text paired datasets [14, 15] . Early works such as LVDM [16] and ModelScope [7] , adapted 2D image diffusion models by flattening the U-Net architecture to a 3D U-Net and training on extensive video datasets. Subsequently, methods like AnimatedDiff [5] have incorporated temporal attention modules into the existing 2D latent diffusion models, preserving the established efficacy of T2I models. More recently, several transformers-based diffusion methods [17, 18, 6] have enabled large-scale joint training of videos and images, leading to significant improvements in generation quality.Most existing text-to-video diffusion models have been trained on fixedsize video datasets due to the increased computational complexity and resource constraints. Consequently, these models are often limited to generating a relatively small number of frames, leading to significant degradation in quality when tasked with generating longer videos. Several advancements [19, 8, 10, 20] have sought to overcome this limitation through various strategies. More recently, Vlogger [9] and SparseCtrl [21] employ a masked diffusion model for conditional frame input. Although these masked diffusion approaches facilitate longer video generation, they often encounter model inconsistencies and quality degradation due to domain shifts in input. StreamingT2V [11] proposes a new paradigm, utilizing a ControlNet [22] -like conditioning scheme to enable auto-regressive video generation. However, due to the low quality of the training data, the final video outputs often exhibit inconsistent and low-quality artifacts.Compositional Video Generation While current video generation models can synthesize textguided videos, they often face challenges in generating videos featuring multiple objects or adhering to multiple complex instructions, which requires the model to compose objects with diverse temporal and spatial relationships. In the realm of text-to-video diffusion models, exploration of such scenarios remains incomplete. Several text-to-image methods like RPG [23] leverage additional layout or regional information to facilitate more intricate image generation [24] [25] [26] [27] 23] . Within video diffusion techniques, approaches like LVD [28] and VideoDirectorGPT [29] employ a layout-to-video generator to produce videos based on spatial configurations. However, these layout-based methods often offer Figure 2 : The overall pipeline of VideoTetris. We introduce Spatio-Temporal Compositional module for compositional video generation and Reference Frame Attention for consistency regularization. For longer video generation, a ControlNet [22] -like branch can be adopted for auto-regressive generation.only rudimentary and suboptimal spatial guidance, struggling particularly with overlapping objects, thereby resulting in videos with unnatural content. In contrast, our method adopts a compositional region diffusion approach. By explicitly modeling the spatial positions of objects with cross attention maps, our approach allows the objects to naturally integrate and blend during the denoising process, resulting in more realistic and coherent video output. 
method: 
 Overview In this section, we introduce our method VideoTetris for compositional text-to-video generation. Our goal is to develop an efficient approach that enables text-to-video models to handle scenes with multiple objects and follow sequential complex instructions. We first introduce Spatio-Temporal Compositional Region Diffusion in section 3.1, which allows different objects to naturally integrate and blend during the denoising process in a training-free manner. Furthermore, for the task of generating long videos with progressive complex prompts, we construct an auto-regressive model based on the ControlNet [22] architecture and introduced a Enhanced Video Data Preprocessing pipeline in section 3.2 to collect a high-quality video-text pair dataset to train our auto-regressive model for enhanced motion dynamics and prompt understanding. Combined with Spatio-Temporal Compositional Region Diffusion, our auto-regressive model can generate long videos with seamless transitions between diverse target scenes. Finally, we propose a consistency regularization with Reference Frame Attention in section 3.3 for better object appearance preserving.Motivation To achieve natural compositional generation, a straightforward approach is to use the layout as a condition to guide the generation process. However, this method presents several challenges: (i) Requiring large-scale training. Given the significant potential for improvement in layout-to-image models, training a layout-to-video model or training temporal convolution and attention layers for a layout-to-image model would require substantial computational resources and may struggle to keep pace with the latest advancements in text-to-video models. (ii) Layoutbased generation models impose significant constraints on object bounding boxes. For long video duration, the need to continuously adjust the positions and sizes of these boxes to maintain coherent video content introduces a complex planning process, which adds complexity to the overall method. Therefore, instead of training a layout-to-video model, we utilize cross-attention for precise generation [30] [31] [32] [33] [34] [35] and propose a training-free approach that directly adjusts the cross-attention of different targets [23, [36] [37] [38] [39] , as is shown in fig. 3 . This approach aims to overcome the limitations of layout-based methods and leverage the potential of more flexible and efficient generation techniques.Localizing Subobjects with Prompt Decomposition For a given prompt p, we first decompose it temporally into contents at different frames: p = {p 1 , p 2 , • • • , p t }, where t denotes the total number of frames and p i denotes the given text prompt at i-th frame. Subsequently, for the i-th Text Prompt: A little dolphin starts exploring an old city under the sea, she first found a green turtle at the bottom, then her huge father comes along to accompany her at the right side.Cross A+en,on Cross Attention Figure 3 : Illustration of Spatio-Temporal Compositional Diffusion. For a given story "A little dolphin starts exploring an old city under the sea, she first found a green turtle at the bottom, then her huge father comes along to accompany her at the right side.", we first decompose it temporally to Text Prompt #1, #2 and #3, then we decompose each of them spatially to compute each sub-region's cross attention maps. Finally, we compose them spatio-temporally to form a natural story. frame, we decompose the original p i spatially into different sub-objects:{p i 0 , p i 1 , • • • , p i n } with their corresponding region masks M i = {M i 0 , M i 1 , • • • , M i n },where n denotes the number of different objects. In this way, we decompose a prompt list temporally and spatially to acquire each sub-object's corresponding region information in the video timeline. We then calculate the cross attention value for the j-th sub-object at i-th frame as follows:EQUATIONwhere Q i represents the query for the latent frame features, W K , W V are linear projections, ϕ denotes the text encoder, and d is the latent projection dimension of the latent frame features.LLM-based Automatic Spatio-Temporal Decomposer (Optional) Alternatively, the spatiotemporal decomposition process can directly utilize a Large Language Model (LLM) to automate tasks, given the robust performance of LLMs in language comprehension, reasoning, summarization and region generation ablilities [23, 28, 27, 26] . We employ the in-context learning (ICL) capability of LLMs and guide the model to use chain-of-thought (CoT) [40] reasoning. Concretely, we first guide the LLM to decompose the story temporally, generating frame-wise prompts, and reception each one of them with LLM for better semantic richness. Then we use another LLM to decompose each prompt spatially into multiple prompts corresponding to different objects, assigning a region mask to each sub-prompt. The specific prompt templates that include task rules (instructions), in-context examples (demonstrations) are detailed in table 4, table 5 and table 6 of appendix A.1.After we decompose the original prompt list temporally and spatially, we then compose them together from spatial to temporal. To this end, we first compute the cross-attention value of all sub-objects CrossAttn i region at i-th frame with:EQUATIONSubsequently, to ensure a cohesive transition across the boundaries of distinct regions and a seamless integration between the background and the entities within each region, we employ the weighted sum of the CrossAttn region and the CrossAttn original for the original compositional prompt p with :EQUATIONHere α parameter is utilized to adjust the balance between global information and individual characteristics, aiming to achieve video content more aligned with human aesthetic perception. Finally, we naturally concatenate all the cross-attention values computed along the temporal dimension:EQUATIONIn this way, either for a pre-trained text-to-video model such as Modelscope [7] , Animatediff [5] , VideoCrafter2 [4] and Latte [6] , or an auto-regressive model for longer video generation like StreamingT2V [11] , this approach can be directly applied in a training-free manner to obtain compositional, consistent and aesthetically pleasing results.Enhancement of Motion Dynamics For auto-regressive video generation, we empirically find StreamingT2V [11] is the most effective in producing consistent content. However, there is a notable tendency for the occurrence of poor-quality cases and color degradation in the later stages of video generation. We attribute this issue to the suboptimal quality of the original training data. To enhance the motion consistency and stability of long video generation, it is imperative to filter the video data to retain high-quality content with consistent motion dynamics. Inspired by Stable Video Diffusion [41] , we empirically observed a significant correlation between a video's optical flow [42] score its motion magnitude. Excessively low optical flow often corresponds to static video frames, while excessively high optical flow typically indicates frames with intense changes. To ensure the generation of smooth and suitable video data, we filter Panda-70M [15] by selecting videos with average optical flow scores computed by RAFT [43] falling within a specified range (s 1 to s 2 ).Enhancedment of Prompt Semantics While the Panda-70M's videos exhibit the best visual quality, the paired prompts tend to be relatively brief, which conflicts with our objective of generating videos that adhere to intricate, detailed, and compositional prompts. Directly using such data for training can result in a video generation model that inadequately comprehends complex compositional prompts.Inspired by recent text-to-image research [23, 44, 45] , it has been demonstrated that high-quality prompts significantly enhance the output quality of visual content. Therefore, after filtering the initial set of videos, we perform a recaptioning process on the selected samples to ensure they are better aligned with our objectives. We employ three multimodal LLMs to generate spatio-temporally intricate and detailed descriptions of each video, followed by a local LLM to consolidate these descriptions, extract common elements, and add further information. More details on this process can be found in appendix A.2.Given our approach involves the addition and removal of different objects in long videos, maintaining the consistency of each object throughout the video is crucial for final outputs. Most consistent ID control methods, such as IP-Adapter [46] , StreamingT2V [11] , InstantID [47] , and Vlogger [9] , typically encode reference images using an image encoder, often CLIP [48] , and then integrate the results into the cross-attention block. However, since CLIP is pre-trained on image-text pairs, its image embeddings are designed to align with text. Consistency control, on the other hand, focuses on ensuring that the feature information of the same object in different frames is similar, which does not involve text. We hypothesize that using CLIP for this purpose is an indirect approach and propose Reference Frame Attention to maintain the inter-frame consistency of object features.Formally, we first directly encode the reference images, which are usually the initial frames where the object appears, using the same autoencoder as the pre-trained T2V model. This ensures that the computational target during latent denoising is spatially consistent with the reference target within the hidden representation space. We then train a 2D convolutional layer and projection layer that are structurally identical to those in the original pipeline. This process can be represented as: where W, Conv denote the projection layer and the 2D convolutional layer, f k:k+l denotes the l frames from index k that are chosen for refernce. After encoding, we insert a Reference Frame Attention block in each attention block that calculates the cross-attention between the current object and the reference object, supplementing the existing attention blocks:EQUATIONEQUATIONIt is noteworthy that to ensure the consistency of different objects across various regions, we need to separately multiply the corresponding object's region mask with Q, K, and V during this computation process, and in practical applications, when a new object emerges in the auto-regressive long video, we precompute its corresponding x ref in the relevant regions for further process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We disscuss the limitications of the work in section 5 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setups: 
 We conducted our experiments in two specific scenarios: Video Generation with Compositional Prompts and Long Video Generation for progressive Compositional Prompts. For the first scenario, we directly applied our Spatio-Temporal Compositional Diffusion on VideoCrafter2 [4] to generate videos with F = 16 frames. For the second scenario, we employed the core ControlNet [22] like branch from StreamingT2V [11] as the backbone and processed the Panda-70M [15] dataset using the Enhanced Video Data Preprocessing methods in section 3.2 as the training set. For both scenarios, we used ChatGPT3 to generate 100 different prompts/prompt lists as input to the models, generated 6 videos for each prompt, and randomly selected one for comparison. Additional model hyperparameters and implementation details of VideoTetris are provided in appendix A.5. To evaluate compositinal video generation, existing metrics, such as CLIPScore [48] and Fréchet Video Distance (FVD) [49] , assess coarse text-video and video-video similarity but do not capture detailed correspondences in object-level attributes and spatial relationships. Instead, we extended the T2I-CompBench [50] to the video domain and introduced the following metrics for compositional text-to-video evaluation: VBLIP-VQA: the average BLIP [51] -VQA score averaged across all frames and VUnidet: the average Unidet [52] score averaged across all frames. In addition, we followed previous work [10] and used CLIP-SIM [48] to measure the content consistency of generated videos by calculating the CLIP [48] similarity among adjacent frames of generated videos.Qualitative Results We compare our VideoTetris with several state-of-the-art text-to-video (T2V) models on their ability to generate videos based on complex compositional prompts. These models include open-source options like LVD [28] , VideoCrafter2 [4] , and Animatediff [5] , as well as commercial models Gen-2 [53] and Pika [54] . Using VideoCrafter2 as a backbone, we directly evaluate our Spatio-Temporal Compositional Diffusion module's training-free performance. In fig. 4 , we show text-to-video synthesis results. For the prompt, "A brave knight and a wise wizard are journeying through a forest," most models generate two similar characters, blending features and losing individual distinctions. This highlights challenges in semantic alignment and compositional modeling for open-source models. In contrast, our VideoTetris preserves the distinct characteristics of each object and integrates them seamlessly with the background without confining them to fixed regions. For the prompt, "A talking sponge on the left and a superhero baby on the right are having an adventure," models like AnimateDiff split the image, while Runaway Gen-2, Pika, and VideoCrafter2 produce misaligned characters. LVD produces entangled features, resulting in disordered representations. In contrast, our method accurately aligns objects to their specified positions while maintaining high video quality, outperforming other methods. Additional examples in fig. 8 demonstrate our model's capability to handle more complex prompts with multiple objects, maintaining high quality and adherence to compositional semantics.We report our quantitative results in table 1. Our VideoTetris achieves the best VBLIP-VQA and VUnidet scores across all models, demonstrating our superiority for complex compositional generation. We also achieved a CLIP-SIM higher than the original backbone VideoCratfer 2 [4] and comparable to commercial models thanks to accurate semantic understanding. This proves that better text-video alignment can benefit overall consistency.User Study For further evaluation, we conducted a user study comparing our method with other video generation models, reported in appendix A.4. Using GPT-4, we collected 100 compositional prompts and generated 100 video samples across diverse scenes, styles, and objects. Users compared model pairs by selecting their preferred video from three options: method 1, method 2, and comparable results.Qualitative Results We compared our VideoTetris with state-of-the-art long video generation models FreeNoise [10] and StreamingT2V [11] . FreeNoise inherently supports multi-prompts, and we provide StreamingT2V with different prompts at various frame indexes for multi-prompt video generation. We present our qualitative experimental results in fig. 1 and fig. 5 . For the multi- prompt sequence from "A brave young knight is journeying through a forest" to "A brave young knight and a wise wizard are journeying through a forest," FreeNoise generates consistent content; however, the new character appears abruptly, and the two characters switch identities by the end. Additionally, FreeNoise consistently produces near-static global motion, with neither the background nor character positions changing. Conversely, StreamingT2V produces bizarre videos in which the knight disappears for half the duration and a merged character appears. In contrast, our method successfully models stable and consistent changes in long videos. The new character appears naturally and integrates seamlessly with the existing background throughout the video. Moreover, our approach achieves significantly more dynamic motion compared to FreeNoise. This further demonstrates our method's capability of generating long videos that fully adhere to the evolving semantics while maintaining overall consistency.FreeNoise StreamingT2VWe report our quantitative results in table 2. We achieve the best VBLIP-VQA and VUnidet scores across all models, demonstrating the robust generation capability of our model in compositional video generation. FreeNoise archives a better CLIP-SIM score due to its unique noise scheduling method, but this empirically damages transitions in a natural story. 3 . We directly compare our auto-regressive generation results with the original StreamingT2V [11] using the original prompts and comparison methods. fig. 6 demonstrates the significant improvements we achieved.For the given prompt our model better captures the semantics of "early morning sunlight." In addition, we generate long videos with all test prompts in StreamingT2V, and report our MAWE [11] , CLIP(image-text alignment), AE [55] and CLIP-SIM scores, which further proves our effectiveness. We also conducted an ablation study about the Reference Frame Attention, as demonstrated in fig. 6 and table 3 . We observe from the result that our Reference Frame Attention achieves more consistent outputs, and the frequency of color artifacts significantly decreases, resulting in a more uniform overall color. This highlights the benefit of aligning reference and noise information semantically in the latent space. We provide more ablation studies about compositional approaches in appendix A.3. 
conclusion and discussion: 
 Conclusion In this study, we addressed the limitations of current video generation models incapable of generating compositional video content and introduced a novel VideoTetris framework that enables high-quality compositional generation. We propose an efficient Spatio-Temporal Compositional module that decomposes and composes semantic information temporally and spatially in the crossattention space. Additionally, to further enhance consistency in auto-regressive long video generation, we introduced an Enhanced Video Data Preprocessing pipeline and designed a brand new Reference Frame Attention module. Extensive experiment results confirmed the superiority of our paradigm in extending the generative capabilities of video diffusion models.Limitations Our proposed method can generate both short and long compositional videos. For fixed text-to-video generation, we can directly enhance the compositional performance of existing models. However, for long videos, due to the current performance limitations of long video generation models, our method inevitably encounters some bad cases. Additionally, using ControlNet [22] for auto-regressive long video generation results in huge computation cost and overly strong control information, leading to an excessive number of transition frames.Broader Impact Recent notable progress in text-to-video diffusion models has opened up new possibilities in creative design, autonomous media, and other fields. However, the dual-use nature of this technology raises concerns about its societal impact. There is a significant risk of misuse of video diffusion models, particularly in the impersonation of individuals. It is essential to emphasize that our algorithm is designed to enhance the quality of video generation, and we do not support or provide means for malicious uses. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of code used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Large Language Models (LLMs) have showcased impressive language comprehension, reasoning, and summarization abilities. Using LLMs for specific region generation has been proven efficient and effective in previous works [23, 28, 27, 26] . In our work, we employ the in-context learning (ICL) capability of LLMs to generate reasonable and natural temporal information and spatial regions, eliminating the need for manual spatio-temporal prompt decomposing for each prompt. We construct prompt templates that include task rules (instructions), in-context examples (demonstrations), and the user's input prompt (test). The specific prompts used for decomposing prompts spatio-temporally and recaptioning sub-prompts are detailed in table 4, table 5 and table 6. In designing our LLM prompts, we focused on clearly defining the roles and guidelines for the LLM, building on insights from previous research [26, 23, 28] . Moreover, by guiding the model to use chain-of-thought(CoT) [40] reasoning, where it articulates its reasoning process during generation, we empirically achieved better outcomes. This CoT method produced more accurate suggestions compared to when the model's reasoning process was not explicitly detailed. In our experiment, we choose GPT-4 [56] # Your Role: Excellent Prompt Recaptioner You are an excellent recaptioning bot. Your task is to recaption each given prompt with a more descriptive prompt while maintaining the original meaning with at least 40 words. You will be given with an caption of a video, this caption is very short and simple, only containing the main entity and perhaps the simplest description of the background. Please take the provided caption and expand to at least 40 words upon it by providing additional details. You can start this procedure by following these rules:## Objective: Recaption each given prompt with a more descriptive prompt while maintaining the original meaning with at least 40 words.## Process Steps 1. If the original prompt contains words about the camera view, such as "top view of" or "camera clockwise", remember to also contain them in the recaption. 2. Describe each entity appearing in this original caption with at least more than two adjectives, making every entity as detailed as possible. 3. Using your knowledge to fulfill the background or any other thing that should or should not appear in this frames. Adding as much details as you can to enrich the caption, but you shouldn't change the original meaning of the prompt or any main entity. 4. Your recaption should contain at least 40 words, and you should keep it within 60 words. 5. Your answer should strictly follow the form : "Recaption: " 6. Your answer must not contain words like "video" or "frame". Only enrich the given prompt.-Example 1: Original Caption: a man and woman are walking down a hallway Recaption: a man and woman in business attire are seen walking down a hallway in a professional building, engaged in a serious discussion with the man holding a book and the woman holding a clipboard, reflecting a professional or academic setting.-Example 2: ...... Your Current Task: You will be given a caption of a video, this caption is very short and simple, only containing the main entity and perhaps the simplest description of the background. Please take the provided caption and expand it to at least 40 words by providing additional details.Original Caption: {the input user prompt} Recaption: 2. Define square images with top-left at [0, 0] and bottom-right at [1, 1] , and the output Box Format: [Top-left x, Top-left y, Width, Height] 3. Assign each sub-object to a specific region. You can start by splitting the original image square. 4. The corresponding regions do not need to be very specific as long as the region includes the sub-object and all regions never overlap 5. Output the result, and present every object and its region with a bounding box.User prompt: A handsome young man and a lady with blonde hair are drinking coffee on a wooden table. Output: ["a handsome young man": "[0.5, 0, 0.5, 0.8]", "a lady with blonde hair": "[0, 0, 0.5, 0.8]", "a wooden table": "[0, 0.8, 1, 0.2]"] -Example 2:...... Your Current Task: Follow the steps closely and accurately output each sub-object's bounding box. Ensure adherence to the above output format.User prompt: {the input user prompt} Output:Table 6 : Our full prompt for the LLM Spatial DecomposerIn this section, we detailed our dataset prompt recaptioning process. We first select the top three caption models ranked highest in [57] , namely Video-LLaMA [58] , Video-ChatGPT [59] and Videollava [60] , and have them generate captions of 40-50 words for each filtered video. We assume that different caption models may be suitable for different types of video input, so we collect outputs from various models to ensure a comprehensive effect. We then append these captions to the original prompt caption provided by Panda-70M. All collected prompts are fed into a local LLM (in our experiment, LLama-34 ), to consolidate the captions, extract common elements, and add details. The final unified caption, around 40-50 words in length, is used for training each filtered video.In this section, we provided detailed explanations and ablations compared with similar prompt decomposing and object composing diffusion methods, LVD [28] , Training-Free Layout Control with Cross-Attention Guidance [61] , and VideoDirectorGPT [29] in table 7 .The decomposition methods in [28] and [29] only isolate specific tokens from the original prompt. This approach struggles with complex attributes or multiple identical objects, making it difficult for the video generation model to understand numeracy and attribute binding, leading to significant performance degradation in these scenarios. In contrast, our decomposition method extracts subobjects and then uses global information for recaptioning, resulting in richer descriptions. Our generated frames are more natural, detailed, and semantically accurate.Next, we have conducted ablation studies about the composition method from [61] and reported the results in the table below. The backward guidance approach in [61] tends to restrict objects within specified boxes, offering low flexibility and poor responsiveness to multiple or overlapping objects.In contrast, our model's local-global information fusion ensures that the final generated images are more harmonious and visually appealing, performing better in compositional generation, even in overlapping regions. To verify the effectiveness of our proposed VideoTetris, we conduct an extensive user study across various scenes and models. Users compared model pairs by selecting their preferred video from three options: method 1, method 2, and comparable results. As presented in fig. 7 , our method (orange in left) obtains more user preferences than others (blue in right), which further proving its effectiveness. In this section, we further detailed our model hyperparameters and our implementation details. The hyperparameters in section 3.2 and section 3.3 are shown in table 8 . In training process, we randomly drop out 5% of text prompts for classifier-free guidance training. We trained our model with batch size = 2 and learning rate = 1e-5 on 4 A800 GPUs for 16k steps in total.We provided more examples in the figures below. Justification: Yes, we disscuss the limitications of the work in section 5 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.chat.openai.comhttps://llama.meta.com/llama3/ 


Decision: The paper received 1 Borderline Accept and 3 Weak Accepts. The reviewers praised the presentation, the presented qualitative results, and general method. However, there were some concerns regarding the breadth of experiments (particularly wrt long videos) and limited technical novelty. In general, I agree with the reviewers and recommend Accept as Poster.

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework designed to improve text-to-video (T2V) generation, especially for complex scenarios involving multiple objects and the composition of different objects.
The proposed VideoTetris introduces spatio-temporal compositional diffusion, a dynamic-aware data processing pipeline, and a consistency regularization method to enhance the quality and coherence of generated videos.
Extensive experiments demonstrate the framework's superior performance in generating both short and long videos with complex and multi-object prompts.
strength_weakness: strength_weakness: - My major concern is that the model heavily relies on the performance of LLM. In the proposed method, LLM acts as a director to split an input story prompt into spatiotemporal prompts. I am wondering whether LLM is robust enough to generate temporal-consistent prompt with spatial-consistent composition.
- Following the concern above, the supplementary only provides one example of long video generation. The authors should more detailed and more samples. For example, provide 100 story prompts and the corresponding frame-wise prompts with object composition. For what we have now, it is hard to fairly justify the model performance.
- Another concern is the autoregressive strategy of long video synthesis. While the long video shows good consistency of motion and subject identity, I found the visual quality gradually decreases in the video. For example, in the sample "fig5-ours.mp4", the contour of leaves is visible at beginning; however, it gets more and more blurry until the end of the video. Is this also happens on other long synthesis videos?
questions: questions: - Does the proposed framework have the scalability limitations in terms of the length and complexity of the generated videos?
- How does the framework perform in real-world scenarios where the input text prompts may be less structured and more diverse?
limitations: limitations: - Scalability: as mentioned in the weakness, the model highly relies on the performance of LLM which has unstable performance and sometimes has hallucination issue. Hence, it requires manual check and limits the scalability.
- Limited length of long video generation: also mentioned in the weakness, autoregressive strategy of long video synthesis limits the length of generated video. Could the author provide more or even longer video to verify this limitation?
paper_summary: paper_summary: The paper proposes VideoTetris, a novel framework for compositional T2V generation. It introduces Spatio-Temporal Compositional Diffusion method for handling scenes with multiple objects and by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, authors propose a novel Dynamic-Aware Data Processing pipeline to enhance auto-regressive long video generation and a consistency regularization method with Reference Frame Attention that maintains coherence in multi-object generation.
strength_weakness: strength_weakness: 1. Novelty
1.1. The paper lacks clarity regarding the specific novelty introduced by the Spatio-Temporal Compositional Diffusion method. As per my understanding, this method comprises two components: Localizing Subobjects with Prompt Decomposition and Spatio-Temporal Subobjects Composition. However, similar components can be found in existing literature [1, 2, 3]. It is recommended to emphasize the unique aspects and novelty of the paper to distinguish it from prior works.

1.2. The authors highlight the proposed Dynamic-Aware Video Data Processing pipeline as one of the paper's major contributions. However, it appears that this pipeline involves filtering videos with an optical flow score, which has already been proposed in Stable Video Diffusion. If this statement is accurate, the pipeline cannot be considered a contribution of the paper since it lacks novelty.

2. Experiments:
2.1. The paper's qualitative results are supported by only three examples for Short Video Generation with Single Multi-Object Prompts and two examples for Long Video Generation with Progressing Multi-Object Prompts (as shown in figures 1, 4, and 5). This limited number of examples is not statistically representative. It is recommended to provide more samples for visual comparison and conduct a user study with at least 50 samples and 15 users to ensure a more comprehensive evaluation.

2.2. For Short Video Generation with Single Multi-Object Prompts, the authors compare their method with standard T2V models that are not specifically designed for Compositional Video Generation. It is recommended to include a comparison with LVD 1, as mentioned in Section 2, to provide a more relevant baseline for evaluation.

2.3. The paper lacks ablations for the Spatio-Temporal Compositional Diffusion method. It is recommended to compare the proposed Localizing Subobjects with Prompt Decomposition pipeline with relevant prior works, such as those mentioned in [1] and [3]. Similarly, the Spatio-Temporal Subobjects Composition should be compared with the approach described in [2] to provide a more comprehensive analysis of the proposed method's effectiveness.
questions: questions: 1. Novelty Clarification: It is crucial for the authors to provide a clear explanation of the novelty introduced in their paper compared to existing works. This clarification should highlight the unique contributions and advancements made in the proposed approach, distinguishing it from prior works in the field.

2. Baseline Comparison and Ablations: The authors should compare their method with stronger baselines specifically designed for Short Video Generation with Single Multi-Object Prompts. This comparison will provide a more comprehensive evaluation and demonstrate the superiority of their approach. Additionally, conducting ablations for this specific component of the proposed method will further enhance the understanding of its effectiveness and showcase its individual contributions.


[1] Lian, Long, et al. "LLM-grounded Video Diffusion Models." *The Twelfth International Conference on Learning Representations.*

[2] Chen M., Laina I., Vedaldi A. Training-Free Layout Control with Cross-Attention Guidance //2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). – IEEE, 2024. – С. 5331-5341.

[3] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." *arXiv preprint arXiv:2309.15091* (2023).
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The paper presents "VideoTetris," a new framework designed to improve text-to-video generation in complex scenarios with dynamic changes and multiple objects. It introduces spatio-temporal compositional diffusion techniques for better alignment with textual semantics and integrates a dynamic-aware data processing and consistency regularization to enhance video consistency. The results from extensive experiments demonstrate significant qualitative and quantitative improvements in T2V generation.
strength_weakness: strength_weakness: 1.Although Spatio-Temporal Compositional Diffusion directly adjusts cross-attention, it still segments videos into different shots, each with a specific layout, suggesting it's essentially a layout-based method. This approach contradicts the motivation outlined in section 3.1.

2.The paper presents limited technical innovations, focusing more on engineering implementation. In Spatio-Temporal Compositional Diffusion, it uses LLMs to pre-process prompts and region masks, which are then sequentially applied during generation. Meanwhile, Dynamic-Aware Video Data Processing enhances data preprocessing for higher quality.
questions: questions: How is continuity ensured between different shots?
limitations: limitations: The manuscript includes discussions on limitations and social impacts.
paper_summary: paper_summary: This paper proposes VideoTetris, a novel framework for compositional text-to-video generation. It addresses the limitations of existing methods in handling complex scenes with multiple objects and dynamic changes. VideoTetris achieves this through several key innovations: I) Spatio-Temporal Compositional Diffusion: Manipulates the cross-attention of denoising networks to synthesize videos that follow complex instructions. II) Dynamic-Aware Video Data Processing: Filters and recaptions video-text pairs to enhance consistency in auto-regressive long video generation. III) Consistency Regularization with Reference Frame Attention: Maintains coherence in multi-object generation by aligning object features across frames. Extensive experiments demonstrate that VideoTetris significantly outperforms state-of-the-art methods in both short and long video generation tasks, showcasing its ability to generate high-quality, coherent, and compositional videos.
strength_weakness: strength_weakness: 1.	This paradigm introduces more computational cost by the decomposition of input text using LLMs and the computation of cross-attention for multiple sub-objects and frames. These operations require more computation, especially in scenarios with numerous objects or a high number of frames, potentially impacting the efficiency of the overall video generation process. Besides, the use of ControlNet for auto-regressive generation leads to a high computational cost, which may limit the practical applicability of the method. 
2.	The Reference Frame Attention module relies on the assumption that object features remain consistent across frames. However, this may not always hold true, especially for dynamic scenes with fast movements or significant changes in lighting. Investigating more robust consistency regularization methods that can handle these challenges would be beneficial.
3.	The generated videos exhibit relatively subtle variations in content, which could result in static or less dynamic scenes. This may limit their practical applicability.
questions: questions: N/A. See weakness for more details.
limitations: limitations: The authors have discussed both limitations and potential negative social impacts.

Comments: title: Thanks for you reply
Response for the Rebuttal
Thanks for your support
Thank you for your support
Please discuss
Please discuss
Please discuss
Gentle reminder - 2 days left for the author-reviewer discussion
Gentle reminder - 2 days left for the author-reviewer discussion
Gentle reminder - 2 days left for the author-reviewer discussion
Thanks for your comment
Response to Rebuttal
comment: Thanks for checking our rebuttal and keeping your positive score! Due to the rebuttal policy, we cannot provide link for more generation results. We will release both codes and weights for demonstrating the superiority of our VideoTetris on long video generation in final version. We sincerely appreciate your valuable comments and the time and effort you put into reviewing our work.

Warm Regards,

The Authors
Thanks authors for the detailed response. While some of my concerns are addressed, one of my main concerns is the visual quality of the long video generation. The supplementary only shows one example and it is hard to judge the quality based on the sampled video frames provided in the pdf which the authors uploaded during the rebuttal period.
So, I will maintain my initial decision as a borderline accept.
Thank you very much for raising the score! We are glad that we have solved your concerns and we sincerely appreciate your valuable comments and the time and effort you put into reviewing our paper. 
Warm Regards,
The Authors
Thanks for checking our rebuttal keeping your positive score! We sincerely appreciate your valuable comments and the time and effort you put into reviewing our work. 

Warm Regards,

The Authors
Thank the authors for the rebuttal. I am satisfied with the response and decide to keep my score.
Thank you to the authors for the detailed response. Most of my concerns have been addressed, thus I have raised my score.
Dear reviewer,

The discussion period is coming to a close soon. Please do your best to engage with the authors.

Thank you,
Your AC
Dear reviewer,

The discussion period is coming to a close soon. Please do your best to engage with the authors.

Thank you,
Your AC
Dear reviewer,

The discussion period is coming to a close soon. Please do your best to engage with the authors.

Thank you,
Your AC
Dear Reviewer Qhp4,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding our method's innovation and continuity ensuring, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards,
The Authors
Dear Reviewer 9Z5k,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding computational cost and motion dynamics, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards,
The Authors
Dear Reviewer Qhp4,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries regarding LLM capabilities, motion dynamics, and model scalability, we have prepared detailed answers.

As the discussion period is set to conclude in two days, we would like to ask if you had a chance to check our response to your rebuttal questions. Should you require any further clarification or improvements, please know that we are fully committed to addressing them promptly!

Thank you once again for your invaluable contribution to our research.

Warm regards,
The Authors
Thank you very much for raising the score! We sincerely appreciate your valuable comments and the time and effort you put into reviewing our paper. We are pleased that the additional experiments were well received and will make sure to include the results in the camera-ready version.

Warm Regards,  
The Authors
Dear Authors,

Thank you for your comprehensive rebuttal. I appreciate the effort you have made to address all my concerns. Based on this, I have decided to raise my score and recommend acceptance of your paper.

I would also like to commend the authors for conducting additional experiments. In my opinion, these results provide valuable insights and would be a useful addition to the article. Therefore, I strongly recommend that they be included in the camera-ready version of the paper.