LLM-grounded Video Diffusion Models
Authors: Long Lian
Baifeng Shi
Adam Yala
Trevor Darrell
Boyi Li
Venue: ICLR 2024 poster (2024)
Keywords: text-to-video generation
diffusion models
large language models
Primary area: generative models
TL;DR: LLM-grounded Video Diffusion improves text-to-video generation by using an LLM to generate dynamic scene layouts from text and guiding video diffusion models with the layouts, achieving high-quality video generation aligned with complex prompts.

Abstract: Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion. To address these limitations, we introduce LLM-grounded Video Diffusion (LVD). Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a diffusion model for video generation. We show that LLMs are able to understand complex spatiotemporal dynamics from text alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world. We then propose to guide video diffusion models with these layouts by adjusting the attention maps. Our approach is training-free and can be integrated into any video diffusion model that admits classifier guidance. Our results demonstrate that LVD significantly outperforms its base video diffusion model and several strong baseline methods in faithfully generating videos with the desired attributes and motion patterns.

Published as a conference paper at ICLR 2024
LLM-GROUNDED VIDEO DIFFUSION MODELS
Long Lian'*, Baifeng Shi'; Adam Yal Trevor Darrell', Boyi Li UC Berkeley UCCF
{longlian, baifeng_shi shi, yala, b
ABSTRACT
Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotem- poral prompts and often generate restricted or incorrect motion. To address these limitations, we introduce ALM-grounded Video Diffusion (LVD). Instead of di- rectly generating videos from the text inputs, LVD first leverages large language model (LLM) to generate dynamic scene layouts based on the text inputs and subse- quently uses the generated layouts guide diffusion model for video generation. We show that LLMs are able to understand complex spatiotemporal dynamics from alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world. We then propose to guide video diffusion models with these layouts by adjusting the attention maps. Our approach is training-free and can be integrated into any video diffusion model that admits classifier guidance. Our results demonstrate that LVD significantly outperforms its base video diffusion model and several strong baseline methods in faithfully generating videos with the desired attributes and motion patterns.
INTRODUCTION
Text-to-image generation has made significant progress in recent years (Saharia et al., 2022; Ramesh et al., 2022). In particular, diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Ho et al., 2022b; Nichol et al., 2021; Nichol & Dhariwal, 2021; Rombach et al., 2022) have demonstrated their impressive ability generate high-quality visual contents. Text-to-vidoo generation, however, more challenging, due to the complexities associated with intricate spatial-temporal dynamics. Recent works (Singer al., 2022; Blattmann al., 2023; Khachatryan al., 2023; Wang al.,
Text Prompt
Baseline
(Ours)
brown bear dancing with pikachu
A wooden barrel drifting
bird flying the right
mecooon
walking
lowards
Figure 1: Left: Existing text-to-video diffusion models such as Wang et al. (2023) often encounter challenges in generating high-quality videos that align with complex prompts. Right: Our training- free method LVD, when applied on the same model, allows the generation of realistic videos that closely align with the input text prompt.
"Equal contribution.
1 Equal advising.

---

Published as a conference paper at ICLR 2024
2023) have proposed text-to-video models that specifically aim to capture spatiotemporal dynamics in the imput text prompts. However, these methods still struggle to produce realistic spatial layouts or temporal dynamics that align well with the provided prompts, as illustrated in Fig. 1.
Despite the enormous challenge for a diffusion model to generate complex dynamics directly from text prompts, one possible workaround is to first generate explicit spatiotemporal layouts from the prompts and then use the layouts to control the diffusion model. In fact, recent work (Lian et al., 2023; Feng et al., 2023; Phung et al., 2023) on text-to-image generation proposes to use Large Language Models (LLMs) (Wei et al., 2022; OpenAI, 2020; 2023) to generate spatial arrangement and use it to condition text-to-image models. These studies demonstrate that LLMs have the surprising capability of generating detailed and accurate coordinates of spatial bounding boxes of each object based on the text prompt, and the bounding boxes can then be utilized to control diffusion models, enhancing the generation of images with coherent spatial relationships. However, it has not been demonstrated if LLMSs can generate dynamic scene layouts for videos on both spatial and temporal dimensions. The generation of such layouts is much harder problem, since an object's motion patterns often depend on both the physical properties (e.g. gravity) and the object's attributes (e.g., elasticity vs rigidity).
In this paper, we investigate whether LLMs can generate spatiotemporal bounding boxes that are coherent with a given text prompt. Once generated, these box sequences, termed Dynamic Scene Layouts (DSLs), and the video. With a simple yet effective attention-guidant leveraged to control the generation of object-level spatial relations and temporal dynamics in training-free manner. The whole method, referred to as LLM-grounded Video Diffusion (LVD), is illustrated in Fig. 2. As shown in Fig. 1, LVD generates videos with the specified temporal dynamics, object attributes, and spatial relationships, thereby substantially enhancing the alignment between the input prompt and the generated content.
serve as intermediate representation bridging the gap between the text prompt
algorithm, these L.LM-generatdd DSLs
To evaluate LVD's ability to generate spatial layouts and temporal dynamics that align with the prompts, we propose benchmark with five tasks, each requiring the understanding and generation of different spatial and temporal properties in the prompts. We show that LVD significantly improves the tixt- alignment compared to several strong baseline models. We also evaluate LVD on common datasets such as UCF-101 (Soomro et al., 2012) and MSR-VTT (Xu et al., 2016) and conducted an evaluator- based assessment, where LVD shows consistent improvements over the base diffusion model that it uses under the hood.
Contributions. 1) We show that text-only LLMs are able to generate dynamic scene layouts that generalize to previously unseen spatiotemporal sequences. 2) We propose LLM-grounded Video Diffusson (LVD). the first training-free pipeline that leverages LLL--generated dynamic scene layouts for enhanced ability to generate videos from intricate text prompts. 3) We introduce benchmark for evaluating the alignment between input prompts and the videos generated by text-to-video models.
RELATED WORK
Controllable diffusion models. Diffusion models have made a huge success in content cre- ation (Ramesh et al., 2022; Song & Ermon, 2019; Ho et al., 2022b; Liu et al., 2022; Ruiz et al., 2023; Nichol al., 2021; Croitoru et al., 2023; Yang et al., 2022: Wu al., 2022). ControlNet (Zhang et al., 2023) proposes an architectural design for incorporating spatial conditioning controls into large, pre-trained text-to-image diffusion models using neural networks. GLIGEN (Li et al., 2023) introduces gated attention adapters to take in additional grounding information for image generation. Shape-gu guided Diffusion (Huk Park al.. 2022) adapts pretrained diffusion models to respond to shape input provided by user or inferred automatically from the text. Control-A- Video (Chen et al., 2023b) train models to generate videos conditioned on a sequence of control signals, such as edge or depth maps. Despite the impressive progress made by these works, the challenge of generating videos with complex dynamics based solely on text prompts remains unresolved.
Text-to-video generation. Although there is rich literature on video generation (Brooks et al., 2022; Castrejon et al., 2019; Denton & Fergus, 2018; Ge et al., 2022; Hong et al., 2022; Tian et al., 2021; Wu et al., 2021), text-to-video generation remains challenging since it requires the model to synthesize the video dynamics based only on text. Make-A- Video (Singer et al., 2022) breaks down the entire temporal U-Net (Ronneberger et al., 2015) and attention tensors, approximating
2

---

Published as a conference paper at ICLR 2024
Dynamic Scene Layouts (DSL)
User Text Prompt
Stage
LLM Spatiotemporal Planner
raccoon walking from the left to the right
Generated Video
Stage
DSL-grounded Video Generator
Figure 2: Our method LVD improves text-to-video diffusion models by turning the text-to-video generation into a two-stage pipeline. In stage I, we introduce an LLM as the spatiotemporal planner that creates plans for video generation in the form of dynamic scene layout (DSL). A DSL includes objects bounding boxes that are linked across the frames. In stage 2, we condition the video generation on the text and the DSL with DSL-ggounded video generator. Both stages are training-freet I.LMss and diffusion models are used off-the-shelf without updating their parameters. By using DSL as an intermediate representation for text-to-video generation, LVD generates videos that align much better with the input prompts compared to its vanilla texi-to-video model counterpart.
them in both spatial and temporal domains, and establishes a pipeline for producing high-resolution videos. Imagen Video (Ho al., 2022a) creates high-definition videos through a foundational video generation model and spatial-temports video super-resolution models. Video LDM (Blattmann et al., 2023) transforms the image generator into video generator by adding a temporal dimension to the latent space diffusion model. Text2Video-7ar (Khachatryan et al., 2023) introduces two post- processing techniques for ensuring temporal consistency: encoding motion dynamics in latent codes and reprogramming frame-level self-attention with cross-frame attention mechanisms. However, these models still easily fail in generating reasonable video dynamics due to the lack of large-scale paired vidoo training data that can cover diverse motion patterns and object attributes, let alone the demanding computational cost to train with such large-scale datasets.
Grounding and reasoning from large language models. Several recent text-to- text-to-image models propose to feed the input text prompt into an LLM to obtain reasonable spatial bounding boxes and generate high-quality images conditioned the boxes. LMD (Lian al., 2023) proposes training- free approach to guide a diffusion model using an innovative controller to produce images based on LLM-generated layouts. LayoutGPT (Feng al., 2023) proposes a program-gus guided method to adopt LLMs for layout-oriented visual planning across various domains. Attention refocusing (Phung al., 2023) introduces two innovative loss functions realign attention maps in accordance with specified layout. Collectively, these methods provide empirical evidence that the bounding boxes generated by the LLMs are both accurate and practical for controlling text-to-image generation. In light of these findings, our goal is to enhance the ability of text-to-video models to generate from prompts that entail complex dynamics. We aim to do this by exploring and harnessing the potential of LLMs in generating spatial and temporal video dynamics. A concurrent work VideoDirectorGPT (Lin et al., 2023) proposes using an LLM multi-scene video generation, which pursues different focus and technical route. It explores training diffusion adapters for additional conditioning, while LVD focuses on inference-time guidance and in-depth analysis on the generalization of the LLMs.
CAN LLMS GENERATE SPATIOTEMPORAL DYNAMICS?
In this section, we explore the extent to which LLMs are able to produce spatiotemporal dynamics that correspond to specified text prompt. We aim to resolve three questions in this investigation:
1. Can LLMs generate realistic dynamic scene layouts (DSLs) aligned with text prompts and discern when apply specific physical properties?
2. Does the LLM's knowledge of these properties come from its weights, or does the understanding of these concepts develop during inference the fly?
3. Can LLMs generalize to broader world concepts and relevant properties based on the given examples that only entail limited key properties?
3

---

Published as a conference paper at ICLR 2024
fol Iolleccnteni cuapple fer aranity A nocean wallnn foo Eoe lef. sigiigh asis janjuriiing on tle-ige is cooes
(b) le-otalool ouarpie clasiidity Aud ball Usona from Lae lol wh right is gaddar
(c) Insosiool ousppi persective cassere projecion.. The is toviig away ficts upasting
Figure 3: In-context examples. We propose to prompt the LLMS with only one in-context example per key desirable property. Example (a)/(b)(c) demonstrates gravitylllasicityprrsspeee respectively. In Section 3, we show LLMs are able to generate DSLs aligned with the query prompts with only these in-context examples, empowering downstream applications such as video generation.
projection,
To prompt the LLM for dynamics generation, we query the LLM with a prompt that consists of two parts: task instructions in text and a few examples to illustrate the desired output and rules to follow, Following the prompt, we query the LLM perform completion, hereby generating the dynamics.
Task instructions. We ask the LLM to act as a "video bounding box generator". In the task setup, we outline task requirements like the coordinate system, canvas size, number of frames, and frame speed. We refer readers to our Appendix A.3 for the complete prompt.
DSL representation. We ask the LLM to represent the dynamics for each video using a Dynamic Scene Layout (DSL), which is set of bounding boxes linked across the frames. LLMs are prompted to sequentially generate the boxes visible in each frame. Each box has a representation that includes the location and size of the box in numerical coordinates, as well as a phrase describing the enclosed object. Each box is also assigned numerical identifier starting from 0 and boxes across frames are matched using their assigned IDs without employing a box tracker.
In-context examples. LLMs might not understand our need for real-world dynamics such as gravity in the DSL generation, which can lead to assumptions that may not align with our specific goals. To guide them correctly, we provide examples that highlight the desired physical properties for generating natural spatiotemporal layouts. A key question arises: How many examples do LLMs need in order to generate realistic layouts, given they are not designed for real-world dynamics? One might assume they need numerous prompts to grasp the complex math behind box coordinates and real-world dynamics. Contrary to such intuition, we suggest presenting the LLM with a few examples demonstrating key physical properties and using only one example for each desired property. Surprisingly, this often suffices for LLMs to comprehend various aspects including physical properties and camera motions, empowering downstream applications such as video generation (Section 4). We also find that LLMs can extrapolate from provided examples to infer related properties that are not explicitly mentioned. This means we do not need to list every desired property but just highlight a few main ones. Our in-context examples are visualized in Fig. 3, with more details in Appendix A.3.
Reason before generation. To improve the interpretability of the LLM output, we ask the LLM to output a brief statement of its reasoning before starting the box generation for each frame. We also include a reasoning statement in each in-context example to match the output format, We refer readers to Appendix A.1 for examples and analysis of reasoning statements from the LLM.
Investigation setup. With the potential downstream of video generation, we categorize the desired physical properties into world properties and camera properties. Specifically, examine gravity and object clasticity as world properties and perspective projection as a camera property. However, the surprising generalization capabilitiess indicate promising opportunities for LLMS to generalize to other uninvestigated properties. As displayed in Fig. 3, we provide the L.M with one example for each of the three properties. We then query the LLM with a few text prompts for both world properties and camera properties (see Fig, 4). We use GPT-4 (OpenAI, 2023) for investigation in this section. Additional models are explored in Section 5, confirming they exhibit similar capabilities.
Discoveries. We discovered that for various query scenarios, either seen or unseen, LLMs can create realistic DSLs aligned with the prompt. Notably, this ability is rooted in the LLM's inherent knowledge, rather than solely relying on provided examples. For instance, given the ball example that shows the presence of elasticity in our desired world for LLM to simulate (Fig. 3(b)). LLM understands that the ball, when thrown, will hit the ground and bounce back (Fig. 4(a)). no matter whether it was thrown from the left (Fig. 3(b) or right. Furthermore, LLM is able to generalize to objects with different elasticity. For example, when replacing the ball with rock without updating in-context examples (Fig. 4(b)), LLM recognizes that rocks are not elastic, hence they do not bounce, but they still obey gravity and fall. This understanding is reached without explicit textual cues about the object's 's nature in either our instructions or our examples. Since this ability is not derived from the input prompt, it must originate from the LLM's weights. To illustrate our point further, we show
4

---

Published as a conference paper at ICLR 2024
bal
paper applon
Ada
(G)AA poper airplane thrown out from right back is driving Rerward
(d) Aar
(c)
from right
from the right
top driving forward
side is driving Feward
Figure 4: DSLS generated by the LLM from input text prompts with only in-context examples in Fig. 3. Remarkably, despite not having any textual explanations on the applicability of each physical property, the LLM discerns and applies them appropriately. (a,b) The LLM selectively applies the elasticity rule to the ball but not the rock, even though the attributes of a rock are never mentioned in the example. (c) The LLM infers world context from the examples and factors in air friction, which is not mentioned either, when plotiing the paper airplane's trajectory. (d-f) reveal the LLM's inherent grasp of the role of viewpoint in influencing object dynamics, without explicit instructions in the examples. These discoveries suggest that the L.M''s 's innate knowledge, embedded in its weights, drives this adaptability, rather than solely relying on the provided examples.
(6) A ball throwa out from the right
(b) rock thrown out from the right
(c) paper airplane is thrown out from the right
(d) car viened from the back driving forward (6) car viewed from the top driving forward (0 car viewed from the side driving forward
Figure 5: Videos generated by our method LVD from the DSLs in Fig. 4. Our approach generates vidcos that correctly align with input text prompts. (a-c) show various objects thrown from the right to the left. (d-f) depict objects of the same type viewed from different viewpoints.
the LLM that the "imagined camera" follows the principle of perspective geometry in Fig. 3(c): as the camera moves away from a painting, the painting appears smaller. We then test the LLM with scenario not directly taught by our examples, shown in Fig. 4(e,f). Even without explicit explanations about how perspective geometry varies with camera viewpoint and object motion, the LLM can grasp the concept. If the car's forward movement perpendicular to the direction that the camera is pointing at, perspective geometry does not apply.
More compellingly, by setting the right context of the imagined world, LLMs can make logical extensions to properties not specified in the instructions or examples. As shown in Fig. 4(d), the LLM recognizes that the principles of perspective geometry applies not just when a camera moves away from static object (introduced in Fig. 3(c)), but also when objects move away from a stationary camera. This understanding extends beyond our given examples. As seen in Fig. 4(c), even without explicit mention of air friction, the LLM takes it into account for a paper airplane and generates DSL which the paper airplane is shown to fall slower and glide farther than other queried objects. This indicates that examples help establish imagined world for the LLM to simulate, removing the need for exhaustive prompts for every possible scenario. LLMs also demonstrate understanding of buoyancy and animals' moving patterns generate realistic DSLs for Fig. in the reasoning statements. We refer the readers to these statements in Appendix A.
These observations indicate that LLMs able to successfully generate DSLs aligned with complex prompts given only the three in-context examples in Fig. 3. We therefore use these in-context examples and present further quantitative and qualitative analysis in Section 5.
4 DSL-GROUNDED VIDEO GENERATION
Leveraging the capability of LLMs to generate DSLS from text prompts, we introduce algorithm for directing video synthesis based on these DSLs. Our DSL-grounded video generator directs an off off-the-shelf text-to-video diffusion model to generate videos consistent with both the text prompt and the given DSL. Our method does not require fine-tuning, which relieves from the need for instance-annotate images videos and the potential catastrophic forgetting incurred by training.
5

---

Published as a conference paper at ICLR 2024
Dynamic Scene
Generated Video
Layouts (DSL)
DSL-grounded Video Generator
raccoor BOOO on
DEL
Denalsing Stup
DeL Cundarice
Denelsing
Franm
Nase Sy
wolking food the Nolt the right
DSL Guidance
Dillusion UNe
Energy function
such that
E' low when
region has high attention
Input Fartally Vidwo
Partssty Denslsed
low
region low attantion
Output
energy
didoo
Extrocting image text crose antectionn reep for phrase "p roccoo""
Snergy minimization via grackent descent on X
5600 tng thern
Gradiort
Figure 6: Our DSL.-grounded video generator generates videos from DSLs using existing text-to- video diffusion models augmented with appropriate DSL guidance. Our method alternates between DSL guidance steps and denoising steps. While the denoising steps gradually transform the random video-shaped Gaussian noise into clean videos, our DSL guidance steps ensure that the denoised videos satisfy our DSL constraints. We designed an energy function minimized when the text-to- video cross-attention maps have high values within the box area and low values outside the box w.r.t. the layout box name. In the DSL guidance step, our method minimizes the energy function E by performing gradient descent on the video, steering the generation follow the constraints.
text-to-video diffusion model (e.g.. Wang et al. (2023)) generates videos by taking in a random Gaussian noise of shape N H x W x C, where N is the number of frames and C' is the number of channels. The backbone network (typically a U-Net (Ronneberger et al., 2015)) iteratively denoises the input based on the output from the previous time step as well the text prompt, and the text- conditioning is achieved with cross- aoos-ntttentoon layers that integrate text information into latent features. For each frame, we denote the gross-attention map from latent layer to the object name in the text prompt by A E R xW where H and W are the height and width of the latent feature map. For brevity, we ignore the indices for diffusion steps, latent layers, frames, and object indices. The cross-attention map typically highlights positions where objects are generated.
As illustrated in Fig. 6, to make sure an object is generated in and moves with the given DSL, we encourage the gross-attention map to concentrate within the corresponding bounding box. To achieve this, we turn each box of the DSL into mask M IR HxW, where the values are ones inside the bounding box and zeros outside the box. We then define energy function Eiop -Topk(A A M) + Topk(A (A (1 M)), where- is element-wise multiplication, and Topk takes the average of top-k values in matrix. Intuitively, minimizing this energy encourages at least & high attention values inside the bounding box, while having only low values outside the box.
However, if the layout boxes in consecutive frames largely overlap (i.e., the object is moving slowly), Eupp may result imprecise position control. This is because stationary objects in the overlapped region also minimize the energy function, which the text-to-video diffusion model often prefers to generate. To further ensure that the temporal dynamics of the generated objects are consistent with the DSL guidance, we propose Center-of-Mann Mass (CoM) energy function ECOM which encourages CoM of A and M to have similar positions and velocities. Denoting the CoM of t-th frame's 's attention map and box mask by p.A and p, the ECOM = I|p,* P. Overall energy E is weighted sum of Ejop and ECAM-
MM113 112 1l pAA) (pM Il.
To minimize the energy E, we add DSL guidance step before each diffusion step where we calculate the gradient of the energy w.r.t. the partially denoised video for each frame and each object, and update the partially denoised video by gradient descent with classifier guidance (Dhariwal Nichol, 2021; Xie et al., 2023; Chen et al., 2023a; Lian et al., 2023; Phung et al., 2023; Epstein et al., 2023).
With this simple yet effective training-fres conditioning stage, we are able to control video generation with DSL and complete the LVD pipeline of text prompt + DSL - video. Furthermore, the two stages of our LVD pipeline are wewremennatiionnnnnsttte and can be easily swapped with their counterparts, meaning that the generation quality and text-video alignment will continue to improve i with more capable LLMs and layout-conditioned video generators. In Fig. 5, we show that our -SL-ggounded video generator can generate realistic videos represented by the DSLs in Fig. 4.

---

Published as a conference paper at ICLR 2024
Table I: Evaluation of the generated DSLs. Our LLM spatiotemporal planner is able to generate
layouts that align well with the spatiotemporal dynamics requirements in several types of prompts.
Method
Numeracy
Attribution
Visibility
Dynamics
Sequential
Average
Retrieval-basee
20%
12%
28%
8%
0%
14%
LVD (GPT-3.5, ours)
100 %
100 %
100 %
71%
16%
77%
LVD (GPT-4, ours)
100%
100
100 %
100 %
88%
98%
+ retrieval examples
98%
100%
100 %
96%
85%
96%
Table 2: Detection-based evaluation of the generated videos. LVD follows the spatiotemporal
keywords in the prompts better than ModelScope, the diffusion model that it uses under the hood.
Generated videos have resolution of 256 x 256. Additional 512 x 512 results are in Appendix A.6.
Method
LLM LLM-grounder grounded
Numeracy Attribution Visibility Dynamics Sequential
Average
ModelScope
x 32%
54%
8%
21%
0%
23.0%
Retrieval-basee
15%
15%
11%
9%
0%
9. 9.7%
LVD (GPT-3.5, ours)
/
52 %
79%
64 %
37%
2%
46.4%
LVD (GPT-4, ours)
/
41 % 64%
55%
51 %
38%
49.4%
EVALUATION
Setup. We select ModelScope (Wang et al., 2023) as our base video diffusion model and apply LVD on top of it. In addition to comparing LVD with the ModelScope baseline without DSL guidance, we also evaluate LVD against a retrieval-based baseline. Given a text prompt, this baseline performs nearest neighbor retrieval from large-scale video captioning dataset WebVid-2NM based on the similarity between the prompt and the videos with Frozen-In- In-Tiee (Bain et al., 2021), and runs GLIP (Li al., 2022) to extract the bounding boxes for the entities in the captions of the retrieved videos and tracks the bounding boxes with SORT (Bewley et al., 2016) to get the DSLs.
Benchmarks. To evaluate the alignment of the generated DSLs and videos with text prompts, we propose a benchmark comprising five tasks: generative numeracy, attribute binding, visibility, spatial dynamics, and sequential actions. Each task contains 100 programmatically generated prompts that can be verified given DSL generations or detected bounding boxes using a rule-based metric, with 500 prompts total. We generate two videos for each prompt with different random seeds, leading to 1000 video generations per evaluation. We then report the success rate of how many videos follow the text prompts. Furthermore, we adopt the benchmarks (UCF-101 (Soomro et al., 2012) and MSR-VTT (Xu et al., 2016)) as referenced in literature (Blattmann al., 2023; Singer et al., 2022) to evaluate the quality of video generation. We also conduct an evaluator-bass assessment as well. We refer readers to the appendix for details about our evaluation setup and proposed benchmarks.
5.1 EVALUATION OF GENERATED DSLS
We evaluate the alignment between the DSLs generated by an LLM and the text prompts without the final video generation. The results are presented in Table 1. We first compare with a baseline model that directly outputs the layout of the closest retrieved video as the DSL without an LLM involved. Its low performance indicates that nearest neighbor retrieval based on text-video similarity often fails to locate video with matching spatiotemporal layouts. In contrast, our method generates DSLS that align much better with the query prompts, where using GPT-3.5 gives an average accuracy of 77% and using GPT-4 can achieve remarkable accuracy of 98%. These results highlight that current text-only LLMs can generate reasonable spatiotemporal layouts based solely on a text prompt, which is hard to achieve with sttrevval-basd solution. Notably, only three in-context examples in Fig. 3 are used throughout the evaluation. To determine if incorporating examples from real videos enhances the performance, we experimented with adding the samples from retrieval. However, these additional examples lead to similar performance, underscoring the LLM's few-shot generalization capabilities. We also noticed that GPT-3.5 struggles with sequential tasks. While it typically begins correctly, it either skips one of the two actions or executes them simultaneously.
5.2 EVALUATION OF GENERATED VIDEOS
Evaluation of video-text alignment. We now assess whether the generated videos adhere to the spatial layouts and temporal dynamics described in the text prompts. For evaluation, we also use the rule-based method, with boxes from detector OWL-ViT (Minderer et al., 2022). With results presented in Table 2, ModelScope underperforms in most of the scenarios and is barely able to
7

---

Published as a conference paper at ICLR 2024
Table 3: A Ablations on the energy function used to condition the video diffusion model on the layouts. We use the same set of layouts generated by the LLM (GPT-4) in the ablation. Neither Erati proposed in Chen et al. (2023a) nor Essner proposed in Boxdiff (Xie et al., 2023) help improve the text-video alignment, whereas simply using Fropk achieves good text-video alignment. Adding F.OM can further boost the text- alignment for sequential tasks. Generated videos are of 256 x 256.
Empk
Brus
Ecconer
Bcom
Numeracy
Attribution
Visibility
Dynamies
Sequential
Average
x
/
x
x
44%
53%
11%
54%
21%
36.4%
/
X
V
38%
60%
53%
48%
31%
45.6%
/
X
x
39%
65%
54%
52%
23%
46.3%
X
X
41%
64% %
55 %
51%
38%
49.4%
Table 4: Video quality evaluation. Lower FVD scores are better. indicates our replication of the results. LVD is able to generate videos with higher quality than ModelScope on both benchmarks.
Method
FVD@ UCF-101 (L)
FVD@MSR-VTT (L)
CogVideo (Chinese) (Hong et al., 2022)
751
CogVideo (English) (Hong al., 2022)
702
1294
MagieVideo (Zhou al., 2022)
699
1290
Make- A- Video (Singer et al., 2022)
367
VMdeoLDM (Blattmann et al., 2023)
551
ModelScope (Wang et al., 2023)
550
ModelScope!
984
590
ModelScope LVD (ours)
828
565
generate videos in two of the five tasks. In addition, videos from the retrieved layouts have worse text-video alignment performance when compared to the baseline method. On the other hand, LVD significantly outperforms both baselines by a large margin, suggesting the improved video-text alignment of the proposed LVD pipeline. Interestingly, the layouts generated by GPT-3.5 lead to better text-video alignment for three tasks, as they have less motion in general and are easier to generate videos from. The gap between the results in Table and Table 2 still indicates that the bottleneck is in the DSL-grounded video generation stage.
Ablations on the energy function. E. Boxdiff (Xie et al., 2023) proposes a corner term Evumer with Eup and Chen et al. (2023a) proposes to use ratio-based energy function E As shown in Table 3, we found that using only Exop achieves good text-video alignment already. We also recommend adding ECOM when sequential tasks are entailed, as it offers significant boosts.
Evaluation of video quality. Since our primary objective is to generate videos that align with intricate text prompts without additional training. improving video quality is not the key objective to demonstrate the effectiveness of our method. We evaluate the vidco quality through FVD score on UCF-101 and MSR-VTT. The FVD score measures if the generated videos have the same distribution as the dataset. Both benchmarks feature videos of complex human actions or real-world motion. Since our method is agnostic to the underlying models and applicable to different text-to to-vi diffusion models, we mostly compare with ModelScope (Wang et al., 2023), the diffusion model that under the hood. Both LVD and our replicated ModelScope generate 256 256 videos. As shown in Table 4, LVD improves the generation quality compared to the ModelScope bascline on both datasets. suggesting our method can produce actions and motions that are visually more similar to real-world videos. ModelScope and LVD surpass models such as Zhou et al. (2022) on MSR-VTT, while being worse on UCF-101, possibly due to different training data.
Visualizations. In Fig. 7, we show our video generation results along with baseline Modelscope. LVD consistently outperforms the base model in terms of generating the correct number of objects, managing the introduction of objects in the middle of the video, and following the sequential directions specified in the prompt. In addition, LVD also able to handle descriptions of multiple objects well. For example, in Fig. 7(b), while ModelScope assigns the color word for the dog to a ball, LVD precisely captures both the blue dog and the brown ball in the generation.
Evaluator- baasdd assessment. To further validate the quality of videos generated with LVD, we conducted an calluatto--ased assessment. We asked two questions to 10 participants about each of the 20 pairs of videos randomly selected from our benchmark: "Which video aligns better with the text prompt?" and "Which video has better quality?". Each set is composed of two videos, one generated by baseline and the other by LVD. We randomly shuffled the videos within each set. For
8

---

Published as a conference paper at ICLR 2024
Baseline (Modelscope)
LVD (Ours)
Numeracy
(a) scene with one moving car
Attribute Binding
(b) scene with brown and a blue dog
Visibility
(c) A scene in which a walking dog appcars only in the sccond half of the vidco
Spatial Dynamics
with moving from of ball right
Sequential
(e) top-down viewed which bird initially the lower right of the first moves the lower left of the seene and then moves to the upper left of the seene.
Figure 7: Despite not aiming for each individual aspect of the quality improvement, LVD generates videos that align much better with the input prompts several tasks that require numerical, temporal, and 3D spatial reasoning. Best viewed when zoomed in.
Table 5: Assessment based human evaluators. Despite sharing the same dissasion weights the baseline model (Wang et al., 2023), our method is preferred significantly more often.
Question
Baseline
LVD
Similar
Which video aligns better with the text prompt? Which
3.2%
85.8%
12.6%
better quality?
14.5%
66.0%
19.5%
each question, the options are "Video #1". "Video #2". and "Similar". We show the averaged score in Table 5. LVD holds substantial edge over the baselines. In 96.8% of alignment cases and 85.5% of visual quality cases, the evaluators indicate that L.D outperforms or least on par with the baseline model. This highlights that LVD) brings about significant enhancement in the performance of baseline video diffusion models in both the alignment and visual quality.
DISCUSSION AND FUTURE WORK
In this work, propose LLM-grounded Video Diffusion (LVD) to enhance the capabilities of text- to-video diffusion models to handle complex prompts without any LLM or diffusion model parameter updates. Although LVD shows substantial enhancement over prior methods, there still potential for further improvement. For example, as shown in the Appendix A.5, LVD still inherits limitations from the base model when comes synthesizing objects or artistic styles it struggles with. We anticipate that techniques like LoRA fine-tuning (Hu et al., 2021) could help mitigate such challenges and leave this exploration for future research. Our aim for this work is 1) to demonstrate LLM's surprising spatiotemporal modeling capability and 2) to show that LLM-grounded representation. such as DSLs, can be leveraged to improve the alignment for text-to-vides generation, with potential extensions to control finer-gra meainer properties, including human poses, to model complex dynamics. Our DSL-to-video stage still simple, and thus more advanced DSL-conditioned video generator may further improve the precision of control and video quality in future work.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We mainly apply and evaluate our method on open-sourced model ModelScope (Wang et al., 2023), allowing the reproduction of results in this work. We present our prompts for DSL generation in Appendix A.3. We present implementation and benchmarking details in Appendix A.4. We will also release code and benchmarks for reproducibility and future research.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 1728-1738, 2021.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE International Conference Image Processing (ICIP), pp- 3464 3468, 2016. doi: 10. 1..1109/CIP22201.75333000
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2023.
Tim Brooks, Janne Hellsten, Miik Aittala, Ting- Chun Wang, Timo Aila, Jaakko Lehtinen, Ming- Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 35:31769-31881, 2022.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrans for video prediction. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 7608-7617,
2019.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint 2rXiv:2304.03373 2023a.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023b.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International conference on machine learning, pp. 1174-1183. PMLR, 2018
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-877 87794, 2021.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffussion self- guidance for controllable image generation. arXiv preprint arXiv:2306.0098: 2023.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 2023.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1166840-6511. 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230) 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-2288 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 15868, 2022.

---

Published as a conference paper at ICLR 2024
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint WrXiv:2106.09685, 2021.
Dong Huk Park, Grace Luo, Clayton Toste, Samanch Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. arXiv e-prints, pp. arXiv-22131 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Tex:2viec--zero; Text-to-image diffusion models are zero-shot video generators. arXiv preprint 11Xii:23033.34339 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
10965-10975, 2022.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 225 1-22221,,,, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm-guided planning. arXiv preprint arXiv:2309. /5091, 2023.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 578777 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095. 2022b.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
OpenAl. Language models like gpt-3, https: //pp:ai./pp/./////////ee
www.///eeercc///eeeeeoooo /gptt 3, 2020. OpenAI. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint 2rX.1:2330.05527,,22 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 06000.006........ 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684- 10695, 2022.
12

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assise 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning -ext-to-mmaee diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14799 2022.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11), 2012.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104. 15069, 2021.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- clscope text-to-vides video technical report. arXiv preprint arXiv:2308. 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv: 20X6v22206.06622, 2022.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2/04. 14806, 2021.
Jay Zhangjie Wu, Yixiao Ge. Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for tixt- generation. arXiv preprint arXiv:2212.11556 2022.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. arXiv preprint axXiv:2307.10816 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5288-5296, 2016.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe 7hu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.110188 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 REASONING STATEMENTS FROM GPT-4
As described in Section 3, we ask the model to output a reasoning statement for interpretability. We present examples of the reasoning statements for DSL generation in Table 6. The corresponding DSLs and generated videos are presented in Fig. 4, Fig. I, and Fig. 5. The first three statements are from ChatGPT web interface. The last four statements are from OpenAl API. All statements are from GPT-4.
These statements provide useful insights about why GPT-4 generates these DSLs in specific ways. For example, the first reasoning statement indicates that GPT-4 takes the flying pattern of birds into account. The second statement shows that GPT-4 applys the buoyancy for the wooden barrel. GPT-4 also considers the reaction of the cat in the third statement. For the fourth statement, GPT-4 demonstrates its understanding that the rock, unlike the ball provided in the in-context example, does not bounce (note that our y-coordinate is top-to-bottom as introduced in the text instructions, so is correct that the y-coordinate increases).
Interestingly, we observe that GPT-4 with ChatGPT web interface often outputs longer reasoning statements compared GPT-4 called with APIs. We hypothesize that GPT-4 used in ChatGPT interface has prompts encouraging more explanations rather than following the patterns in the examples.
A.2 ADDITIONAL QUALITATIVE VISCALIZATIONS
We present additional qualitative visualizations of LVD in Fig. 8.
Handling ambiguous prompts. As shown in Fig. 8(a), our LVD can successfully generate videos for "a walking dog", which is prompt that has ambiguity, the walking direction of the dog is not given. Despite the missing information, the LLM is still able to make a reasonable guess for the walking direction in DSL generation, leading reasonable video generation. Furthermore, this assumption is included in the reasoning statement:
Reasoning: dog walking would have a smooth, continuous motion across the frame. Assuming it is walking from left to right, its x-coordinate should gradually increase. The dog's s y-coordinate should remain relatively consistent, with minor fluctuations to represent the natural motion walking. The size of the bounding box should remain constant since the dog's size doesn't t change.
This explicitly indicates that the LLM can detect the missing information and make reasonable assumptions for generation.
Handling overlapping boxes. Our DSL-to-video stage can process highly overlapping boxes. In addition to video frames, we also show the dynamic scene layouts generated by the LLM in dashed boxes (white for the raccoon and blue for the barrel) Fig. 8(b). Although the box of the raccoon overlaps about half of the barrel, the generation of the raccoon the barrel is still realistic and the motion still makes sense.
Handling dynamic background. ModelScope often prefers to generate background with little motion. Although this preference is inherited by our training-free method, our model still works when asked to generate background that changes significantly. For example, in Fig. 8(c) query the model to generate a brown bear taking a shower in a lake when the lightning strikes, and our method generates both successfully, with significant variations on the background. Note that the model generates the effect of rain when only lightning is mentioned in the prompt, which is a reasonable assumption inherited from our base model.
A.3 OUR PROMPTS AND IN-CNNEEXT EXAMPLES FOR DSL GENERATION
Table 10 presents the prompts that for DSL generation in the quantitative evaluation. To generate DSLs, we replace the "User Text Prompt for DSL Generation" in the template with the text query that we would like to generate DSLs from and pass the prompt an LLM. We use
14

---

Published as a conference paper at ICLR 2024
Table 6: Reasoning statements from GPT-4.
Query Prompt
LLM Reasoning statements
A bird is flying from the left to the right
bird flying from left to right will have its x-coordinate steadily increasing. Birds tend to fly in somewhat leveled manner, but they can have slight vertical movements (up and down) due to flapping or gliding. Civen the short timeframe of six frames, we'll assume the bird has a slight upward movement initially and then glide downward, reflecting a flap-glide pattern typical for many birds.
A wooden barrel is drifting on river
wooden barrel drifting on a river will follow the flow direction, typically moving from left to right or right to left. The barrel, being buoyant, will remain on the surface and will be subjected to the river's currents. As such, the x-coordinate of the barrel will change, indicating its drift, but its y-coordinate will remain largely consistent since it is floating on the surface. For simplicity, let's assume the river flows from left to right.
A raccoon is walking towards
The raccoon starts from one side and moves closer to the cat. As both the raccoon and the cat are made larger, their bounding boxes will be more extensive. The raccoon' 8 x-coordinate should increase and it should get closer to the x-coordinate of the cat. The cat might be stationary at first, but as the raccoon approaches, it might move backward a bit due to its natural instinct maintain distance. Both the raccoon and the cat will have a constant y-coordinate because they are on the same ground level.
cat
A rock thrown out from the
rock is thrown from the right, so its x-coordinate should decrease. Due to gravity, its y-coordinate should increase, and the speed should be faster in later frames until it hits the ground. Unlike a ball, a rock does not bounce back when it hits the ground.
right
A car viewed from the side is
the car is driving forward, its x x-coordinate should increase over time. The y-ccordinate should remain constant as the car is moving on flat surface. The size of the car should remain constant as it is viewed from the side.
driving forward
A car viewed from the top is
The car is moving forward, so its y-coordinate should decrease. The x-coordinate should remain the sane as there is no lateral movement. The car's size remains the same as the camera is stationary.
driving forward
A car viewed from its back is driving forward
As the car is driving forward and the camera is viewing from the back, the will appear to get smaller as it moves away from the camera. The x-coordinate and y-coordinate will remain the same, but the width and height the bounding box will decrease to represent the car moving away.
15

---

Published as a conference paper at ICLR 2024
(4) walking dog
(b) barrel flaating
loking lake, lightning strikes
Figure 8: Additional visualizations of LVD. (a) Given prompt that has ambiguity (the walking direction of the dog is not given), LLM is still able to make a reasonable guess and often includes the assumption in the reasoning statement, leading reasonable generation. (b) Our DSL-to-video stage is able to process highly overlapping boxes (outlined in the dashed boxes): the box of the raccoon overlaps about half of the barrel, yet the generation of the raccoon on the barrel is still realistic and not abrupt. (c) Although our base video diffusion model ModelScope often generates static background, which inherited by our training- method, our model still works when asked to generate background that changes significantly such lightning and raining. Note that the model generates raining when only lightning is mentioned the prompt, which reasonable assumption.
the chat completion API for both GPT-3.5 and GPT-4, For API calls, line 1-3 assigned with role "system". Each caption line in the in-context examples and user query are assigned with role "user". The example outputs are assigned with role "assistant". The final "Reasoning" line is omitted. For querying through ChatGPT web interface we do not distinguish the roles of different lines. Instead, we merge all the lines into one message. The benchmarks are conducted by querying the LLMS through the API calls. The visualizations in Fig. are conducted by querying the GPT-4 through the ChatGPT web interface (with horizontal videos generated by Zeroscope a model fine-tuned from ModelScope optimized for horizontal video generation).
A.4 DETAILS ABOUT OUR SETUP AND BENCHMARKS
Setup. Since our method is training-free we introduce our inference setup in this section. For DSL-grounded video generation, we use DPMSolverMultiStep scheduler (Lu et al., 2022a:b) to denoise 40 steps for each generation. We use the same hyperparams as the baselines, except that we employ DSL guidance. For ModelScope (Wang al., 2023), we generate videos of 16 frames. For qualitative assessment in our benchmarks, we generate videos with resolution of both 256 x 256 and 512 512 for comprehensive comparisons. For FVD evaluation, we generate videos with resolution 256 x 256, following our baseline. For maaluator-based assessment, we generate videos with resolution 512 x 512 with only Epot For Zeroscope examples in visualizations, we generate 576 x 320 vidcos of 24 frames. For DSL guidance, we scale energy function We perform DSL guidance 5 times per step only in the first 10 steps to allow the model to freely adjust the details generated in the later steps. We apply background weight of 4.0 and foreground weight of 1.0 to each of the terms in the energy function, respectively. The A in Topk was selected by counting 75% of the positions in the foreground/background in the corresponding term, inspired by
factor of 5.
'nttps://plat http://ww..ppppaaaoooo/////////a Phttps::/chat.oppen com// Phttps://huggiinggaac co/ httpe//eeeereeeeeerrrrrr........
16

---

Published as a conference paper at ICLR 2024
previous work for image generation (Xie et al., 2023). ECOM is weighted by 0.03 and added to Exop to form the final energy function E. The energy terms for each object, frame, and cross-attention layers averaged. The "learning rate" for the gradient descent follows vI â; for each denoising step I, where the notations are introduced in Dhariwal & Nichol (2021).
Proposed benchmark. To evaluate the alignment of the generated DSLs and videos with text prompts, we propose benchmark comprising five tasks: generative numeracy. attribute binding, visi- bility, spatial dynamics, and sequential actions. Each task contains 100 programmatically generated prompts that can be verified given DSL generations detected bounding boxes using rule-based metric, with 500 text prompts in total. We then generate videos for each text prompt, resulting in 1000 videos generated. For LLM-grounder experiments, generate one layout per prompt and generate two vidcos per layout.
The generative numeracy prompts are generated from template A realistic lively video of a scene with [number] [ob ject type], where object types are sampled from [' car', 'cat', 'bird', 'ball', ' '.
The attribute binding prompts are generated from template A realistic lively video of scene with [color1] [object typel] and [color2] [object type2] where colors are sampled from ['red', 'orange', 'yellow', 'green', 'blue', 'purple', 'pink', 'brown', 'black', 'white', 'gray').
The visibility prompts are generated from template A realistic lively video of a scene in which ch [object type] appears only in the [first/second] half of the video.
The spatial dynamies prompts are generated from template A realistic lively video scene with [object type] moving from the [left/right] to the [right/left] and template A realistic lively video a scene with [object typel] moving from the [left/right] of a [object type2] to its (right/left).
The sequential actions prompts are generated from template A realistic lively video of a scene in which a [object typel] initially on the [location]] of the scene. It first moves to the [location2] of the scene and then moves to the [location3] of the scene. The three locations are sampled from [('lower left', 'lower right', 'upper right'), ('lower left', uupper left', 'upper right(), ('lower right', 'lower left', 'upper left'), ('lower right', 'upper right', 'upper left')).
UCF-101 and MSR-VTT benchmarks. For UCF- 101, we generate 10k videos, each of 16 frames and 256 x 256 resolution, with either the baseline or our method and using captions randomly sampled from UCF-101 data, and evaluate FVD between the 10k generated videos and videos in UCF-101 following (Blattmann et al., 2023; Singer al., 2022). We first generate template prompt for each class, use LLM to generate multiple sets of layouts for each prompt so that there are in total 10k video layouts from all the prompts following the class distribution of the training set. We then calculate the FVD score with the training set of UCF- 101. The video resolution is 256 256 and each video contains 16 frames. For evaluation on MSR-VTT, we use LLM to generate layout for each caption of each video in the test set, which contains 2990 videos. We then randomly select 2048 layouts, generate videos for each layout, and calculate the FVD score with the test set videos.
A.5 FAILURE CASES
As shown in Fig. 10 (left), the base model (Wang et al., 2023) does not generate high-quality cartoons for "an apple moving from the left to the right", with the apples in weird shapes as well as random textures in the background. Although our model is able guide the motion of the apple generation with DSL- OSL-guidance, shown in Fig. 10 (right), our model still inherits the other problems that are present in the baseline generation. Techniques that can be used for enhancing the ability for Nomain-speciffe generation such as performing LoRA fine-tuning (Hu et al., 2021) the diffusion models may alleviate such issues. Since the scope of this work does not include fine-tuning diffusion models for higher quality generation, leave the investigation future work.
17

---

Published as a conference paper at ICLR 2024
Baseline (Modelscope)
LVD (Ours)
Figure 10: One failure case: A cartoon of an apple moving from the left to the right. Since LVD is wraining-free, it does not improve the quality on domains/styles that the base model is not good at. Although LVD correctly generates a video that is aligned with the prompt, neither the baseline or LVD generates background that fits the apple in the foreground.
LVD (Ours)
Figure 11: Another failure case: A wooden barrel is drifting on a river. Since LVD controls the cross-altention map between the image and the text, the DSL guidance sometimes leads to undesirable generations during the energy minimization process. In this failure case, the model attempts to hide the barrel on the left with leaves and pop a barrel up from the water to reach the desired energy minimization effect, while the DSL means to move the barrel to the right.
Another failure case is shown in Fig. 11, in which the DSL grounding process actually minimizes the energy function an unexpected way. Instead moving the barrel to the right in order to render the drifting effect, the model attempts to hide the barrel on the left with leaves and pop barrel up from the water to reach the desired energy minimization effect the DSL grounding process. Reducing the ambiguity caused by sess-atteniion-bseee control is needed to resolve this problem. We leave its investigation as a future direction.
We also show failure case of LLM generating DSLs. From Table we show taht DSLs generated by GPT4 have 100% accuracy in all categories except for sequential movements. This means LLM can still fail when generating layouts for long-horizon dynamics. Here we show a failure case in Figure 9. The prompt is A ball first moves from lower left upper left of the scene, then moves to upper right of the scene. We can see that instead of moving top left, the generated box only moves slightly above the middle of the frame box, which is inconsistent with the prompt. Since LLM queries are independent, we found that this accounts for most of the fluctuations in the performance in our layout generation stage.
Frans
Framee
ball
Figure 9: Failure case of L.MM-generated DSL.
A.6 ADDITIONAL QUALITATIVE RESULTS WITH HIGHER VIDEO RESOLUTION
We also present additional results on our proposed benchmark in Table 7 with resolution 512 x 512. We found that as the resolution increases, the performance gaps between our method and baselines are more significant, which is likely because it is casier to control the motion with high-resolution latents and the detection boxes are more accurate and have lower levels of noise in evaluation. Our full method, with ECaM. surpasses the baseline without guidance by large margin. We also ablated other choices of energy functions, and show that Eratio. proposed in Chen et al. (2023a), leads to degradation in performance. Adding Ecomne proposed in Xie al. (2023) leads to similar performance in terms of text-videe alignment. Since different types of energy terms are involved, we also experimented with giving more weights Ecomer. However, the performance fluctuates between 59. 59.2% and 60.9% as we give 1x to 10x more weights to Egomer- We hypothesize that the spatial normalization term in Foomerr leads to its insignificant change in performance, and thus we also implemented version
18

---

Published as a conference paper at ICLR 2024
Table Additional results and ablations on videos of higher resolution. Generated videos are of 512 x 512. indicates unnormed corner energy function.
Empk
Emaoo
Exsener
EcaM
Numeracy Attribution
Visibility Dynamics Sequential Average
No guidance
X
X
X
X
66%
19%
0%
18.6%
With only Fratio
X
/
X
X
29%
75%
13%
49%
23%
37.5%
With Everer
/
X
V
X
64 %
94%
47%
66%
36%
60.9%
With Eloener
X
d
X
17%
86%
10%
36%
5%
30.6%
With only Exogle
X
X
X
62 %
92%
48%
65%
32%
59.6%
With ECOM
X
X
V
56%
92%
62%
75%
52%
67.2%
Table Ablations the number of in-context examples. We found that in-context examples are sufficient for stable performance. i: We observe instabilities when we only use one example with GPT-3.5, where two of the three runs could not finish due invalid DSLs generated.
Number of in-context examples
(dcfault) 5
DSL evaluation accuracy (GPT-4) 96.2% 1.99%
97.3% 96.6%
Table Ablations on the number DSL guidance steps between two denoising steps in LVD.
We use 5 DSL guidance steps as the performance saturates with more DSL guidance steps.
DSL guidance steps
3 5 (default)
7
Detection based evaluation accuracy 35.3%
46.9%
49.4%
49.4%
of the Feomeer that removed the spatial normalization. However, we observe that this significantly degrades the text-video alignment compared to using only Eiop
A.7 ADDITIONAL ABLATION EXPERIMENTS
A.7.1 ABLATIONS THE L.M-GROUNDED DSL GENERATION STAGE
In-context examples. We test prompts with 1, 3, and in-context examples. We first test performance of the LLM-grounded DSL generator with each of the 3 in-context examples that composed. As shown Table we find that GPT-4 could still generate nigh-quality DSLs with only 1 in-context example, despite slightly degraded performance and increased performance fluctuation. However, for GPT-3.5, we observe instabilities that two of three runs could not finish the whole benchmark within three attempts for each prompt, mainly because they not able to generate DSLs that can be parsed (e.g., some DSLs do not have boxes for all the frames). Therefore, we still recommend least three examples. Furthermore, we composed two additional in-context examples and evaluated the quality of the DSLs with 1) randomly chosen DSLs from the 5 DSLs, and 2) all DSLs in-context examples. We find that our method is stable with different sets of in-context examples when 3 examples are given. In addition, we did not find additional benefits from adding more DSLs. We therefore keep 3 DSLS as our default choice.
A.7.2 ABLATIONS ON THE DSL-GROUNDED VIDEO GENERATION STAGE
DSL guidance steps. We test 1, 3, 5, and 7 steps of DSL guidance between denoising steps, respectively. The results are shown in Table 9. We can see that the average accuracy improves when adding more DSL guidance steps from to 5 steps, and the performance plateaus when we use more than 5 steps. This means 5 steps is enough and probably the best choice considering the trade-off between performance and efficiency. We therefore 5 DSL guidance steps.
19

---

Published as a conference paper at ICLR 2024
Table Our prompt for DSL generation. When using API calls, line 1-3 presented to GPT-4 with role "system". Each caption line in the in-context examples and user query assigned with role "user". The example outputs are assigned with role "assistant". These lines are merged into one message when querying GPT-4 through the ChatGPT web interface.
You are an intelligent bounding box generator for videos. You don't t need to generate the videos themselves but need to generate the bounding boxes. I will provide you with caption for a video with six frames, with two frames per second. Your task is to generate a list of realistic bounding boxes for the objects mentioned in the caption for each frame well
background keyword. The video frames are of size 512x512. The top-left corner has coordinates [0, 0]. The bottom-right corner has coordinates 1512, 512) The bounding boxes should not overlap or go beyond the frame boundaries.
Each frame should be represented
'[('id'; unique object identifier incrementing from 0, 'aaee' object name, 'box'; [box top-left x-coordinate, box top-left y-coordinate, box width, box height]), ... '. Each box should not include more than one object. Your generated frames must encapsulate the whole scenario depicted by the caption. Assume objects move and interact based on real-world physics, considering aspects such as gravity and elasticity. Assume the camera follows perspective geometry. Boxes for an object should have the same id across the frames, even if the object may disappear and reappear. If needed, you can make reasonable guesses. Provide de conci reasoning statement that is not longer than a few sentences before each generation. Refer to the examples below for the desi red format. Never use markdown or other formats not in the examples. Do not start each frame with t-1, Do not include comments in your response.
[In-contes examples]
Caption: (User Text Prompt for DSL Generation)
Reasoning:
20

---

Published as a conference paper at ICLR 2024
Table 11: Our in-context examples for DSL generation. We use these three in-context examples that we use throughout our work unless stated otherwise.
Caption: A woman walking from the left to the right and a man jumping on the right in room
Reasoning: woman is walking from the left to the right her x-coordinate should increase with her y-coordinate fixed. man is jumping on the right so his x-coordinate should be large, and his y-coordinate should first decrease (upward movement) and then increase (downward movement due to gravity).
Frame 1: [('id'; 0, 'name'; 'walking woman', 'box': [0, 270, 120, 2001 h,
(rid':
'name': 'junping man', "box': [380, 290, 120, 180]]]
Frame 2: [('id': 0, 'name': : 'walking woman', 'box': [50, 270, 120,
2001,, h, ('id': 1, 'name' 'jumping man', "box': [380, 205, 120, 2001
Prame 3: [("id': 0, 'name': 'walking woman', "oox [100, 270, 120,
2001), (fid': 1, 'name' "jumping man', "bo'': [380, 175, 120, 2001H1
Prame 4: [(rid': 0, 'name': 'walking woman', "box': [150, 270, 120,
200] h, ['id': 1, 'name' "umppin man', 'box': 1380, 175, 120, 2001
Frame 5: [['id': 0,
2001 ('id': 'name' 'walking woman', 'box' [200, 270, 120,
'name' jumping man', 'box': 1380, 205, 120, 2001 'name' 'walking woman', "box': [250, 270, 120,
Frame [('id'; 0,
2001 ('id': 1, namee: jjupping man', 'box' [380, 290, 120, 1801
Background keyword: room
10
Caption: A red ball is thrown from the left the right in garden 12 Reasoning: ball thrown from the left to the right, so its x-coordinate should increase. Due to gravity, its y-coordinate should increase, and the speed should be faster in later frames
until it hits the ground. Due to its elasticity, the ball bounces back when it hits the ground.
13 Prame 1: [(rid': 0, 'name': 'red ball', 'box': [0, 206, 50, 50]]]
14 Prrae 2:
[['id': 0, 'name': 'red ball', 'box': [80, 246, 50, 50]]]
15 Frame 3: [['id' 0, 'name" 'red ball', "box': [160, 326, 50, 50]]]
Frame [['id':
'name': 'red ball', "box [240, 446, 50, 50]]]
17 Frame [f'id': 0, 'name' : 'red ball', box' [320, 366, 50, 50]]]
18 Frame 6: [('id': 0, 'nmme': 'red ball', box': [400, 446, 50, 50]
19 Background keyword: garden
Caption: The camera is moving away from a painting
22 Reasoning: Due to perspective gecmetry, the painting will be smaller in later timesteps as the distance between the camera and the object is larger.
Frame [['id':
'name' 'painting', 'box' 1156, 181, 200, 150
24 Frame 2:
[['id"; 0, "name's 'painting', "box' 1166, 189, 180, 135
35 Frame 3: [('id'; 0, 'name'
'painting', 'box': [176, 196, 160, 120]
26 F'rame 4: [['id': 0, 'name": : 'painting', "box': [186, 204, 140, 105]:
Frame [['id": "name": 'painting', "box': [196, 211, 120, 90]11
Frame 5: [('id'; 0,
'name": 'painting', "bx' [206, 219, 100, 75])]
29 Background keyword: room
21


Reviews:
[{"review_id": "DkaqDWnnxV", "reviewer": "Reviewer_6SgA", "summary": "The paper introduces a novel approach to text-conditioned video generation that seeks to address the limitations of current models, which struggle with complex spatiotemporal prompts and often produce videos with restricted or incorrect motion patterns. The key contribution is the LLM-grounded Video Diffusion (LVD) model that separates the video generation task into two steps: (1) using a Large Language Model (LLM) to generate dynamic scene layouts (DSLs) from text inputs, and (2) using these layouts to guide a diffusion model in generating the video. The approach is described as training-free and can be integrated with existing video diffusion models that allow for classifier guidance. Moreover, they introduce a benchmark for evaluating the alignment between input prompts and generated videos.", "questions": "- Can the authors elaborate on how the model performs with ambiguous or complex text prompts that might yield multiple valid interpretations in terms of spatial and temporal dynamics?\n- Could the authors discuss any observed limitations or failure modes of the LVD approach, particularly in cases where the LLM might generate less accurate DSLs?\n- (Minor point) Typographical error in Section 4, second paragraph. The sentence in question should indeed conclude with \"feature map\" instead of \"feature ma.\" A revision is recommended for accuracy and clarity.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposal of a training-free approach presents a pipeline that is well-suited for the application of off-the-shelf LLMs and diffusion models. Its simplicity yet effectiveness stands out as a notable strength.\n- The discovery that LLMs can generate spatiotemporal layouts from text with only a limited number of in-context examples is noteworthy. It highlights the potential for a straightforward integration of LLM reasoning into text-to-video tasks.", "weaknesses": "- The idea of guidance via energy functions and cross-attention maps seems to be basically derived from BoxDiff (Xie et al., 2023;) and Chen et al. 2023a;. It is unclear how much of this work is based on previous research and how much is new. Since they are dealing with video generation using layouts, it would have been nice to see the authors' contribution in extending to the temporal axis, but this is not evident, which is disappointing.\n- I am concerned that the scale of the sample size for the proposed DSL benchmark may be too small to conduct a sufficiently robust evaluation.\n- The paper's contribution appears to lack novelty. There is existing work in text-to-image generation that has already established the capability of LLMs to create layouts, and this research seems to merely extend that to assess whether the same capability applies to temporal understanding. I didn't perceive any novel ideas stemming from the temporal aspect of the problem that would distinguish this work significantly from its predecessors.\n- The paper seems to lack a detailed analysis or ablation studies concerning the prompts given to the LLM for generating Dynamic Scene Layouts (DSLs). Such investigations are crucial to understand how different prompts affect the LLM's output and the subsequent quality of the video generation. Further exploration in this area could significantly bolster the robustness of the presented approach.\n- The paper's current framework could indeed benefit from additional ablation studies or analytical experiments to demonstrate the effectiveness of using DSLs for training-free guidance of text-to-video diffusion models. Moreover, a theoretical explanation of why this particular approach is effective would be valuable. It's important for the research to not only present the method but also to thoroughly validate and explain why certain choices were made and how they contribute to the overall performance and reliability of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbIDQ3ilfA", "reviewer": "Reviewer_34cZ", "summary": "Grounded Text-to-image generation has been studied by several papers recently. However, text-to-video geneartion with layout control is still unexplored. This paper tackles this task by proposing a training-free method by adding layout information through adjusting the attention maps of the diffusion UNet. Speficically, this paper first utilizes LLMs (GPT-4) to generate a multi-frame object layouts, then designs a layout-grounded video generator that encourages the cross-attention map to concentrate more on the bounding box areas. Extensive experiments for spatiotemporal dynamics evaluation have demonstrated the effectiveness of the proposed method.", "questions": "- Could the authors provide some reasoning why they report video-text similarity metric in Bain et al., 2021? It would be nice to also report CLIPScore, since its widely reported in other text-to-video generation baseilnes.\n\n- The examples provided in the paper are with non-overlapping bounding boxes. Will the proposed method work well with overlapping layouts?\n\n- If there are multiple objects, is the final energy function summing over the energy function corresponding to each object?\n\n- It seems that the background areas of the generated images with proposed method are quite static (Fig1, 7, 8, 9). Is this because the model encourages static background, or becuase the visualized images happens to have relatively static background? \n\n- Based on my understanding, another concurrent work, VideoDirectorGPT [1], is also for text-to-video generation with layout guidance. Even though the technical routes are different from this paper, it would be nice to have some discussions and comparison in the related work section. \n\n[1] Lin, Han, et al. \"Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.\" arXiv preprint arXiv:2309.15091 (2023)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n\n- The proposed method is training-free, which avoid the need for costly training with image/video data.\n\n- Using LLM-generated layouts for videos is relatively unexplored. And it's natural to use the knowledge embedded in LLMs to general layouts for downstream video generation.", "weaknesses": "- Even though the proposed method is training free, it takes longer time during inference to generate videos due to the optimization steps needed for the energy function.\n\n- Training-free layout control already exists in previous literatures [1, 2]. Therefore, the design of the energy function and backward guidance is not that novel. \n\n- Ablation study of the model design is not given (e.g., number of DSL guidance steps, energy function design).\n\n\n[1] Chen, Minghao, Iro Laina, and Andrea Vedaldi. \"Training-free layout control with cross-attention guidance.\" arXiv preprint arXiv:2304.03373 (2023)\n\n[2] Lian, Long, et al. \"LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models.\" arXiv e-prints (2023): arXiv-2305", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1BDSDpzPWa", "reviewer": "Reviewer_ZTWf", "summary": "In this paper, the authors propose a new text-to-video generation pipeline called LLM-grounded Video Diffusion (LVD). In particular, it first uses LLM to generate the layouts of the video and then uses the generated layout to guide a pre-trained video diffusion model. The whole process does not update the weights of both the LLM and video diffusion model. Besides, the authors show that LLMs’ can generate spatiotemporal dynamics aligned with text prompts in a few-shot setting. Qualitative results and quantitative results show that LVD generates higher quality videos that also align more with text.", "questions": "- From Table 3, we can see that LVD improves the video quality. What causes the improvement?\n\n- Is LLM able to generate reliable layouts using text without direction information, such as “a walking dog”.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Overall, the paper is well-organized and easy to follow. The figures and tables are informative.\n\n- The finding that LLMs can generate good spatiotemporal dynamics with only three examples is interesting and well supported by the experiments. The exploration of physical properties contained in LLM is also inspiring and deserves further research. \n \n- The results generated by LVD are promising compared to the baseline, ModelScope.", "weaknesses": "- The idea of using LLM to generate layout is already explored in LayoutGPT (Feng et al., 2023) and LMD (Lian et al., 2023). LMD also adapts in a training-free manner. It is beneficial for the authors to include a more detailed comparison.\n\n- The technical contribution is limited. The first layout generation part is similar to LMD, and the second part is a straightforward application of other training-free layout-to-image methods in the video domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)