VideoDirectorGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Authors: Han Lin
Abhay Zala
Jaemin Cho
Mohit Bansal
Venue: COLM (2024)
Keywords: Text-to-Video Generation, Large Language Models, Layout-Guided Video Generation, Temporal Consistency, Multi-Scene Video Generation, Layout Control
TL;DR: We propose VideoDirectorGPT, a novel framework for consistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation.

Abstract: Recent text-to-video (T2V) generation methods have seen significant advancements. However, the majority of these works focus on producing short video clips of a single event (i.e., single-scene videos). Meanwhile, recent large language models (LLMs) have demonstrated their capability in generating layouts and programs to control downstream visual modules. This prompts an important question: can we leverage the knowledge embedded in these LLMs for temporally consistent long video generation? In this paper, we propose VideoDirectorGPT, a novel framework for consistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation. Specifically, given a single text prompt, we first ask our video planner LLM (GPT-4) to expand it into a ‘video plan’, which includes the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities. Next, guided by this video plan, our video generator, named Layout2Vid, has explicit control over spatial layouts and can maintain temporal consistency of entities across multiple scenes, while being trained only with image-level annotations. Our experiments demonstrate that our proposed VideoDirectorGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. Detailed ablation studies, including dynamic adjustment of layout control strength with an LLM and video generation with user-provided images, confirm the effectiveness of each component of our framework and its future potential.

Published as a conference paper at COLM 2024
VIDEODIRECTORGPT: Consistent Multi-Scene Video Generation via LLM-Guided Planning
Han Lin
Abhay Zala Jaemin Cho
Mohit Bansal
UNC Chapel Hill
[hanlincs, aszala, jmincho, mbansal êcs unc. http:://videddrrectorrp... git..b..ttuboo..
Abstract
Recent text-to-video (T2V) generation methods have seen significant ad- vancements. However, the majority of these works focus on producing short video clips of single event (i.e., single-scene videos). Meanwhile, recent large language models (LLMs) have demonstrated their capability in generating layouts and programs to control downstream visual modules. This prompts an important question: can we leverage the knowledge em- bedded in these LLMs for temporally consistent long video generation? In this paper, we propose VIDEODIRECTORGPT, novel framework for con- sistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation. Specifically, given a single text prompt, we first ask our video planner LLM (GPT-44 to expand it into a 'video plan', which includes the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities. Next, guided by this video plan, our video gen- erator, named Layout2Vid, has explicit control over spatial layouts and can maintain temporal consistency of entities across multiple scenes, while being trained only with image-level annotations. Our experiments demon- strate that our proposed VIDEODIRECTORGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. Detailed ablation studies, including dynamic adjustment of layout control strength with LLM and video generation with ussv-ddovided images, confirm the effectiveness of each component of our framework and its future potential.
1 Introduction
Text-to-video (T2V) generation has achieved rapid progress following the success of text- to-image (T2I) generation. Most works in T2V generation focus on producing short videos (e.g., 16 frames at 2fps) from the given text prompts (Wang et al., 2023b; He et al., 2022; Ho et al., 2022; Singer et al., 2023; Zhou et al., 2022). Recent studies on long video generation (Blattmann et al., 2023b; Yin et al., 2023; Villegas et al., 2023; He et al., 2023) aim at generating long videos of few minutes with holistic visual consistency. Although these works could generate longer videos, the generated videos often display the continuation or repetitive patterns of single action (e.g, driving car) instead of transitions and dynamics multiple changing actions/ events (e.g,, five steps about how to make caraway cakes). Meanwhile, large language models (LLMs) (Brown et al., 2020; OpenAl, 2023; Touvron et al., 2023a;b; Chowdhery et al., 2022) have demonstrated their capability in generating layouts and programs to control visual modules (Didac et al., 2023; Gupta & Kembhavi, 2023), especially image generation models (Cho et al., 2023b; Feng et al., 2023). This raises an interesting question: Can we leverage the knowledge embedded in these LLMs for planning consistent multi-scene video generation?
1

---

Published as a conference paper at COLM 2024
VideoDirectorGPT
Planner
Video
Scone Descriztion
Video Generator
Spattal-Tempon Blocks
Layne
Multi Scene Video
Prompt hungry cat 15 finding food
(GPT-4 +)
Endiles (hames layouts)
Video
(Layout2Vid)
Guided ZD Albention
Background
Plan
Consistency Grouping
Yext
Figure 1: VIDEODIRECTORG framework. First, we employ GPT-4 video planner to craft multi-component video plan. Next, we utilize Layout2Vid, a grounded video generation module, render multi-scene videos with layout and consistency control.
In this work, we introduce VIDEODIRECTORGPT, a novel framework for consistent multi- scene video generation. As illustrated in Fig. 1, VIDEODIRECTORGPP decomposes the T2V generation task into two stages: video planning and video generation. For the first video planning stage (see Fig. blue part), we employ an LLM to generate video plan, which is an overall plot of the video with multiple scenes, each consisting of a text description of the scene and entity names/ hames/layouts layouts, and a background. It also consists of consistency groupings of specific entities that re-appear across scenes. For the second video generation stage (see Fig. yellow part), we introduce Layout2Vid, a novel grounded video generation module that generates multi-scene videos from the video plan. Our framework provides the following strengths: (1) employing an LLM write video plan that guides the generation of videos with multiple scenes from a single text prompt, (2) layout control in video generation by only using image-level layout annotations, and (3) generation of visually consistent entities across multiple scenes.
In the first stage, video planning (Sec. 3.1), we employ an LLM (e-g., GPT-4 (OpenAI, 2023)) as a video planner to generate a video plan, a multi-component video script with multiple scenes to guide the downstream video synthesis process. Our video plan consists of four components: (1) multi-scene descriptions, (2) entities (names and their 2D bounding boxes), (3) background, and (4) consistency groupings (scene indices for each entity indicating where they should remain visually consistent). We generate the video plan in two steps by prompting an LLM with different in-context examples. In the first step, we expand a single text prompt into multi-step scene descriptions with an LLM, where each scene is described with a text description, a list of entities, and a background (see Fig. 2 blue part for details). We also prompt the LLM generate additional information for each entity (e.g., color, attire, etc.), and group together entities across frames and scenes, which will help guide consistency during the video generation stage. In the second step, we expand the detailed layouts of each scene with an LLM by generating the bounding boxes of the entities per frame, given the list of entities and scene description. This overall 'video plan' guides the downstream video generation.
In the second stage, video generation (Sec. 3.2), we introduce Layout2Vid, a grounded video generation module to render videos based on the generated video plan (see yellow part of Fig. 2). For the grounded video generation module, we build upon ModelScopeT22 (Wang et al., 2023b), an off-the-shelf T2V generation model, by freezing its original parameters and adding spatial/ ppatial/consistenr coosistency control of entities through a small set of trainable parameters (13% of total parameters) through the gated-attention module (Li et al., 2023). This enables our Layout2Vid to be trained solely on layout-annotated images, thus bypassing the need for expensive training on annotated video datasets. To preserve the identity of entities across scenes, we use shared representations for the entities within the same consistency group. We also propose use a joint image+text embedding as entity grounding conditions which we find more effective than the existing text-only approaches (Li et al., 2023) in entity identity preservation (see Appendix H). Overall, our Layout2Vid avoids expensive video- level training, improves the object layout and movement control, and preserves objects cross-scene temporal consistency.
We conduct experiments on both single-scene and multi-scene video generation. Experi- ments show that our VIDEODIRECTON PORGT demonstrates better layout skills (object, count, spatial, scale) and object movement control compared with ModelScopeT2V (Wang et al., 2023b) as well as more recent video generation models including AnimateDiff (Guo et al., 2023), I2VGen-XL. (Zhang et al., 2023), and SVD (Blattmann et al., 2023a). In addition,
2

---

Published as a conference paper at COLM 2024
Prompt
VideoDIrectorGPT
Scene
Scane
Scane 3
hungry cat Finding food
Video Planner
Video Plan
Video Generator (Layout2Vid)
(GPT-4
Scene Description
Entities layouts) Consistancy Grouping
Background
Video Planner
Scene
cat is lying deer
aud
Frane 21 Ci flafity Scamess cat (euty 0.00, 6.N0 pluns bedge bae's (9,00) 9.90, 5.09 1.00)
Bedroon
Scene
Hen sie gets us
Frone 21 6% flaffty Scieen catts [0.2 30, 6.30 0.00),
plush belge beels (9.00, 9.00, 1.00, 1.003
Bodinpon
Scene 3
Ste gaee to the actchse and esta trace
Froee 21 E1 Plartyy Mumerio
(e
Tgiameet
19, 9:45, Ri w.ss)
kitchen
Scene Description
Layout Consistency Grouping
Layout2Vid
Guided 2D Attention
Video Generator
Scene
fiarit stanet Scana: [t, 4]
Spotial Temporal Blocks
B
Scoee: [L,
Spatial
Seff-Artm
Scene
gets
gaanet saac
Temporal Comy
Spatial
Scene
She poss to kitchen
Scane: [L 2]
Scene
Text
Crou-dite
Figure 2:
VideoDIrecTorGPT details. 1st stage: we employ the LLM as a video planner to craft video plan, which provides overarching plot for multi-scene videos. 2nd stage: we utilize Layout2Vid, a grounded video generation module, to render videos based on the video plan. (Sec. 3.2).
VIDEODIRECTORGIT is capable of generating multi-scene videos with visual consistency across scenes, and competitive with SOTAs on single-scene open-domain T2V generation (Sec. 5.1 and Sec. 5.2). Detailed ablation studies, including dynamic adjustment of layout control strength, video plans generated from different LLMs, and video generation with user-provided images confirm the effectiveness and capacity of our framework (Sec. 5.3 and Appendix H). Our main contributions can be summarized as follows:
A new T2V generation framework VIDEODiRECTORRPT with two stages: video content planning and grounded video generation, which is capable of generating a multi-scene
video from a single text prompt.
We employ LLMs to generate multi-component 'video plan' with detailed scene descrip- tions, entity layouts, and entity consistency groupings to guide video generation.
We introduce Layout2Vid, a novel grounded video generation module, which brings together image/ text-based layout control ability and entity-level temporal consistency. Our Layout2 Vid can be trained efficiently using only image-level layout annotations.
Empirical results demonstrate that our framework can accurately control object layouts and movements, and generate temporally consistent multi-scene videos.
2 Related Works
Text-to-video generation. Training T2V generation model from scratch is computationally expensive. Recent work often leverages pre-trained T2I generation models such as Stable Diffusion (Rombach et al., 2022) by fine-tuning them on text-video pairs (Wang et al., 2023b; Blattmann et al., 2023b). While this warm-start strategy enables high-resolution video generation, it comes with the limitation of only generating short video clips, as T2I models lack the ability to maintain consistency through long videos. On the other hand, recent works on long video generation (Blattmann et al., 2023b; Yin et al., 2023; Villegas et al., 2023; He et al., 2023) aim at generating long videos of several minutes. However, the generated videos often display the continuation or repetitive actions instead of transitions of multiple actions/ rutions/events: events. In contrast, our Layout2Vid infuses layout control and multi-scene temporal consistency into a pretrained T2V generation model via data and parameter- training while preserving its original visual quality.
3

---

Published as a conference paper at COLM 2024
(a) Training
"doy"" "cit" "chall"
(c) Guided 2D Attention
"A dog and cat chaic"
text encoder
Grounding Tokens
Image encoder
Image
Grounding
Self-Attn
bos:|0..0, 10.10, 0.50 9.53]
bost (0.57, 0.34, 0.82, a.6l)
Bbox Emb.
Gated Self-Attn
bos: (8.35, 0.11, 0.73, 0.75,0.37)
(b) Inference
Grounding Tokens
fluffy Stamese car* plush beige bed" "gourmet cat snack!
encoder
Text Emb.
Cross- Attn
unCLIP Prior
Image Emb.
Visual Tokens
box [0.25 0., 1,00,0 8.700
Boar 10.00, 1,00, 1,800
Bbox Emb.
Descriptions
8.85]
Figure 3: Illustration of Layout2Vid. (a) Training, Our Layout2Vid efficiently trained on image-level bounding box annotations. We construct entity grounding tokens with an MLP that takes text, image, and bounding box embeddings as inputs. (b) Inference. we obtain image embeddings of entities from their text descriptions with unCLIP Prior. To ensure temporal consistency, we use the same entity embeddings across scenes. (c) Guided 2D Attention. We modulate the visual representation with grounding tokens and text tokens.
Bridging text-to-image generation with layouts. To achieve interpretable and controllable generation, line of research decomposes the T2I generation task into two stages: text-to- layout generation, and layout-to-image generation. While early models train the layout generation module from scratch (Hong et al., 2018; Tan et al., 2019; Li et al., 2019a; Liang et al., 2022), recent methods employ pretrained LLMs to leverage their knowledge in generating image layouts from text (Cho et al., 2023b; Feng et al., 2023; Qu et al., 2023). To the best of our knowledge, our work is the first to utilize LLMs to generate structured video plans from text, enabling accurate and controllable long video generation.
VIDEODIRECTORGPI
3.1
Video Planning: Video Plans with LLMs
Video plan. As illustrated in the blue part of Fig. 2, GPT-4 (OpenAI, 2023) acts a planner, providing a detailed video plan to guide the video generation. Our video plan has four components: (1) multi-scene descriptions: a sentence describing each scene, (2) entities: names and their 2D bounding boxes, (3) background: text description of the location of each scene, and (4) consistency groupings: scene indices for each entity indicating where they should remain visually consistent. The video plan is generated in two steps by prompting GPT-4 independently. The input prompt template for each step is displayed in Appendix C.
Step 1: Generating multi-scene descriptions, entity names, and consistency groupings. In the first step, we employ GPT-4 to expand single text prompt into multi-scene video plan. Next, we group entities and backgrounds that appear across different scenes using an exact match. For instance, if the 'chef' appears in scenes 1-4 and 'oven' only appears in scene 1, we form the entity consistency groupings as (chef 11,2,3,4]] oven: [1] J. In the subsequent video generation stage, we use the shared representations for the same entity consistency groups to maintain their temporally consistent appearances (see Sec. 3.2).
Step 2: Generating entity layouts for each scene. In the second step, we generate
list of bounding boxes for the entities in each frame based on the entities and the scene description. For each scene, we produce layouts for frames, then linearly interpolate to gather information for denser frames (e.g., 16 frames per scene). In line with VPGen (Cho et al., 2023b), adopt the (xo, yo, X1, format for bounding boxes, where each coordinate normalized to fall within the range [0,1]. For in-context examples, we present 0.05 as the minimum unit for the bounding box, equivalent to 20-bin quantization.
4

---

Published as a conference paper at COLM 2024
3.2 Video Generation:
Generating Videos from Video Plans with Layout2Vid
Preliminaries. Given T' frame video x RTx3xHxW with video caption c, we first uses an autoencoder to encode the video into latents z = {x) E RTxCxH' xw" where C= 4 represents the number of latent channels, and H' = H/8 W W/8 represent the spatial dimension of the latents. In Latent diffusion model (LDM) (Rombach et al., 2022), the forward process is fixed diffusion process which gradually adds noise the latent variable
2: (()((Z = [z / - B,1), where B, E (0, is the variance schedule with t E (1, T). The reverse process gradually produces less noisy samples =т - 20 starting from 2T through learnable denoiser model eg. With as timestep uniformly sampled from {1,... T), the training objective of our Layout2Vid can be expressed as:
ming LLDM = Ez,=-(.. t,
Layout2Vidd Layout-guided T2V generation. We implement Layout2Vid by integrating layout control capability into ModelScopeT2V (Wang et al., 2023b), public T2V gener- ation model based on Stable Diffusion (Rombach et al., 2022). The diffusion UNet in ModelScopeT2V consists of series of spatio-temporal blocks, each containing four mod- ules: spatial convolution, temporal convolution, spatial attention, and temporal attention. Compared with ModelScopeT22, our Layout2Vid enables layout-guided video generation with explicit spatial control over a list of entities represented by their bounding boxes, as well as visual and text content. We build upon the 2D attention create the guided 2D attention (see Fig. 9 in Appendix D). As shown in Fig. 3 (c), the guided 2D attention takes two conditional inputs to modulate the visual latent representation: (1) grounding tokens, conditioned with gated self-attention (Li et al., 2023), and (2) text tokens that describe the current scene, conditioned with cross-attention.
Temporally consistent entity grounding with image+text embeddings. While previous layout-guided T2I generation models commonly used only the CLIP text embedding for layout control (Li et al., 2023; Yang et al., 2023), use the CLIP image embedding in addition to the CLIP text embedding for entity grounding. We demonstrate in our ablation studies (see Appendix H) that this method is more effective than using either text-only or image-only grounding. As depicted below, the grounding token for the jth entity, h;, is 2-layer MLP which fuses CLIP image embeddings fing (e;), CLIP text embeddings frex( and Fourier features (Mildenhall et al., 2021) of the bounding box 1; = [No, yo, X1, y1]- We use learnable linear projection layers, denoted as Ping/textr on the visual text features, which we found helpful for faster convergence during training.
h MLP(Pimg (figg(:)) Piexa (iex(()))))))) Fourier(4:)
The training and inference procedure of our Layout2Vid are presented in Fig. 3 parts (a) and (b) respectively. During training, the image embeddings fing (e) are obtained by encoding the image crop of the entities with CLIP image encoder. During inference, since all the inputs are in text format (e-g., from the video plan), we employ Karlo (Lee et al., 2022), public implementation of unCLIP Prior (Ramesh et al., 2022) to transform the CLIP text embedding into its corresponding CLIP image embedding. Moreover, our image embeddings can also be obtained from user-provioe eree-provdee exemplars during inference by simply encode the images with the CLIP image encoder.
Parameter and data-efficient training. During training, we only update the parameters of the guided 2D attention (13% of total parameters) inject layout guidance capabilities into the pretrained T2V backbone while preserving its original video generation capabil- ities. Such strategy allows us to efficiently train the model with only image-level layout annotations, while still equipped with multi-scene temporal consistency via shared entity grounding tokens. Training and inference details are shown Appendix D.
4 Experimental Setup
Evaluated models. We compare our VIDEODIRECTORGPT with a total of 9 text/ text/image-te video generation models (see Sec. 5.1 and Appendix F for baseline model details).
5

---

Published as a conference paper at COLM 2024
Prompts for single-scene video generation. For single-scene videos, we (1) evaluate layout control via VPEval Skill-based prompts (Cho et al., 2023b), (2) assess object dynamics through Actionbeench-Drrectoon prompts adapted from ActionBeecc (Wang et al,, 2023c), and (3) examine open-domain video generation using MSR-VTT and UCF-101 (Xu et al., 2016; Soomro et al,, 2012). We introduce ActionBench-Direction prompts by sampling video captions from ActionBench-SSVZ (Wang et al., 2023c) and balancing the distribution of movement directions. Prompt preparation details are presented in Appendix F.
Prompts for multi-scene video generation. For multi-scene video generation, we experi- ment with (1) a list of sentences describing events ActivityNet Captions (Krishna al., 2017) and Coref-SV prompts based on Pororo-SV (Li et al., 2019b), and (2) single sentence from which models generate multi-scene videos HiREST (Zala al., 2023). Coref-SV is new multi-scene text description dataset that we propose to evaluate the visual consistency of objects across multi-scene videos. Prompt preparation details are given in Appendix F.
Automated evaluation metries. Following previous works (Hong et al., 2022; Wu et al., 2022b; Wang et al., 2023b), we use FID (Heusel et al., 2017), FVD (Unterthiner et al., 2019), and IS (Salimans et al., 2016) scores as video quality metrics, and CLIPSIM (Wu et al., 2021) score for text-video aligmment. Given that CLIP fails to faithfully capture detailed semantics such spatial relations, object counts, and actions in videos (Otani et al., 2023; Cho et al., 2023a;b; Hu et al., 2023), we further propose the use of the following fine-grained metrics:
VPEval accuracy: we employ it for the evaluation of VPEval Skill-based prompts (object, count, spatial, scale), which is based on running skill-specific evaluation programs that execute relevant visual modules (Cho et al., 2023b). Since the VPEval accuracy described above does not cover temporal information, we propose a metric that takes into account temporal information as well as spatial layouts for ActionBench-Directin
prompts.
Object movement direction accuracy: we introduce this new metric to evaluate Cirionneenh-Drectiio prompts, which takes both temporal information and spatial layouts into consideration. Firstly, we assess whether the target objects move in the direction described in the prompts. The start/end locations of objects are obtained by detecting objects with GroundingDINO (Liu et al., 2023) on the first/last video frames. We then evaluate whether the x (for movements left or right) or y (for movements up or down) coordinates of the objects have changed correctly and assign a binary score of 0 or 1. For instance, given the prompt "pushing a glass from left to right" and a generated video, we identify 'glass' in both the first and last video frames. We assign a score of 1 if the x-coordinate of the object increases by the last frame; otherwise, assign a score of 0.
Multi-scene object temporal consistency: we propose this new metric for the evaluation of ActivityNet Captions and Coref-SV, which measures the consistency of the visual appearance of a target object across different scenes. We also introduce a new metric to measure the consistency of the visual appearance of a target object across different scenes. Specifically, we first detect the target object using GroundingDINO from the center frame of each scene video. Then, we extract the CLIP (ViT-B/32) image embedding from the crop of the detected bounding box. We calculate the multi-scene object consistency metric by averaging the CLIP image embedding similarities across all adjacent scene pairs: N E.N=T cos(CIs) CLIPimg
(+t))) ), where N is the number of scenes, cos(.,.) is cosine similarity, and ILIP the CLIP image embedding of the target object in æ-th scene.
Human evaluation. We conduct a human evaluation on the multi-scene videos generated by both our VIDEODIRECTORO we know the target entity and its co-reference pronouns in the Coref-SV prompts, we can compare the temporal consistency of the target entities across scenes. We evaluate the human preference between videos from two models in each category of Quality, Text-Vider Alignment, and Object Consistency. See Appendix F for setup details.
and ModelScopeT2y on the Coref-SV dataset. Since
Step-by-step error analysis. We conduct an error analysis at each step of our single-sentence to multi-scene video generation pipeline on the HiREST dataset. We analyze the generated multi-scene text descriptions, layouts, and entity consistency groupings to evaluate our video planning stage, and examine the final video to evaluate the video generation stage.
6

---

Published as a conference paper at COLM 2024
5 Results and Analysis
5.1 Single-Scene Video Generation
Layout control results (VPEval Skill-based prompts). Table (left) displays the VPE- val accuracy on the VPEval Skill-based prompts. Our VideoD significantly outperforms ModelScope and recent T2V /1V generation models including AnimateD-- iff (Guo et al., 2023), 12VGen-XL (Zhang et al., 2023), and SVD (Blattmann et al., 2023a) on all layout control skills. These results suggest that layouts generated by our LLM are highly accurate and greatly improve the control of object placements during video generation. Fig. 4 (left) shows an example where our LLM-generated video plan successfully guides Layout2Vid to accurately place the objects, while ModelScope fails to generate 'pizza'. In Appendix J, we show additional examples (see Fig. 12) that our video plan can generate layouts requiring understanding of physics (e.g., gravity, perspectives).
VPEval Skiel-based
ActionBench-Direcee
Method
Object
Count
Spatial
Scale
Overall Acc. (%)
Movement Direction Acc. (%)
ModelSeopeT2V
89.8
38.8
18.0
15.8
40.8
30,5
Dnim
96.7
52.7
22.5
15.5
46.8
29.0
12VGen-XL
96.5
62.0
35.2
23.7
54.3
35.2
SVD
93.1
46.7
29.2
15.0
45.7
30.7
VIDEDDIRECTORPTT
97.1
77.4
61.1
47.0
70.6
46.5
Table 1:
Evaluation on VPEval Skill-based and ActionBench-Direecion prompts.
ActionBench Direction
pizza elephant
Pushing
Frone
ModelScopeiTNV T2V
VideooIrectorGPT (Ours)
Figure4: 4: Generated examples on VPEval Skill-based and ActionBench-Direction prompts.
Object movement results (Actioneend--Directon)) Table 1 (right) shows the performance on the ActhonBench-Direcctoon prompts. For each object, we give prompts that include four movement directions: 'left to right', 'right to left', 'top to bottom', and 'bottom to top'. Therefore, for random guess, the movement accuracy is 25%. Our VIDEODIRECTORGIP outperforms ModelScopeI2V and other T2V /1VV models in object movement direction accuracy by a large margin, demonstrating that our LLM-generated layouts can improve the accuracy of object dynamics in video generation. Fig. 4 (right) shows that our LLM- generated video plan guides the Layout2Vid module to correctly locate and move the 'pear' in the accurate direction, whereas the 'pear' in the ModelScope video just gets grabbed.
Open-domain results. Table 2 presents the FVD, FID, and CLIPSIM scores MSR-VTT. It's worth noting that our model is built on the pre- trained ModelScope backbone. Despite this, our method demonstrates good improvements in FVD and competitive scores in FID and CLIP-
MSR-VTT
Method
FVD
FI)
CLIPSIM
Different arch Training data
NUWA
47.68
0.2439
CogVidea (Chinese)
24.78
CogVideo (English)
1294
23.59
0.2631
MagieVideo
1290
SIM. In addition, our method achieves better or comparable performance to models trained with larger video data (e.g., Make-A-Vided A-Video) with higher resolution (e.g., VideoLDM). Evaluation results on UCF-101 are presented in Appendix I, where we achieve significant improvement in FVD and similar Inception Score.
VideoL.DM
Make-A-Vides
13:17
0.3049
Same video backbone & Test prompls
ModelSopee2yy VIDIODRRCTORGPT
683
12.32
0.2909
12.22
0.2860
Table 2: Comparison results on MSR- VTT. ModelScope T2 Our replication on the same 2990 random test prompts.
7

---

Published as a conference paper at COLM 2024
Method
ActivityNet Captions
Coret-SVV Consistency (1)
HIREST
FTD (V)
Consistency (1)
FVD
FTD (1)
ModelScopeT22
18.12
23.79
ModelScopeT2V (with GT co-reference, oracle) ModelScopeT2V (with prompts from our nidao plane) VIDEODIRECTORGPT (Ours)
37,9
918
18,96
805
16.30
64.8
42.8
733
18.54
Table 3: Multi-scene video generation on ActivityNett Captions, Coref-SV, and HiREST. GT co-reference: replacing co-reference pronouns in Coref-SV with the original object names (e.g., "his friends" may become "dog's s friends"). Prompts from our video plan: an enhanced benchmark for ModeIScopeT2V using scene descriptions from our video plan.
Gonl-SV
HREST
Caraany
ModelSoapeTTV
[0un)
ModedScopaTTV
Froms
Prac
Trane
Scona
mnaeelt billing 1o0k rosss Aeo
Carstey Cakes 11-
S
palingpeeta Feww
1 palling 2iid Tower
Male Cakes 1i-311
antinng and lrdin dogg
lakiigg hobiin Hight
Caianay Cabes
post an Booa so
Figure 5: Generation examples Coref-SV (left) and HiREST (right).
5.2 Multi-Scene Video Generation
Multiple sentences to multi-scene videos generation (ActivityNet Captions / Coref-SV). As shown in the left two blocks of Table 3, our VIDEODIRECTORGPI outperforms Mod- elScopeT2V in visual quality (FVD/FID /FD) and multi-scene object temporal consistency. No- tably, for Coref-SV, our VIDEODIRECTORGPT achieves higher object consistency than ModelScope even with GT co-reference (where pronouns are replaced with their orig- inal noun counterparts, acting as oracle information; e.g., "she picked up ..."" becomes "cat picked up ..."). As do not have ground-truth videos for Coref-SV, we only use to evaluate consistency when co-references are used across scenes. Fig. 5 (left) shows video generation example from Coref-SV, where the LLM-generated video plan can guide the Layout2Vid module to generate the same mouse across scenes consistently, whereas ModelScope generates a hand and a dog instead a mouse later scenes.
Single sentences to multi-scene videos generation (HiREST). The right block of Table 3 shows our VIDFODIRECTORGPT achieves better visual quality scores (FVD/FID) than ModelScope112V on the HiREST dataset. Moreover, by comparing with an enhanced Mod- elScope11V benchmark, where videos are generated from each scene description in our GPT-4-produced video plan, we demonstrate the effectiveness of prompts generated by GPT-4. However, still performs worse than our VIDEODIRECTORGPT, also showing the effectiveness of our Layout2Vid. As shown in Fig. 5 (right), our LLM can generate step-by-step video plan from a single prompt and our Layout2Vid can generate consistent videos following the plan, showing how to make caraway cakes (a British seed cake). In contrast, ModelScope repeatedly generates the (visually inconsistent) caraway cake.
5.3 Additional Analysis
Generating videas with custom image exemplars. Our Layout2Vid can obtain CLIP image embeddings either from user-provided image exemplars from entity text descriptions via
8

---

Published as a conference paper at COLM 2024
Ently Grounding
Generated Scenes
<8> then gets up from
Soene a goes drtam-ooloord
noxt to
Teat
<B "whito oor"
Inager
Figure 6: Video generation with text-only (top) and image+text (bottom) inputs with ViDEODiRECTORGPT By giving text or image+text inputs, users can generate images where the identities of the entities are preserved across multiple scenes.
If Denoising steps with lay- out guidance (out of 50)
MSR-VTT
ActionBench--Drecttee
FVD (L)
FID (1)
CLIPSIM (1)
Move Direction Acc. (%)
& 0.1 (5 steps)
550
12.22
0.2860
46.5
& 0.2 (10 steps)
588
17.25
0,2700
59.8
& 0.3 (15 steps)
17.17
0.2702
57.5
I.I.-DDnamiic4 (5-15 steps)
13.75
0.2790
56.8
Table 4: Ablation of the denoising steps with layout guidance (via guided 2D attentions) in MSR-VTT and ActionBench--Dreectonn prompts, where # stepp with layout guidance
# total steps
the unCLIP Prior. In Fig. 6, we demonstrate that our Layout2Vid can flexibly take either text-only image+text descriptions as input generate consistent multi-scene videos.
Dynamic layout strength control with GPT-4. The number of denoising steps with layout guidance a controls layout control strength of Layout2Vid (see Appendix for more detailed illustration). Instead of using a static a value, we explore dynamically adjusting it during the video plan generation by asking LLM how much layout guidance needs to be used for each prompt. Table 4 compares static a values and dynamic values (LLM-Dynamica)) showing that LLMS help achieve good balance in the quality-layout trade-off.
Video Planner LLMs: GPT-4, GPT-3.5-turbo, and LLaMA2. We conducted an ablation study using the open-sourced LLaMA2-7B and LLaMA2-13B models (Touvron al., 2023b) as well as Gl'T-3.5-turbo in the video planning stage MSR-VTT and ActionBench-Direction evaluated LLaMA2 in both fine-tuned and zero-shot settings. To collect data for fine-tuning LLaMA2, randomly sampled 2000 prompts from WebVid-10M, (Bain et al., 2021) and generated layouts with GPT-4. In addition to the visual quality (FID/FVD) and video- text alignment (CLIPSIM) scores MSR-VTT and movement direction accuracy score we also report the number of samples whose output layouts can be successfully parsed (egg, #Samples). Evaluation setup details are given in Appendix H. Table 5 shows that zero-shot LLaMA2-13B struggles to generate layouts that can be suc- cessfully parsed (e.g,, only 82 out of 600). It also achieves worse scores on all three metrics (FID, FVD, and CLIPSIM) on MSR-VTT, as well as the movement direction accuracy on ActionBenchh-Drrcctorr which shows that GPT models have stronger in-context learning skills and can generate more reasonable layouts. On the other hand, all scores can be improved by fine-tuning on layouts generated by GPT-4. Fine-tuning open-sourced L.MSs can be future direction worthwhile to explore. Another point to note is that GPT-3.5-turbo and GPT-4 achieve very similar performance on MSR-VIT, but GPT--35-turbo performs significantly worse than GPT-4 ActionBench-Directte is less effective at handling prompts requiring strong layout control compared to GPT-4.
We
This suggests that GPT-3.5-turbo
Challenging cases. The bounding box-based control of our method poses two main chal- lenges. Firstly, some objects might not follow the bounding box control well when there are too many overlapping bounding boxes. For example, in Fig. 5 right, the "hand mixer" and

---

Published as a conference paper at COLM 2024
MSR-VTT
ActionBench- Dinrcctioonni
LLMs
(I) (J)
CLIPSIM
#Samples
Movement Direction Ace
#Samples
Fine-famed
LLaMA2-7B
12.69
0,2851
2749
35.8
LLaMA2-13B
556
12.40
0.2854
2786
53.3
561
Zero-shot
LLaMA2-13B
573
13.47
0.2792
2236
GPT-3.5-turbo
558
12.27
0.2852
2990
49.0
600
GPT-4 (default)
12.22
0.2860
2990
59.8
600
Table 5: Ablation of video generation with video plans generated from different LLMs MSR-VTT and ActionBenchh-irrctioos.
"unsalted butter", which have relatively small bounding box sizes, do not follow the box control well. Secondly, the control of whether the background is static or moving is usually determined by the prior knowledge of the video generation model. For example, in Fig. 13, it's hard for Layout2Vid to generate moving bottle with static background and moving boat with a completely static background (see paragraph "Object movement prompts with different types of objects" in Appendix J.2 for details).
shown
5.4 Human Evaluation
Human Preference (%)
Evaluation category
Human preference. We conduct a hu- man evaluation study on the multi-scene videos generated by both our VIDEODIREC- TORGPT and ModelScope112 on the Coref- SV dataset. We show 50 videos to 10 crowd- annotators from AMT to rate each video (56 annotators in total, 10 annotators per video) and calculate human preferences for each video with average ratings. Table 6 shows that the multi-scene videos (with Coref-SV prompts) generated by our ViDEODIRECTORGPT are preferred by human annotators than ModelScope videos in all three categories (Qual- ity, Text-Video Alignment, and Object Consistency).
Ours
ModeIScopeI2V
Tie
Quality Text-Video Alignment
50
34
16
58
36
Object Consistuncy
28
Table Human preference on multi-scene videos generated with Coref-SV prompts.
Step-by-step error analysis. In our error analysis of video planning and generation stages, we measure 1-5 Likert scale accuracy of four intermediate generation steps: scene descrip- tions, layouts, consistency groupings, and final video. While the first three planning steps achieve high scores (2 4.52), there is a big score drop in the aayout-guide video gener- ation (4.52 - 3.61). This suggests that our VIDEODIRECTORGP" could generate more accurate videos, once we have access to a stronger T2V backbone than ModelScopeTTV. See Appendix G for more detailed analysis.
Conclusion
In this work, we propose VIDEODIRECTORGPT novel framework for consistent multi- scene video generation, leveraging the knowledge of LLMs for video content planning and grounded video generation. In the first stage, we employ GPT-4 as video planner to craft video plan, a multi-component script for videas with multiple scenes. In the second stage, we use Layout2Vid, a grounded video generation module, to generate videos with layout and consistency control. Experiments demonstrate that our proposed framework substantially improves object layout and movement control over state-of-the-art methods on open-domain single-scene video generation, and can generate consistent multi-scene videos while maintaining visual quality.
Amazon Mechanical Turk: https: http://wwwtttttt.ooo/
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank the reviewers for the thoughtful discussion and feedback. This work was sup- ported by ARO W911NF2110220, DARPA MCS N66001-19-2-4033, DARPA KAIROS Grant FA8750-19-2--15 19-00044-, NSE-AI Engage Institute DRL2112633 ONR N00014-23--2355 Accelerate Foundation Models Research program, and a Bloomberg Data Science Ph.D. Fellowship. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency,
References
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arv preprint arXio:2311.15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: social biases of text-to-image generation models. In ICCV, 2023a.
Probing the reasoning skills and
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image genera- tion and evaluation. In NeurlPS, 2023b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanu- malayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Surís Dídac, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. In NeuriPS, 2023.
11

---

Published as a conference paper at COLM 2024
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communi- cations of the ACM, 63(11))139-444, 2020.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. Xrv preprint ar Xiv::330000000 04725, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953- 14953-14962, 2023.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video dif- fusion models for high-fidelity video generation with arbitrary lengths, arXiv preprint arXiv:2211.13221, 2022.
Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-story: Storytelling with retrieval-augmented video generation. ar arv preprint ar axXit:2307.06940) 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systemss, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In CVPR, 2018
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2022.
Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439, 2023.
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. Deepstory: Video story qa by deep embedded memory networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI'17, pp. 2016-2022. AAAI Press, 2017. ISBN 9780999241103.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In JCLR, 2014.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In International Conference on Computer Vision (ICCV), 2017.
12

---

Published as a conference paper at COLM 2024
Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Baek, and Saehoon Kim. Karlo-v1.0.alpha on coyo-100m and cc15m. https //ggtt/gbbt com/ kakaobrain/karlo, 2022.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12174-12182, 2019a.
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In AAAI, 2017.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. Storygan: A sequential conditional gan for story visualization. In CVPR, 2019b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chun- yuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22511-22521, 2023.
Jiadong Liang, Wenjie Pei, and Feng Lu. Layout-bridging text-to-image synthesis. ArXio, abs/2208.06162, 2022. URL https: / /api.. semanticscholar htte//ooppsID::515555
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022. URL
nttps://openrevieet wwwpeeewwwwwweeeeeeeeeeee tete/foormmtti ii-.ummiidd...
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text-to-image transformers for story continuation. In European Conference on Computer Vision, pp. 70-87. Springer, 2022.
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106 2021.
OpenAI. Gpt-4 technical report, 2023.
Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In CVPR, 2023.
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutllm-I2i: Eliciting layout guidance from 11m for text-to-image generation. In ACM MM, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL https:
s/api..seaatticcchol... wtt.ee/c........oooo.o..aaaaaa..
13

---

Published as a conference paper at COLM 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICCAI, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
A Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
Fuwen Tan, Song Feng, and Vicente Ordonez. Text2scene: Generating compositional scenes from textual descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6710-6719, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristían Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Díana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michal- ski, and Sylvain Gelly. Fvd: A new metric for video generation. In ICLR Workshop, 2019.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. In ICLR, 2023.
Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen- I-video: Multi-text to long video generation via temporal co-denoising. arxiv preprint arXit:230511264, 2023a.
Jiuniu Wang, Hangje Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report, 2023b.
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching action knowledge in video-language foundation models. In NewrIPS, 2023c.
14

---

Published as a conference paper at COLM 2024
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXio preprint ar Xir::2044114866 2021.
Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. In NeurlPS, 2022a.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European conference on computer vision, pp. 720-736. Springer, 2022b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video de- scription dataset for bridging video and language. In CVPR, June 2016.
URL https: http:///wwwmccooootttcoo http////wuurrreeeerrammaaiiiaaaa aagiiggiiggiiieee---d--aaagggg
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to- image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Ming Gong, Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. NUWA-XL: Diffusion over diffusion for eXtremely long video generation. In Proceedings of the 61sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1309-1320, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 http//222...llloo.7..... URL attps://aclantholggg org/2023 acl acl- -ong. 73.
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23056-23065, 2023.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint 1xxx:2311.047// 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint XxXie:2211.11018, 2022.
15

---

Published as a conference paper at COLM 2024
Appendix
A Reproducibility Statement
17
Additional Related Works
17
Video Planning
17
C.1 GPT-4 Prompt Templates
17
C.2 API Cost
18
D Layout2Vid
18
D.1 Preliminaries: T2V Generation Backbone
18
D.2 Layout2Vid Spatio-Temporal Blocks
19
D.3 Training and Inference Details
19
Human-in-the-Loop Video Plan Editing
19
Experiment Setup
20
F1
Evaluated Models
20
F2 Prompts for Single-Scene Video Generation
21
F3 Prompts for Multi-Scene Video Generation
21
Human Evaluation Details
22
Ablation Studies
23
H.l
Video Planner LLMs: GPT-4, GPT-3.5-turbo, and LLaMA2
23
H.2
Entity Grounding Embeddings: Image Text
24
H.3 Layout Control: Bounding Box v.s. Center Point
24
H.4 Trainable Layers: Gated Self-Attention Only v.s. Entire Guided 2D Attention
24
I Additional Experiments
L.1
Comparison on UCF-101
25
J Additional Visualization and Qualitative Examples
25
J.1 Visualization of Different Layout-Guided Denoising Steps.
25
J.2 Additional Qualitative Examples
27
Limitations
30
16

---

Published as a conference paper at COLM 2024
A Reproducibility Statement
Our model is built upon the publicly available code repository from GLIGEN (Li et al., 2023)² and ModelScopeT2V (Wang et al., 2023b)>. Please see Sec. 3.2/ Appendxx D for model architecture details, Sec. 4/ Appendix F.2 Appendii F.3 for dataset details. We provide code and video samples (video.samples. html) in the supplementary material.
B Additional Related Works
Text-to-video generation. The text-to-video (T2V) generation task is to generate videos from text descriptions. Early T2V generation models (Li et al., 2017; 2019b) used varia- tional autoencoders (VAE) (Kingma & Welling, 2014) and generative adversarial networks (GAN) (Goodfellow et al., 2020), while multimodal language models (Hong et al., 2022; Wu et al., 2022a; Villegas et al., 2023; Maharana et al., 2022; Ge et al., 2022; Wu et al., 2021) and denoising diffusion models (Ho et al., 2022; Singer et al., 2023; Blattmann et al., 2023b; Khachatryan et al., 2023; Wang et al., 2023a; Yin et al., 2023) have become popular for recent works. Since training T2V generation model from scratch is computationally expensive, recent work often leverages pre-trained text-to-image (T2l) generation models such as Stable Diffusion (Rombach et al., 2022) by finetuning them on text-video pairs (Wang et al., 2023b; Blattmann et al., 2023b). While this warm-start strategy enables high-resolution video generation, it comes with the limitation of only being able to generate short video clips, as T2I models lack the ability to maintain consistency through long videos. Recent works on long video generation (Blattmann et al., 2023b; Yin et al., 2023; Villegas et 2023; He et al., 2023) aim at generating long videos of a few minutes. However, the generated videos often display the continuation repetitive patterns of a single action (e.g,, driving car) instead of transitions and dynamics of multiple changing actions/ actions/evente events (e.g., five steps about how to bake a cake). In this work, we address this problem of multi-scene video generation with a two-stage framework: using an LLM (e.g., GPT-4) to generate a structured video plan (consisting of stepwise scene descriptions, entities and their layouts) and generating videos using Layout2Vid, a tayout-guide text-to-video generation model with consistency control. Our Layout2 infuses layout control and multi-scene temporal consistency into pretrained T2V generation model via data and parameter-efficient training, while preserving its original visual quality.
C
Video Planning
C.1 GPT-4 Prompt Templates
In this section, we provide the prompt templates we give to our video planner (Sec. 3.2). The video plan is generated in two steps by prompting GPT-4+ with different in-context examples (we use 1 and 5 in-context examples for the first and second steps, respectively). In the first step (see Fig. 7), we ask GPT-4 to expand a single text prompt into a multi-scene video plan. Each scene comes with text description, a list of entities, and background. In the second step (see Fig. 8), we generate list of bounding boxes for the entities in each frame based on the list of entities and the scene description. In line with VPGen (Cho et al., 2023b), we utilize the [xo, yo, X1, V1] format for bounding boxes, where each coordinate is normalized to fall within the range [0.1]. For in-context examples, we present 0.05 as the minimum unit for the bounding box, equivalent to 20-bin quantization over the [0,1] range.
2GLIGEN:
MoodeSScope122 https://githhb. wowm//EppoeettiaaaM/eeeeeeee http:tttt--iiidde---tttnnnneeeee wwweeeninnnntttteeeeeennn weaiiiinngg/eeeeaiiin We employ the gp6-4 version.
17

---

Published as a conference paper at COLM 2024
Instructions:
Given single text prompt, you need to ervision a multi-seene video by generating a sequence of stepwiis prompts to describe the text prompt, For each step, you also need to generate the set of entities needed and describe the background scene where the video should occur Related stops should maintain similar entibles background scones. Before you write each stepwise descoptton, you must these instructions:
1. Each step prompt must contain only a single motion or action.
Each stap prompt must include relevant objects and describe the environment scane.
3. Make sure each step prompt must be easy destribed by a shot video clip of 8 seconds at zfas.
Use the following format: [|stop_ |prompt, entities, background]; step 2: [prompt, entities, background]] ... stop_n: [prompt, entities, background]]
In-context example:
Input: stepwise prompts for the prompt: make butter biscuits
Output:
Step 1: |Prompt: chief preheats the oven; Enüties:
Background: kitchen counter];
Step 2: |Prompt: chief creams together butter and sugar a mixing bowe; Fntitfes: chief, mixing bowl, butter, sugar, hand mixer; Background: kitchen]
Step 3: |Prompt: A chief bests in eggs into the mixing bowl, Entities: chief, eups, mixing bowl; Background: kitchen];
Input text: Provide stepwise prompts for the text prompt: lisaort single tost promot hocel
Figure 7: Prompt template for the 1st video planning step (scene descriptions and enti- ties/backsa hossbbakkround generation).
Instructions:
Assuming size normalized the range 0-1, you give possible B-frame layout bounding boves scene description. Each object in the image is one rectangle or square box the layout and size of baves should be large as possible. You need to generate layouts from the close camera view of the event. The layout difference between two adjacent frames must be small considering the small also need generate caption that describes the image
entities given
frame. generating
design. bottom]ll, caption....
Use format:
Frame_ 1: [[object1] (left, top, right, bottomi], [object2, [left, top, right, bottom]l.... .., [object_,, n. [left, top, right, bottom]ll, caption:...
Frame [[object], (laft, top, right, bottomil. [ooject2, |left, right, bottomil.... (ooject [left,
Frame_& & [[object1, (Deft, top, right, bottomi], [object2, [left, top, right, buttom]].... .., [object_n, [left, top, right, bottom]ll, captionc.. Reasoning....
In-context example:
Input: Please
bounding box coordinates the following entitess based on the scene description. Entitiis: I'table', 'cat, (hhir") Soone description: cat jumps from a chair to table.
Output:
Frame_ 1: [l'table', [0.1, 0.6, 1.01. I'tat", [0.4, 0.55, 0.6, 0.6]I. I'chair,, 10.05, 0.6, 0.4, 1.01I,, caption: cat standing the table, preparing Frame 2: [["table", [0.1, 0.6, 0.3, 1.0,], I"ca",, (0.45, 0.65, 0.59]]. I"chair", (o.05, 0.6, 0.4, 1.0]]], caption: cat has just left the table and
Reasoning:
The carl's bounding box Frame is on the table, showing the cat in the take-off off position. From Frame to Frame the cat is shown various the
stages mid-flight, with catlys bounding moving
Input text: Provide bounding box coordinates for the prompt: [nseett entilees and sonne descriation here)
Figure 8: Prompt template for the 2nd video planning step (layout generation).
C.2 API Cost
Using GPT-4 tokenizer, the average input/outiy token lengths of each step are 2K/1K / for the first step and 6K/1K 1K for the second step. Using GPT-4, it takes 0.12 USD and 0.24 USD for the inference of the first and second steps, respectively.
D Layout2Vid
D.1 Preliminaries: T2V Generation Backbone
We implement Layout2Vid by injecting layout control capability into ModelScope T2V (Wang et al., 2023b), public text-to-video generation model based on Stable Diffusion (Rombach et al., 2022). ModelScopeT22 consists of (1) a CLIP ViT-H/14 (Radford et al., 2021) text encoder, (2) an autoencoder, and (3) a diffusion UNet (Ronneberger et al., 2015; Ho et al., 2020). Given a T frame video E I xx3xHx> with video caption and frame-wise layouts , MwddeMoooelpp11 first uses autoencoder to encode the video into latent repre- sentation = {x). The diffusion UNet performs denoising steps in the latent space to generate videos, conditioned on the CLIP text encoder representation of video captions. The UNet comprises series of spatio-temporal blocks, each containing four modules: spatial convolution, temporal convolution, spatial attention, and temporal attention. Since the original TV does not offer control beyond the text input, we build upon the 2D
18

---

Published as a conference paper at COLM 2024
Spatio-emmporal Block
Spatial Conv
Temporal Conv
Spatial
Temporal Attn
Norm
Norm
Norm
Norm
Linear
Linear
Act
Act
Guided
Attn
2D
Conv
Linear
xN,
KN
XN,
xN,
Figure 9: Spatio-temporal blocks within the diffusion UNet of Layout2Vid, The spatio- temporal block comprises four modules: spatial convolution, temporal convolution, spatial attention, and temporal attention. We adopt settings from ModelScopeT2V, where (N1, N2, N3, N4) are set to (2, 4, 2, 2). We use "Act" to represent activation layers.
attention module in the spatial attention module to create 'Guided 2D Attention', which allows for spatial control using bounding boxes.
D.2 Layout2Vid Spatio-Temporal Blocks
Fig. illustrates the spatio-temporal blocks within the diffusion UNet of Layout2Vid. The spatio-temporal block comprises four modules: spatial convolution, temporal convolution, spatial attention, and temporal attention. Following et al. (2023), we build upon the 2D attention to create the guided 2D attention, which enables layout-guided video generation with explicit spatial control over a list of entities represented by their bounding boxes, as well as visual and text content.
D.3 Training and Inference Details
Table 7 contains the model architecture details and sirriinngiiffeeccee parameter settings for our Layout2Vid.
Training. The highlight of our Layout2Vid training is that it was conducted solely on image-level data with bounding box annotations. We trained the MLP layers for grounding tokens and the gated self-attention layers with the same bounding-box annotated data used in GLIGEN (Li et al., 2023), which consists of 0.64M images. We train Layou(2Vid for 50k steps, which takes only two days with 8 A6000 GPUs (each 48GB memory). All the remaining modules in the spatio-temporal block (see main paper Fig. 4) are frozen during training. We illustrate the training and inference procedure of Layout2Vic in main paper Fig. 3.
Inference. During inference, entities to convert the texts into their corresponding CLIP image embeddings, and CLIP text encoder to get their corresponding text embeddings. We use CLIP ViT-L/14 as backbone during training to be consistent with Karlo. This helps us to preserve the visual consistency of the same object by using the same image embedding across scenes. In addition, we use PLMS Rombach et al. (2021) (based on PNDM Liu et al. (2022)) as our default sampler following GLIGEN (Li et al., 2023), since we found no consistent im improvements for visual quality and video-text alignment scores when switching to the DDIM (Song et al., 2020) sampler used in ModelScope12v. The comparison of the PLMS and DDIM samplers is shown in Table 8.
the Karlo implementation of unCLIP Prior to the
Human-in-the-Loop Video Plan Editing
One useful property of our VIDEODIRECTORGPT is that it is very flexible to edit video plans via human-in-the-loww editing. Given an LLM-generated video plan, users can gener- ate customized content by adding/ deletineleg////// rrepaacigellaiigggg the entities, adding/ caaoongg//hannngggg the background, and modifying the bounding box layouts of entities. Figure 10 showcases the
19

---

Published as a conference paper at COLM 2024
Architectare
Training
z-shape
32 x 32 x
Train steps
50k
Channels
Learning rate
5e-5
Depth
Batch size per GPU GPUs
32
Attention scales
1,0.5,0.25
8
Number of heads
8
GPU-type
A6000 (48GB)
Head dim
64
Loss type
MSE
Channel multiplier
12244,
Optimizer
AdamW
CA Conditioniug
Inference
Context dimension
1024 77
Sampler
PLMS
Sequence length
#denoising steps
50
Table 7: Hyperparameters for Layout2Vid.
Samplers
FVD (J)
FID (4)
CLIPSIM (t)
DDIM
508
12.52
0.2848
PLMS
550
12.22
0.2860
Table Comparison between PLMS and DDIM sampler on MSR-VTT.
example videos generated from the text prompt 'A horse running" and videos generated from the video plan modified by users (i.e., changing the object sizes and backgrounds).
frame
frame
frame
frame
frame
frame 16
Make smaller
Editing
Original prompt: 'A horse running"
Edit 2: Add "grassland" background
Add "night street" background
Figure 10: Video generation examples for human-in-the-long editing. Users can modify the video plan (e-g., add/delete objects, change the background and entity layouts, etc.) to generate customized video contents. Given the same text prompt "A horse running", we provide visualizations with smaller horse and different backgrounds (i.e., "night street" and "grassland").
Experiment Setup
We provide additional details on our experiment setups (Sec. in the main paper) below.
F1 Evaluated Models
We compare VIDEODIRECTORGPT. to 6 popular T2V generation models, NUWA NUWA (Wu
et al., 2022b), CogVideo (Hong et al., 2022), VideoLDM (Blattmann et al., 2023b), Mag- icVideo (Zhou et al., 2022), Make-A-Video (Singer et al., 2023), and ModelScope (Wang
20

---

Published as a conference paper at COLM 2024
et al., 2023b). Since NUWA, VideoLDM, MagicVideo, Make-A-Video, and CogVideo (En- glish) are not publicly available, we primarily compare our VIDEODIRECIORGPT with Mode and present comparisons with the other models on the datasets for which their papers have provided results. ModelScope is the closest baseline to our framework among all these models, because our Layout2Vid utilizes its frozen weights and only trains small set of new parameters to add spatial control and temporal consistency across multiple scenes.
F2 Prompts for Single-Scene Video Generation
For single-scene video generation, we conduct experiments with VPEval Skill-based prompts to evaluate layout control (Cho et al., 2023b), ActionBench-Direction prompts to assess object dynamics (Wang et al., 2023c), and MSR-VTT to cover diverse open-domain scenes (Xu et al., 2016).
VPEval Skill-based prompts evaluate different object-centric layout control skills in text- to-image/ /video generation. We randomly sample 100 prompts for each of the four skills: Object (generation of a single object), Count (generation of specific number of objects), Spatial (generation of two objects with a spatial relation; e.g., left right/ /aboee/ /below) and Scale (generation of two objects with a relative scale relation; e.g., bigger /ngeer/smalerr smaller/same)) same).
prompts evaluate the action dynamics (object movement direc- tions) in video language models. We prepare the prompts by sampling video captions from OctionBench-SSV.. (Wang et al., 2023c) and balancing the distribution of movement directions. Concretely, we select captions from the Action Bench-BSyyee validation split that include phrases like 'right to left' or 'left to right' (egg. 'pushing glass from left to right'), which are common phrases describing movement directions in the captions. Then we augment these prompts by switching the directions to each of four directions: 'left to right', 'right to left', 'top to bottom', and 'bottom to top' to create 100 prompts for each direction. We call the resulting 400 prompts as CirtoonBenc--Diectttio prompts. These prompts ensure balanced distribution of movement directions while maintaining diversity in objects.
MSR-VTT is an open-domain video captioning dataset, which allows us to check our Layout2Vid maintains the original visual quality and text-video alignment performance of the ModelScope backbone after integration of the layout/ movement control capabilities. The MSR-VTT test set comprises 2,990 videos, each paired with 20 captions. Following VideoLDM (Blattmann et al., 2023b), we sample one caption from the 20 available captions for each video and use the 2,990 corresponding generated videos for evaluation.
F3 Prompts for Multi-Scene Video Generation
For multi-scene video generation, experiment with two types of input prompts: (1) list of sentences describing events - Activity Captions (Krishna et al., 2017) and Coref-SV prompts based on Pororo-SV (Li et al., 2019b) and (2) a single sentence from which models generate multi-scene videos - HiREST (Zala et al., 2023).
ActivityNet Captions dense-captioning dataset designed for detecting and describing multiple events in videos using natural language. For the multi-scene video generation task, we use 165 randomly sampled videos from the validation split and use the event captions as input for ModelScopeT2V and our VideoDirecToRGPT When calculating object consistency, we find the subject of the first event caption via spaCy dependency parser (Honnibal & Montani, 2017) and check its appearance in multiple scenes.
Coref-SV is a new multi-scene text description dataset that we propose to evaluate the consistency of object appearances across multi-scene videos. We prepare the Coref-SV prompts by augmenting the Pororo-SV dataset (Li et al., 2019b; Kim et al., 2017), which consists of multi-scene paragraphs from the "Pororo the Little Penguin" animated series. To evaluate the temporal consistency of video generation models trained on real-world videos, extend its original animation characters (e.g., Pororo) to humans and common animals and examine their appearance across different scenes. Concretely, we sample 10 episodes, each consisting of multiple scenes (6.2 scenes per episode average). Then,
21

---

Published as a conference paper at COLM 2024
we replace the first appearance of a character with one of the predefined 10 real-world entities (egg, person/ dog, etc.) and replace the remaining appearances of the character with pronouns (e.g,, hte//he/ite hte//tt/etc)) In total, we obtain 100 episodes (=10 episodes x 10 entities) in Coref-SV. In order to generate visually consistent entities, the multi-scene video generation models would need to address the co-reference of these target entities across different scenes. We use the final scene descriptions as input for both ModelScope112V and VIDEODIRECTORGPT When calculating object consistency, we use the selected entity as the target object.
HiREST provides step annotations for instructional videos paired with diverse 'How to' prompts (e.g., a video paired with 'how to make butter biscuits' prompt is broken down into a sequence of short video clips of consecutive step-by-step instructions). For the multi-scene video generation task, we employ 175 prompts from the test splits, where we only include the prompts with step annotations, to ensure that it is possible create multi-scene videos from the prompts. Note that instead of providing list of scene description sentences like in ActivityNeet Captions/ Coreff-V, we only give the single high-level 'How to' prompt and let the models generate multi-scene video from it. In VIDEODIRECTORGPY our LLM can automatically generate the multi-scene video plan and video from the input prompt. In contrast, for the ModelScopeT2V baseline, we help the model understand the different number of scenes to generate by pre-defining the number of scenes N, and independently generate N videos by appending the suffix "step n n/N" the prompt for n-th scene (egg,, "Cook Beet Greens, step 1/10"). To ensure that our VIDEODIRECTORGE ModelScope videos are equal in length, we use the same number of scenes generated by our LLM during the planning stage for ModelScopeT2V. As mentioned in main paper Sec. 5.2, we also try an ablation with giving ModelScopeT2V the LLM generated prompts instead of just "step n/N".
videos and
G Human Evaluation Details
We provide details of our human evaluation and error analysis setups, as well as the error analysis results described in Sec. 4.
Human evaluation details. We conduct human evaluation study on the multi-scene videos generated by both our VIDEoDIRECTORGP and Mo on the Coref-SV dataset. Since we know the target entity and its co-reference pronouns in the Coref-SV prompts, we can compare the temporal consistency of the target entities across scenes. We evaluate the human preference between videos from two models in each category of Quality, Text- Alignment, and Object Consistency:
Quality: it measures how well the video looks visually.
Text-Video Alignment: it assesses how accurately the video adheres to the input sentences.
Object Consistency: it evaluates how well the target object maintains its visual consistency across scenes.
We show 50 videos to ten crowd-annotators from AMT to rate each video (28 unique annotators, ten annotators rate each video) and calculate human preferences for each video with average ratings. All videos are randomly shuffled such that annotators do not know which model generated each video. To ensure high-qualite annotations, we require they have an AMT Masters, have completed over 1000 HITs, have a greater than 95% approval rating, and are from one of the United States, Great Britain, Australia, or Canada (as our task is written in the English language). We pay annotators $0.10 to evaluate video (roughly $12-14/hr).
Step-by-step error analysis details. We conduct an error analysis on each step of our single sentence to multi-scene video generation pipeline for HiREST prompts. We analyze
Ammazon Mechanical Turk: https: // www...turk.com
22

---

Published as a conference paper at COLM 2024
the generated multi-scene text descriptions, layouts, and entity/ /backgroudd consistency groupings to evaluate our video planning stage and the final video to evaluate the video generation stage.
Multi-Scene Text Descriptions Accuracy: we measure how well these descriptions depict the intended scene from the original prompt (e-g,, the original prompt is "Make buttermilk biscuits" the descriptions should describe the biscuit-making process and not the process for pancakes).
Layout Accuracy: we measure how well the generated layouts showcase scene for the given multi-scene text description (e.g., the bounding boxes of ingredients should go into a bowl, pan, etc. instead of randomly moving across the scene).
Entity/Background Consistency Groupings Accuracy: we measure how well the gener- ated entities and backgrounds are grouped (e.g., entities/ entittess/bcckgroumt backgrounds that should look consistent throughout the scenes should be grouped together).
Final Video Accuracy: we measure how well the generated video for each scene matches the multi-scene text description (e.g., if the multi-scene text description is "preheating an oven", the video should accurately reflect this).
We ask an expert annotator to rank the generations (multi-scene text description, layouts, etc.) on a Likert scale of 1-5 for 50 prompts/ viieeos. Analyzing the errors at each step enables us check which parts are reliable and which parts need improvement. As single prompt/viden can contain many scenes, to simplify the process for layout and final video evaluation of a prompt/viie / we sub-sample three scene layouts and corresponding scene videos and average their scores to obtain the "Layout Accuracy" and "Final Video Accuracy."
Step-by-step error analysis results. As shown in Table 9, our LLM-guided planning scores high accuracy on all three components (up to 4.52), whereas the biggest score drop happens the layout-guided video generation (4.52 + 3.61). This suggests that our ViDEODIRECTORGP1 could generate even more accurate videos, once we have access to stronger T2V backbone than ModelScopeT2V.
Stage 1: Video Planning GPT-4)
Stage Video Generation (with LayoutZVid)
Multi-soene Text Descriptions (1)
Layeuts (T)
Entity / Backgroudd Consistency Groupings (1)
Final Video (T)
4.92
4,89
4.52
3.61
Table 9: Step-wise error analysis of ViDeoDIRECTORGP" video generation pipeline HiREST prompts. We use Likert scale (1-5) to rate the accuracy of the generated compo- nents each step.
H Ablation Studies
In this section, we provide ablation studies on our design choices, including using dif- ferent LLMs for video planning, the number of layout-guided denoising steps, different embeddings for layout groundings, and layout representation formats.
H.1 Video Planner LLMs: GPT-4, GPT-3.5-turbo, and LLaMA2
We conducted an ablation study using the open-sourced LLaMA2-7B and LLaMA2-13B models (Touvron et al., 2023b) well as GPT-335-turbo in the video planning stage MSR-VTT and SirionBeenc-Diretttiin
Evaluation settings. this study, we evaluated LLaMA2 in both fine-tuned and zero-shot settings. For the zero-shot evaluation, used the same number of in-context examples (i.e., 5 examples) for both GPT-3.5-turbo and GPT-4, but only 3 in-context examples for the LLaMA2 models due to their 4096 token length limits. To collect data for fine-tuning
23

---

Published as a conference paper at COLM 2024
LLaMA2, randomly sampled 2000 prompts from Web (Bain et al., 2021) and generated layouts with GPT-4 using the same prompts as described in our main paper. We fine-tuned the LLaMA2 models for 1000 steps with learning rate of 2e-4 and cosine scheduler. In addition to the visual quality (FID/FVD) and video-text alignment (CLIPSIM) scores MSR-VT USR-VTT and movement direction accuracy score on ActionBench. Bench-Drrrcciionn, also report the number of samples whose output layouts can be successfully parsed (e.g,, #Samples). We consider the output valid if it contains 9 frames, and each frame includes at least one object with bounding box layout. Videos are generated with 0.1 for MSR-VTT and 0.2 for ActionBench-Dirrettin
H.2 Entity Grounding Embeddings: Image v.s. Text
As discussed in Sec. 3.2 in our main paper, we compare using different embeddings for entity grounding on 1000 randomly sampled MSR-VTT test prompts. As shown in Table 10, CLIP image embedding is more effective than CLIP text embedding, and using the CLIP image-text joint embedding yields the best results. Thus, we propose to use the image embeddings for the default configuration.
MSR-VTT
Coref-SV
Entity Grounding
FVD (4)
FID (1)
CLIPSIM (t)
Consistency (%)
Image Emb.
18.38
0.2834
42.6
Text Emb.
875
23.18
0.2534
36.9
Image+ Text Emb. (default)
606
14.60
0,2842
42.8
Table 10: Ablation of entity grounding embeddings of our Layout2Vid on MSR-VTT and Coref-SV.
H.3 Layout Control: Bounding Box Center Point
In Table 11, we compare different layout representation formats on 1000 randomly sampled MSR-VTT test prompts. We use image embedding for entity grounding and 0.2 for layout control. Compared with no layout ('w/o Layout input') center point-based layouts (without object shape, size, or aspect ratio), the bounding box based layout guidance gives better visual quality (FVD/FL /FDD) and text-video alignment (CLIPSIM).
Layout representation
FVD (1)
FID (1)
CLIPSIM (t)
w/o layout input
639
15.28
0.2777
Center point
816
18.65
0.2707
Bounding box (default)
606
14.60
0.2842
Table 11: Ablation of layout representation of our VIDEODIRECTORGGT on MSR-VTT.
H44 Trainable Layers: Gated Self-Attention Only v.s. Entire Guided 2D Attention
Training settings. In our Layout2Vid, only the MLP layers for grounding tokens and the gated self-attention layers are trained, accounting for 13% the total parameters. We conducted an ablation study to unfreeze more parameters from the ModelScope backbone. Specifically, all layers in the guided 2D attention module, such as self-attention, gated self-attention, and cross-attention, are made trainable, which constitutes 27% of the total parameters. Starting from the model trained in our main paper, we further train this variant with more trainable parameters for 50k steps, keeping all other training hyper- parameters constant.
Result analysis. As shown in Table 12, unfreezing all parameters in the guided 2D at- tention module results in significantly worse performance compared to training only the additional gated self-attention layers. This decline in performance occurs because the tem- poral attention and temporal convolution layers, which inactive during our image-data
24

---

Published as a conference paper at COLM 2024
Trainable Layers
FVD (1)
FID (4)
CLIPSIM (1)
Entire Guided 21 Attn (GatedSA SelfAttm CrossAttm) GatedSA only (default)
1626
70.47
(1.2231
550
12.22
0.2860
Table 12: Comparison of only updating parameters in gated self-attention (GatedSA) between updating all parameters in guided 2D attention (Entire Guided 2D Attn) of our VIDEODIRECTORGPT MSR-VTT.
training, cause misalignment in the spatial and temporal layers during inference when the temporal layers are activated.
I Additional Experiments
1.1 Comparison on UCF-101
In addition to the MSR-VTT dataset for open-domain single-scene video generation, we also conduct experiments for the UCF-101 dataset (Soomro et al., 2012). Since ModelScopeT2V does not report their evaluation results on UCF-101 dataset in their technical report, we use their publicly released checkpoint and run evaluation on the same 2048 randomly sampled test prompts as used our VideoDirectorGPT evaluation. Compared with ModelScope T2V, achieve significant improvement in FVD (ModelScopeI2V 1093 vs. Ours 748) and competitive performance addition, notice that MV is not good at video generation this UCF-101 dataset, which is also observed in other recent works (He et al., 2023). We could expect our VideoDirectorGPT to perform better with stronger T2V backbones.
the Inception Score (ModelScope 19.49 vs. Ours 19.42). In
UCF-101
Method
FVD (1)
IS (t)
Different arch / Training data
LVDM
917
VidVideo Video (Chinese)
751
23.55
Sigeideo (English)
701
25.27
MagicVider
699
VideoLDM
550
33.45
Make-A-Video
367
33.00
Same video backbone & Test prompts
ModeIScope12y*
1093
19.49
VIDRODIRECTORGGT 748
19.42
Table 13: Evaluation results on UCF-101. ModelScopeT2y*:: Our replication with 2048
randomly sampled test prompts.
J Additional Visualization and Qualitative Examples
J.1 Visualization of Different Layout-Guided Denoising Steps.
Balancing layout control strength with visual quality. During video generation, we use two-stage denoising in Layout2Vid following et al. (2023), where we first use layout- guidance with Guided 2D attention for & N steps, and use the denoising steps without Guided 2D attention for the remaining (1 - a) * N steps, where N is the total number [0,1] is the ratio of layout-guidance denoising steps. In our additional analysis (see main paper Table 4), we find that a high & generally increases layout control but could lead to lower visual quality, which is also consistent with the finding in Li
of denoising steps, and
25

---

Published as a conference paper at COLM 2024
frame 1
frame 8
frame 16
frame
frame
frame
alpha=0.1
alpha=0.3
alpha=0.7
shutters
shu
ck
erstock
alpha=1.0
Visualization of objects with different strength of layout control (different values). The prompts are "A dog moving from right to left" and "À AA car moving from left to right" respectively. Increasing the & value can make the objects better following bounding box layouts. An value between 0.1 0.3 usually gives videos with best trade-off between visual quality and layout control.
Figure 11:
Frame
Frame
Frame 16
Frame
Frame
Frame 16
Figure 12:
Generated layouts from prompts that require physical understanding: "A stone thrown into the sky" (left) and "A car is approaching from a distance" (right).
et al. (2023). By default, use = 0.1 and N 50 denoising steps. We also explore using the LLM to determine the a value within the range [0, 0.3) during the video plan generation (see main paper Sec. 5.3 for details).
Visualization and Analysis. Here, we provide visualizations of objects with prompts that involve explicit layout movements to illustrate the effect of changing the a hyper-parameter As can be seen in Fig. 11, small value (e.g., 0.1) can provide guidance for object movement, enabling the Layout2Vid to generate videos that do not strictly adhere to bounding box layouts. This approach fosters better creativity and diversity in the generated videos, and enhances visual quality well as robustness to LLM -LM--prddced layouts. On the other hand, an value of 0.3 or higher is sufficient to control objects to follow bounding box trajectories. In conclusion, for prompts requiring explicit movement control, larger value typically results in better performance (as demonstrated by the results from ActionBench- Seanchh---eeennss in Table 4 of our main paper). Meanwhile, for prompts that do not require large movements, a smaller a value generally leads to improved visual quality (as indicated by the results from MSR-VTT in Table 4 in the main paper).
26

---

Published as a conference paper at COLM 2024
Frame
Frame
Frame 16
Frame
Frame 8
Frame 16
Frame
Frame 8
Frame 16
Figure 13: Generated videos from movement prompts with a static object (bottle) and objects that can naturally move (airplan/bon /bot).. We use prompts "A {bottle/ airplane /boat) moving from left to right."
Figure 14: Multi-scene video generation example from prompt: "A boy playing in sand and flying kites." "
J.2 Additional Qualitative Examples
Prompts requiring understanding physical world. Fig. 12 shows that our GPT-4 based video planner can generate object movements requiring an understanding of the physical world such as gravity and perspective.
Object movement prompts with different types of objects. In Fig. 13, we show additional examples for single-scene video generation with prompts involve movements. For static objects (e.g., bottle), movement is often depicted via the camera. For objects that can naturally move (e.g., an airplane and a boat), the videos show the movements of objects.
Generated Scenes
Entily Grounding
<B>
plush
kitchen
window
Time
That Input
<S> = "wiite cat"
Input
Figure 15: Video generation examples with text-only and image+text inputs. Users can flexibly provide either text-only or image+text descriptions place custom entities when generating videos with VideoDIRecTORGPP entity grounding examples, the identities of the provided entities are well preserved across multiple scenes.
For both text and image+text based
27

---

Published as a conference paper at COLM 2024
Generating videos with custom image exemplars. In Fig. 15, we demonstrate that our Layout2Vid can flexibly take either text-only or image+text descriptions as input to generate multi-scene videos with good entity consistency.
Multi-scene video generation with human appearance. In Fig. 14, we show an additional example of multi-scene video generation with human appearance. Our VIDEODIREC- TORGPT can well preserve the boy's appearances across scenes.
VPEval Skill-based. Fig. shows that LLM-generated video plan successfully guides the Layout2Vid module to accurately place objects in the correct spatial relations and to generate the correct number of objects. In contrast, ModelScope112) fails to generate 'pizza' in the first example and overproduces the number of frisbees in the second example.
paza
elephant
ModeiBcopeT2V Frame
VideeDineclorP1
Frize1
Frama
Form
Examplo
four frisbees
Figure 16: Video generation examples on VPEval Skill-based prompts for spatial and count skills. Our video plan, with object layouts overlaid, successfully guides the Layout2Vid module to place objects in the correct spatial relations and to depict the correct number of objects, whereas ModelScope12V fails to generate 'pizza' in the first example and overproduces the number of frisbees in the second example.
Pushing
right
ModelticapaT2V
VideoBreciorGPP (Ours) Foreh
Fooe
Foyas
Exmmpe
Pushing pear right
Examle
Figure 17: Video generation examples
CirionBeenc-Direttion prompts. Our video plan's object layouts (overlaid) can guide the Layout2Vid module to place and move the 'stuffed animal' and 'pear' in their correct respective directions, whereas the objects in the ModelScope12V videos stay in the same location or move in random directions.
ActionBench-ddrrecton. Fig. 17 shows that our LLM-generated video plan can guide the Layout2Vid module to place the 'stuffed animal' and the 'pear' in their correct starting positions and then move them toward the correct end positions, whereas the objects in the ModelScope videos stay in the same location or move in random directions.
Coref-SV. Fig. 18 shows that our video plan guide the Layout2Vid module to generate the same dog and maintain snow across scenes consistently, whereas ModelScope generates different dogs in different scenes and loses the snow after the first scene.
28

---

Published as a conference paper at COLM 2024
ModetScapeTZV
VidroDimedeerGPt (Ours)
Farm
Frams
Soone
snowing outAls.
Scone
and dancing
Scare 4
ag
Figure 18: Video generation examples on Coref-SV prompts. Our video plan's object layouts (overlaid) can guide the Layout2Vid module to generate the same brown dog and maintain snow across scenes consistently, whereas MV generates different dogs in different scenes and loses the snow after the first scene.
"How To Make Strawberry Surpris
ModeiScope T2V
VideoDirecterPPT (Ours) Prrm
Fiars
Fare
Frares
Flans
Soane
Make Stuobery
Scone 6
Stowbberyy
and spon Mooe scop canan
Scene
aftivice
Figure 19: Video generation examples HiREST prompts. Our VIDEODRECTORGPT generates detailed video plan that properly expands the original text prompt, ensures accurate object bounding box locations (overlaid), and maintains consistency of the person across the scenes. ModelScope only generates strawberries, without strawberry surprise.
HiREST. Fig. 19 shows that our LLM can generate step-by-step video plan from single prompt, and our Layout2Vid can generate consistent videos following the plan. Our ViDEODIRECTORGPT breaks down the process and generates complete video showing how to make a strawberry surprise (a type of dessert consisting of vanilla ice cream and strawberries). ModelScope repeatedly generates strawberries.
29

---

Published as a conference paper at COLM 2024
K Limitations
While our framework can be beneficial for numerous applications (eg,, user- controlled/ / hhuman-in-thellopp video generation/ manipulation and data augmentation), akin to other video generation frameworks, it can also be utilized for potentially harmful purposes (e.g., creating false information or misleading videos), and thus should be used with caution in real-world applications (with human supervision, e.g., as described in Ap- pendix - human-in-the-loop video plan editing). Also, generating video plan using the strongest LLM APIs can be costly, similar to other recent LLM-based frameworks. We hope that advances in quantization/ distillation and open-source models will continue to lower the inference cost of LLMs. Lastly, our video generation module (Layou(2Vid) is based on the pretrained weights of a specific T2V generation backbone. Therefore, we face similar limitations to their model, including deviations related to the distribution of training datasets, imperfect generation quality, and only understanding the English corpus.
30


Reviews:
[{"review_id": "FHx692esDD", "reviewer": "Reviewer_TCNG", "summary": "This paper introduces VideoDirectorGPT, which leverages large language models (LLMs) to derive object/layout plans for multi-scene long video generation. Specifically, the video planner expands the input caption into scene descriptions, including entities with bounding boxes and the overall background. Then, the proposed Layout2Vid can control explicit spatial layout and temporal coherence during video generation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback! \n\n**W1: Our novelty, and comparison with GLIGEN**\n\nPlease see our response to reviewer **ts3J**’s **W2** for more details.\n\n**W2: Is the multi-scene video only constrained to 16 frames?**\n\nWe would like to clarify that the generated multi-scene video consists of multiple single-scene videos, while each single-scene video contains 16 frames. The video plan can contain as many scenes as is needed for the video. \n\nIn addition, the number of frames in each single scene (16 here) are dependent on the video generation model (ie, ModelScopeT2V), which can be easily extended with backbones that can generate more frames (25 frames in SVD, and even 221 frames in open-sora-plan). Some training-free frame-extension methods (e.g., FreeNoise [1]) can be seamlessly integrated into our framework.\n\n**W3: Failure cases**\n\nOur method inherits two shortcomings of bounding-box-based control.\n\nFirstly, some objects might not follow the bounding box control well when there are too many overlapping bounding boxes. When multiple objects are in the same overlapping bounding box area, the CLIP image embeddings from the cropped object regions share pixels, making it difficult to disentangle their identities. As we can see from the gifs under the folder **./Failure Cases** in this link: https://anonymous.4open.science/r/VideoDirectorGPT_COLM-CDC8. The “fresh lemon” and “sliced onion”, which have relatively small bounding box sizes, are not following the box control well. Adopting layer-wise object generation (e.g., LayerDiffuse [2]) could be a potential solution to this problem.\n\nSecondly, when a bounding box does not move across frames (e.g., box of background), it is hard to make sure the background is kept static. As shown in Fig. 13, it’s hard for us to generate a moving bottle with a static background, and a moving boat with a completely static background. The static/dynamic of the background are usually determined by the prior knowledge of the video generation model. Adopting methods that can explicitly handle camera movement (e.g., CameraCtrl [3]) could be a potential solution.\n\n[1] Qiu, et al. \"Freenoise: Tuning-free longer video diffusion via noise rescheduling.\" arXiv preprint arXiv:2310.15169 (2023)\n\n[2] Zhang, et al. \"Transparent Image Layer Diffusion using Latent Transparency.\" arXiv preprint arXiv:2402.17113 (2024)\n\n[3] He, et al. \"Cameractrl: Enabling camera control for text-to-video generation.\" arXiv preprint arXiv:2404.02101 (2024)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and easy to follow.\n- The idea of adopting LLMs to derive scene description/layout is well-motivated, which has the potential to benefit the planning of long video generation.\n- They provide detailed ablation studies and comprehensive visualized examples in the Appendix.", "reasons_to_reject": "- Since there are several previous works [1,2] adopting LLMs to improve the layout planning of T2I, the novelty of this project can be the issue. Furthermore, the proposed Layout2Vid is an extension of the existing GLIGEN [3]. I am afraid that this project can not achieve the bar of a top-tier conference.\n- Though the proposed video planner can craft the plan, Layout2Vid only generates 16 frames (2-4 seconds) as the output video. Can this short clip actually contain multiple scenes (e.g., different entities and backgrounds)? I am worried about the practicality of the proposed framework.\n- Apart from those successful cases, failure examples are also crucial to discuss, which can guide future research on this topic.\n\n**Reference**\n- [1] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models\n- [2] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models\n- [3] GLIGEN: Open-Set Grounded Text-to-Image Generation", "questions_to_authors": "Please see Reasons To Reject", "extra_scores": {}}, {"review_id": "HNAyUhyqJS", "reviewer": "Reviewer_HYoe", "summary": "This papar propose video-director-GPT, a framework to generate consistent multi-scene videos.\nThe method includes a GPT-4 planning process and a layout2vid module to generate video based on the plans.\nThe experiments show that the quality is improved.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer HYoe, thank you for your valuable feedback! We address your concerns with clarifications as follows.\n\n\n**W1: experiments with other LLMs**: \nWe would like to remind you that we have provided detailed ablation experiments on different language models and whether to fine-tune them in the section H.1 Video Planner LLMs: GPT-4, GPT-3.5-turbo, and LLaMA2. \nPlease let us know if you have any further questions about ablations of different LLMs, we are very glad to address.\n\n\n**W2: is exact match weakening entity semantics?**\n\nThat is a great question! We would like to point out that when we ask GPT-4 to generate entity names, the rich semantics (e.g., object attributes) can be well maintained. For example, as shown in Fig 2 and Fig 5, the entities generated by GPT-4 contain “a fluffy Siamese cat”, “a plush beige bed”, “woman in a green apron”, “modern kitchen counter”, and so on. These texts, instead of just “cat”, “bed”, “woman”, and “kitchen”, are used to construct object embeddings that are shared across all scenes.\nIn addition, if the user wants more fine-grained control over object appearance, they can provide a reference image in addition to text descriptions (as shown in Fig. 6). For example, if the same woman changes from a \"woman in a green apron\" in scene 1 to a \"woman in a red T-shirt\" in scene 2, we can make such changes while maintaining the appearance of the woman by editing the reference image from a \"green apron\" to a \"red T-shirt\" using off-the-shelf image-editing tools (e.g., Instructpix2pix [1]).\n\n\n[1] Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. \"Instructpix2pix: Learning to follow image editing instructions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the idea and the method is clear and well-motivated\n2. the evalution is strong, including automatic and annoatation evaluations.", "reasons_to_reject": "1. the experiments are somehow limited, I understand the fact that running LLM servers might be difficult, still, only using GPT-4 as the planner without any other alternative LLMs might weaken the results.\n2. If I'm not mistaking any details, the GPT-4 generated plans are later grouped into a shared representation based on exact match. Is this a method that weakens the power of LLMs? The reasoning, continuous semantics are all discarded then?", "questions_to_authors": "Please see the comments above.", "extra_scores": {}}, {"review_id": "s77s15AYPT", "reviewer": "Reviewer_ts3J", "summary": "This paper proposes VIDEODIRECTORGPT for video generation. It comprises two stages: the first stage is an LLM planner that generates the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities with the given prompt; the second stage is a video generator mainly based on the weights of ModelScopeT2V. The authors claim their experiments demonstrate that their proposed framework improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. Ablation studies confirm the effectiveness of each component of the proposed framework.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! \n\n**W1: Supplementary materials**\n\nWe would like to bring your attention to our “supplementary materials” below the abstract. Please feel free to let us and ACs know if you didn't find the supplementary materials in your openreview system.\n\n**W2: Novelty/Contributions**\n\nWe would like to clarify that LLM-grounded Diffusion [1] and our paper handle different tasks with different methods. Firstly, [1] only covers T2I generation, while our work proposes a new framework for multi-scene T2V generation. Secondly, [1] is about inference-time optimization, while our method focuses on model training without inference-time optimization.\n\n**In addition, we would like to elaborate our novelty and contributions:**\n\n- **1.** We introduce **the first multi-scene video generation framework by leveraging the LLM knowledge for planning and ensuring consistency across scenes.**\n\n- **2(a) Video Planner:** Our video planner introduces a complex multi-component video plan (scene descriptions, entity descriptions and layouts, and consistency groupings) for multi-scene video generation.\n\n- **2(b) Layout2Vid:** We propose to insert layout control layers (similar to GLIGEN but with enhanced cross-scene text/image entity representation) within frozen T2V backbone, which supports multi-scene consistency control and image-only training, while preserving the knowledge of the T2V model.\n\n- **3. We provide comprehensive qualitative/quantitative experiments** including novel evaluation metrics (movement direction accuracy / entity consistency) / new datasets (ActionBench-Direction/Coref-SV) for video generation, as recognized by all other reviewers.\n\n**W3: Video examples**\n\nWe would like to bring your attention to the html file in our supplementary materials, which includes many visualization examples in **.gif** format. Besides, we provide human evaluation in Section 5.4. We also further increased the number of human evaluators from 5 to 10, as suggested by reviewer Ycer (see the response to reviewer **Ycer W3** for details).\n\n**W4: LLM ablation**\n\nWe would like to bring your attention to Appendix section H.1, where we have provided detailed ablation experiments on different language models: GPT-4, GPT-3.5-turbo, and LLaMA2. \n\n**Considering that 3 out of the 4 reasons to reject were already addressed in our appendix/supplementary materials, and we have elaborated our novelty/contributions, we kindly request to raise the score accordingly. Thanks!**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to understand.\n\n2. The experiments are sufficient in many aspects, including single-scene and multi-scene video generation, with metrics like FID, FVD, IS, and human evaluations.", "reasons_to_reject": "1. The authors mentioned that \"We provide code and generated video examples in the supplementary material,\" but I did not find where the supplementary material is. Thus, I am concerned about the completeness of this work, as well as the reproducibility and impact of the proposed method due to the lack of related code and models.\n\n2. The application is not novel at all. It mainly relies on GLIGEN and ModelScopeT2V. One image version work is [1]. Thus, this work is a little bit below the bar of CoLM.\n\n3. For a video generation paper, related videos should be appended to demonstrate the real performance. As we all know, some quantitative evaluation metrics are inconsistent with real performance, as many studies have proved.\n\n4. There is a lack of related ablation experiments on different language models and whether to fine-tune them. At an LLM conference, the authors should focus more on the LM itself instead of video generation.\n\n\n[1] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models, https://llm-grounded-diffusion.github.io/", "questions_to_authors": "Refer to weaknesses. \n\nThe authors' response addresses my first reject reasion, which is appreciated. However, after carefully checking the code, I maintain my reservation regarding the heavy reliance on GLIGEN and ModelScopeT2V, which dominates the described method. Moreover, the emphasis on video rather than LM suggests that this work falls short of the expectations set by the CoLM standards.", "extra_scores": {}}, {"review_id": "7F7oh4e4bh", "reviewer": "Reviewer_tgxZ", "summary": "The authors proposed a two-stage text-to-video pipeline, VideoDirectorGPT, with two major components: (1) a video planner and (2) a video generator. The video planner takes a video text prompt to generate video plans using GPT4, which also groups the same entities across scenes and generates layout (bounding box) of entities in addition to background and scene description. THe video planner is finetuned from ModelScopeT2V for training temporally consistent entity grounding component. Note that only 13% of total model weights are updated for this component.\n\nExperiment results show that the proposed VideoDirectorGPT outperforms ModelScopeT2V in single-scene and multi-scene settings on both automatical and human evaluation. Overall, the authors did a great job evaluating the video generator component.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuing our comprehensive evaluation, the technical novelty of our proposed method, and our user-friendly two-stage framework design! \n\nAs suggested, we will include examples of full video planner output in the appendix in our camera-ready version. Please let us know if you have any other questions or concerns!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Comprehensive evaluation of generated video generator, including automatic evaluation and human evaluation with ablation studies. \n2. Proposed method can take not only textual input but also image + text input.\n3. The two-stage pipeline allows users to modify each of the components, such as background or bounding boxes.", "reasons_to_reject": "Not that I can think of. Basically, all of my questions are answered in the appendix.", "questions_to_authors": "Comments/Suggestions:\nIt would be helpful to have an example of full video planner output in the appendix.", "extra_scores": {}}, {"review_id": "dMCeWIDW8b", "reviewer": "Reviewer_Ycer", "summary": "This paper presents VideoDirectorGPT, a multi-scene video generation framework focusing on generation consistency, which leverages the knowledge embedded in LLMs for video content planning and grounded video generation. VideoDirectorGPT consists of two stages. In the first stage, given a single text prompt, GPT-4 is employed as a video planner to craft a video plan, a multi-component video script with multiple scenes to guide the downstream video synthesis process. In the second stage, a novel grounded video generation module named Layout2Vid is proposed to render videos based on the generated video plan. Layout2Vid is built upon the off-the-shelf T2V generation model ModelScopeT2V, thus enabling Layout2Vid to be trained solely on layout-annotated images. Experiments demonstrate that VideoDirectorGPT can generate high-quality videos with good consistency.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback!\n\n**W1: Comparison with more baselines**\n\nAs requested, we include **both qualitative and quantitative** comparisons with additional baselines.\n\nFirstly, we compare VideoDirectorGPT quantitatively on layout and object movement control abilities with three more strong baselines in video generation - AnimateDiff, I2VGen-XL, and SVD. The table columns below are same as Table 1 in the main paper. Among the new baselines, I2VGen-XL achieves the highest overall accuracy, while VideoDirectorGPT still outperforms all the baselines including I2VGen-XL by a large margin.\n\n||||||||\n|-|-|-|-|-|-|-|\n|Method|VPEval Object|VPEval Count|VPEval Spatial|VPEval Scale|VPEval Overall Acc. (%)|ActionBench Movement Direction Acc (%)|\n|ModelScopeT2V|89.8|38.8|18.0|15.8|40.8|30.5|\n|AnimateDiff|96.7|52.7|22.5|15.5|46.8|29.0|\n|I2VGen-XL|96.5|62.0|35.2|23.7|54.3|35.2|\n|SVD|93.1|46.7|29.2|15.0|45.7|30.7|\n|VideoDirectorGPT (ours)|**97.1**|**77.4**|**61.1**|**47.0**|**70.6**|**46.5**|\n\nSecondly, since Phenaki does not released official code, we compare our multi-scene video generation results qualitatively with Phenaki's. The generated gifs from our VideoDirectorGPT are under folder **./Phenaki Prompts** in this link: https://anonymous.4open.science/r/VideoDirectorGPT_COLM-CDC8. Compared with Phenaki, our model is able to generate objects (astronaut, teddy bear) with comparable consistency while equipped with additional layout control.\n\n\n**W2: Technical novelty**\n\nPlease see our response to reviewer **ts3J**’s **W2** for more details.\n\n\n**W3: Number of human annotators**\n\nThanks for this question! We are happy here to further increase the number of human evaluators from 5 to 10. We listed our previous result with 5 human annotators and the new result with 10 (previous 5 + new 5) annotators below.\n\nAs we can see, except for the percentage of Ties for Text-Video Alignment, which decreases from 24% to 6%, the relative preference between the two models in the three criteria remains very similar. In both results, our VideoDirectorGPT consistently outperforms ModelScopeT2V in all metrics.\n\n\n**Previous** result with **5** human annotators\n|||||\n|-|-|-|-|\n||Ours|ModelScopeT2V|Tie|\n|Text-Video Alignment|**54**|22|24|\n|Quality|**52**|30|18|\n|Object Consistency|**62**|28|10|\n\n\n**New** result with **10** human annotators\n|||||\n|-|-|-|-|\n||Ours|ModelScopeT2V|Tie|\n|Text-Video Alignment|**58**|36|6|\n|Quality|**50**|34|16|\n|Object Consistency|**62**|28|10|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using LLMs as a video plan generator is straightforward and technically sound, and empirically works quite well.", "reasons_to_reject": "- VideoDirectorGPT is only compared against ModelScopeT2V. The authors should also benchmark other state-of-the-art baseline alternatives such as Phenaki.\n\n- The technical contribution of the paper is limited since the main video generation module, Layout2Vid, is built upon the off-the-shelf T2V generation model ModelScopeT2V.\n\n- Only 5 crowd-annotators from AMT were used, which makes the human evaluation results less reliable.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for planning video scenes using LLMs. Reviewers had several concerns regarding novelty and evaluation. The authors provided results for more baselines and evaluations. Overall I agree there are potentially interesting ideas here, and I propose that the authors address the comments raised.
Decision: Accept