Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
Authors: Hanzhuo Huang
Yufan Feng
Cheng Shi
Lan Xu
Jingyi Yu
Sibei Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Video
Zero-Shot Generation
Large Language Model
Latent Diffusion Models
TL;DR: This work introduces Free-Bloom, a zero-shot, training-free, semantic-coherent text-to-video generator that is capable of generating high-quality, temporally consistent, and semantically aligned videos.

Abstract: Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt. This study focuses on zero-shot text-to-video generation considering the data- and cost-efficient. To generate a semantic-coherent video, exhibiting a rich portrayal of temporal semantics such as the whole process of flower blooming rather than a set of ``moving images'', we propose a novel Free-Bloom pipeline that harnesses large language models (LLMs) as the director to generate a semantic-coherence prompt sequence, while pre-trained latent diffusion models (LDMs) as the animator to generate the high fidelity frames. Furthermore, to ensure temporal and identical coherence while maintaining semantic coherence, we propose a series of annotative modifications to adapting LDMs in the reverse process, including joint noise sampling, step-aware attention shift, and dual-path interpolation. Without any video data and training requirements, Free-Bloom generates vivid and high-quality videos, awe-inspiring in generating complex scenes with semantic meaningful frame sequences.  In addition, Free-Bloom is naturally compatible with LDMs-based extensions.

Free-Bloom:
Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator
Hanzhuo Huang"
Yufan Feng"
Cheng Shi
Lan Xu
Jingyi Yu
Sibei Yang!
School of Information Science and Technology, ShanghaiTech University
bttpa:///gthule www./oooaa/a//eeeeeBBB
Abstract
Text-to-video is a rapidly growing research area that aims to generate semantic, identical, and temporal coherence sequence frames that accurately align with the input text prompt. This study focuses zero-shot text-to-vide generation considering the data- and cost-efficient. To generate semantic- exhibiting rich portrayal of temporal semantics such the whole process of flower blooming rather than a set of "moving images", we propose a novel Free- Bloom pipeline that harnesses large language models (LLMs) as the director prompt sequence, while pre-trained latent diffusion
video,
generate a semantic-coherent models (LDMs) the animator to generate the high fidelity frames. Furthermore, to ensure temporal and identical coherence while maintaining semantic coherence, we propose series of annotative modifications to adapting LDMs in the reverse process, including joint noise sampling, step-awart attention shift, and dual-path interpolation. Without any video data and training requirements, Free-Bloom generates vivid and high-quality videos, awe-inspiring in generating complex scenes with semantic meaningful frame sequences. In addition, Free-Bloom naturally compatible with LDMs-based extensions.
Introduction
Recent impressive breakthroughs [31, 33, 35] in text-to-image synthesis have been attained by training diffusion models on large-scale multimodal datasets [37, 38] comprising billions of text- image pairs. The resulting images unprecedentedly diverse and photo-realistic while maintaining coherence with the input text prompt. Nevertheless, extending this idea to text-to-video generation poses challenges it requires substantial quantities of annotated text-video data and considerable computational resources. Instead, recent studies [53, 19, 40, 28, 5, 49, 17] introduce adapting pre- trained image diffusion models to the video domain in data-efficient and cost-efficient manner, showing promising potential in one-shot video generation and zero-shot video editing.
This work takes the study of zero-shot text-to-video generation further, enabling the generation of diverse videos without the need for any video data. Instead of relying on a reference video for generation or editing [19, 28, 5, 40], the proposed approach generates videos from scratch solely conditioned on the text prompt. While concurrent work, Texi2Video-Zers [17], similarly focuses on zero-shot video generation, our study and Text2 Video-Zeo differ significantly not only in terms of technical contributions but also motivation: we aim to generate complete videos that encompass meaningful temporal variations, which distinguish it from the generation with "moving images". As shown in Figure 1. with the text prompt of "a flower is gradually blooming". our approach generates video that thoroughly depicts the entire process, seamlessly progressing from flower bud to the
"Equal contribution.
I Corresponding author, yangsbbshaaghaittech.oo
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Video of "fower gradually blooming?
...closed buds... ...slwll unfurl...
..veevelingg center...
opened...
curvved.....
...flll bloom...
"A flower gradually blooming."
Van Gugh Style, A cluster flowers gradually blooming"
8
modern Disney style "A prince is riding horse [V].
DDIM Inversion
Personalize
Prediction
"A river is freezing"
Figure 1: The zero-shot Free-Bloon generates vivid and high-quality videos that are imbued with semantic significance according the input prompt. On top of this, Free-Bloom compatible with the existing I.DM-based extensions and can be applied to other tasks such as personalization of user-specific concepts and video prediction from a start frame
full blooming stage. This stage of the flower bloom. In other words, our resulting video is comantic-coheren that exhibits rich portrayal of temporal semantics.
apart from other methods [17, 21, 10] that often portray a single
In this work, we introduce Free Bloom, zero-shot, training-free, semantic-coherent text-to-vide video generator that is capable of generating high-quality, temporally consistent, and semantically aligned videos. Our key insight is to harness large language models (LLMs) as the director, while pre- trained image diffusion models the frame animator. The underlying idea is that LLMs [7, 30, 45], pre-trained on massive amounts of text data, encode general world knowledge [32, 27), thus being able to transform the text prompt into sequence of prompts that narrates the progression of events over time. Simultaneously, pre-trained image diffusion models, such as latent diffusion models (LDMs) [33] generate the sequential frames conditioned on the prompt sequence to create video. Moreover, considering the temporal resolution of the prompt sequence is difficult to fine-grained each frame of video, thereby a zero-shot frame interpolation is employed expand the video to higher temporal resolution. Our insights shape Bire-Bloom's 's progressive pipeline, which consists of three sequential modules: serial prompting, video generation, and interpolation empowerment.
The naïve attempt directly applying LLMs and LDMs respectively to the first two steps of our pipeline resulted failure. Not surprisingly, the resulting images completely independent of each other. Although they each match with their own prompt, the semantic content of these images is entirely disjointed (semantic coherence), the overall content vary greatly including the foreground and the background (identical coherence), and the adjacent frames cannot be smoothly connected (temporal coherence). To overcome these problems, we propose a series of novel and effective technical solutions. For semantic coherence, we instruct LLMs in a multi-stage manner, completing the missing information in the generated prompt sequence caused by discourse cohesion, and ensuring that each prompt in the sequence accurately describes the detailed visual content while maintaining consistent linguistic structure across prompts. ensure both identical coherence and temporal coherence, we propose two innovative modifications for jointly adapting LDMs for video generation: (1) Through modeling the joint distribution of initial noise latent across frames from both unified and individual noise latent distribution, we enhance temporal and content consistency while preserving suitable levels perturbation. Notably, in order conform the LDMSs assumption [33], our joint distribution ensures that the noise latent at every single frame follows normal Gaussian distribution

---

when the noise latent of other frames is not provided. (2) Considering the trade-off between continuity and adherence to annle-framee semantics, we carefully incorporate cross- fessm- attention [53]. which focuses on contextual frames, into self-attention layers according to denoising time steps.
We further propose the training-free interpolation empowerment module to improve the temporal resolution of the generated videos. In order to maintain the aforementioned semantic coherence, identical coherence, and temporal coherence, we consider semantic contents of both contextual and current frames. We perform a novel dual interpolation on latent variables relying on both contextual frames and self-denoising paths, as the former path enables smooth transitions between neighboring frames and the latter path ensures high fidelity of single frame.
In summary, our contributions are multi- (1) We introduce the zero-shot text-to-video task. Our pipeline effectively harnesses the rich world knowledge of L.Mss and the powerful generative capability of L.MMs, proposing insight into adapting text-to-image models for video generation. (2) We propose a video generation module incorporating joint noise sampling and step-aware attention shift, ensuring identical coherence and temporal coherence while expressing the semantic content. (3) We introduce a training- fareing--reeeae dual-path interpolation strategy, ensuring consistency with context while maintaining fidelity. (4) Free-Bloom can generate impressive videos imbued with semantic significance corresponding with the contextual narrative.
novel pipeline to tackle
2 Related Work
Diffusion Models for Images. Diffusion models [42] and its variants DDPM [12] and DDIM [43], have achieved breakthroughs on text-to-image generation tasks [31, 8, 35, 33]. Moreover, diffusion models have thriving research and application ecosystem, including multiple works [22, 11, 57, 23] as well as emerging open-source communities and libraries [48] with frameworks and plugins.
Open-Domane Text-to- Video Generation. Currently, both non-diffusion-basel [15, 52, 15, 47] and diffusion-basedd [14, 41, 59, 21, 1, 13, 10, 3] T2V methods often conduct training on large-scale video datasets such as Web Vid and HD-Vila 100M[2, 55]. while leveraging T2I priors or jointly training with images to maintain the quality and the content diversity. However, even with datasets of millions of videos, the quality and quantity of the training set are still not comparable to images for training.
Zero/Few-shot Video Synthesis. Recently, tuning pre-trained T2I models under prrrossot/ffwwshhe settings has also been found promising for video generation. Tune-A- V-iieo [53] uses reference video, generating videos conditioned on varied prompts while maintaining the original motion. Several works [19, 40, 28, 5, 49, 54] further explore the video editing task but are not able to either change the motion or generate videos with new events. Our concurrent work -ede2Vidoo-Zero [17], which firstly proposes zero-shot T2V pipeline started from pre-trained T2I models. While it models the motion flow by adding linear transformations to latent codes, we address that complex state transitions should be considered instead.
LLM-assisted Generations. Large Language Models (LLMs) [7, 4, 30, 45] have made significant impact by exhibiting remarkable performance across multiple Natural Language Processing tasks. From the immense training corpus, LLMs have captured open- world knowledge in various fields [24, 4, 27, 32, 44, 58]. To assist generation by LLMs, previous methods [51, 25, 20, 16, 39] are mainly based on prompt engineering. aiming to make the text prompts more expressive toward their goals. Our methods further exploit knowledge from LLMs to derive frame visual content in complete video.
3 Preliminaries
Denoising Diffusion Probabilistic Model. DDPM [12] is type of probabilistic model that learns to approximate the probability distribution of the true data. The forward diffusion process over Xo. ... XT gradually adds Gaussian noises in T time steps to corrupt an image. Then, the model is optimized to learn the denoising transitions pe (xxx images. The forward posteriors can be expressed as
the reverse process to turn noises into
(x:-1 Xt, Xo) -
11-1i (x1, xo) 3.1
(1)
where ft, Xx) = vār- 9
+ (ae(1-ār-1 1-ar
X1 and Be == 1-a, Br
I-a,
Xo
I-a.
3

---

Denosing Diffusion Implicit Models. DDIM [43] is generalized form of DDPMs by introducing the Mun-Marovvian processes. DDIM can speed up inference by using fewer denoising steps with the same training objective of DDPMs, where the reverse process can be written as follows,
X; v1
ace; 0)
x var-1
04-1 of E (xx; 0)
Vau
"direction pointing ddcnoted D.
randomn noise
"predicted Xa ddcooted as P.
where Pt :=
(x.)) and D, = 01-1 -1- -1-1--0-c,(x (x))
Voi
Latent Diffusion Models. LDMs [33] map data to a low-dimensional space, termed latent space, which possesses strong representational capacity and can capture complex and abstract features. LDM employs a multi-layer attention-base U-Net architecture for noise prediction. The attention blocks [46] contain both self-attention and cross-attention, which focus on image content and textual content respectively. In the self aelf-attention layer, the input feature z, is projected into query, key, and value through linear transformations, then they are utilized to compute the attention weights.
Method
4.1 Free-Bloom: LLMs as The Director and LDMs as The Frame Animator
Serial Prempting
Joins noise sampling
Pd Defiusion C Net
AAn astronaut waving his hands
Video Gemeration
Jos Earkoding
Allention blocks
Self- Self-Attetion
Deeerbeee the vidaa...
lingrenk unverese..
Cross-Ateeniinn
Conper
CREOUAES Interpolation
Shigiing
Attentionn Shitt
Poome 2: Leveering pstrorace ieady to woviing
Framee 2: Abairsnnall inca white spacenatt, loweriing arsas
Jemovinnng
Combo
widh enile. he eas eesriiiiaall begita ware
Aumonoue, wiilc apaccaall mase right hand
A Video
Fod Bakkdiin
Denoising
Figure 2: Overview of Free-Blooms Our pipeline consists of three sequential stages. In Serial Prompting stage, the LLM is prompted to generate serial frame prompts. In Video Generation stage, modifications are made to the LDM to generate coherent frames by joint noise sampling and step-aware attention shift. In Interpolation Empowerment stage, a dual latent space interpolation conditioned on both contextual path and denoising path is proposed generate intermediate frames.
We propose a zero-shot text-to-video generation pipeline named Free-Bloom, in which the generated videos directed by LLMs and animated by LDMs. As shown in Figure 2, our pipeline consists of three sequential stages: serial prompting, video generation, and interpolation empowerment. Given text prompt T, Free-Bloom first generates a video V Xfxxxxx with low frame rates the first two stages, which include ʃ frames with the frame size of h x w. Then, the interpolation empowerment module fills in gaps between frames to improve continuity, resulting in the final video.
At the serial prompting stage, we first prompt the LLM with their general knowledge to transform the raw prompt into a series of prompts indicating the change of semantic content over time. Then, instruct the LLM with referential resolution on the prompts to ensure that each prompt accurately describes the detailed visual content while maintaining consistent linguistic structure across prompts, resulting in a prompt sequence, T = (T), ................................ TT }. The prompt sequence accurately depicts the overall narrative and can effectively enlighten LDM [33] generate semantic-coherence frames.
At the video generation stage, we employ two novel modifications to LDM, enabling to generate semantic coherent, identical coherent, and temporal coherent video frames. We first simultaneously sample the noise at every frame from their joint Gaussian distribution, which is constructed by considering unified noise the video level and an individual noise at the frame level, which makes

---

it easier to generate coherence frames with certain variations. Then, we propose a strategy to modify the self-attention layer during the inference process. The modified attention layer adjusts attention to contextual information and self-consistent content according to the denoising time step.
At the interpolation empowerment stage, dual latent space interpolation strategy is proposed to generate the intermediate frame between two neighboring frames. In addition to jointly interpolating the latent variables of neighboring frames for temporal coherence, we condition the latent variable generated by performing DDIM [43] on the interpolated text embedding to ensure semantic coherence. Also, the weights of the two paths vary over the denoising time step to ensure smoothly compatible.
4.2 Serial Prompting for Zero-Shot Text-to- Generation
When discussing the scenario of "a teddy bear jumping into the water", humans naturally imagine the series of events: the teddy bear crouches down, then propels itself into the air, and finally lands in the water with a splash. The prior knowledge of jumping water allows us humans to derive semantic meaning of sequential events from a general description, which inspires us to ask: do LLMs also encode this kind of knowledge without further training? In our investigation of LLMs, we prompt the LLMs with instructions such as "describe each frame". We find that LLMs incorporate extensive world knowledge and can provide temporal transition knowledge.
Prompting Video.
However, the generated initial descriptions have free linguistic structure, and the text is fragmented in every single description due discourse coherence, leading to falling short of adequately sufficient information for a single frame, as shown in Figure 2. Therefore, to ensure semantic coherence of prompts across frames, we further instruct the LLMs with the above-obtained initial descriptions to generate a sequence of / prompts T (T2,. T' with consistent linguistic structures, where each prompt accurately describes the visual content in detail.
Zero-Shot Video Generation with LDMs. The video generation module aims to condition the LDMs the prompt sequence T to generate frames that are semantic, identical, and temporal coherence. However, simply generating f still images from different prompts results in a collection of unrelated images that cannot be sequenced into coherent video. To address this issue, we propose two novel modifications for adapting LDMs to generate videos rather than a collection of images without the need for additional training.
Joint noise sampling following LDMs' assumption. To model the coherence across frames, we propose to sample the initial noises in the diffusion process of frames from their joint probability distribution instead of independent distribution. To construct such joint distribution, we first investi- gate the effects of independent noise and united noise over sequential frames: (1) The same noise for every frame results in LDMs generating a sequence of images with similar content under similar textual conditions. While this feature may contribute achieving temporal coherence among frames, it can potentially restrict the natural variation of the subject, significantly reducing the overall video quality. (2) Generating images from f independent noises issues in maintaining consistency across frames, although the generated images with naturally varying content showcase more diversity and independently captivating elements.
These observations motivate us to construct the joint distribution by considering the independent and united distribution jointly. Specifically, we propose to obtain the initial noisy latent variable by weighted summation over a unified noise across the video frames and an individual perturbed noise to maintain consistency across frames while introducing appropriate variation. Moreover, to conform to the LDMs' assumption, we design the weighting coefficient to ensure that without giving initial noise latent at other frames, the initial noise at every single frame follows normal Gaussian distribution. Let us denote the unified video noise as x; and the independent noise as x the joint distributions l:f and 6 are formulated follows,
x7
X
l:f [x]
x1I Xi N1
+ pix - @ I.)
81:1 = [
x xt NN
p(
where Jy denotes the all-ones matrix with size of f x f and @ represents the Kronecker product. Then, model the mixture noise by
x J: cos( ग "))x///
1:5 + sin( 2)0
(4)
5

---

where the À is coefficient of variance rate. This mixed noise follows the joint distribution as
p(x+') )- = (0,, sin 2 ( + cos2 "I , @ Ln)
An) = -O, + cos"( ()) - 1,)
2
(5)
@ I,)).
When A = 0, the initial noise becomes unified noise, and when À = 1, it becomes independent noise.
Step-aware attention shift. To further maintain identical coherence while ensuring semantic coherence, we shift the attention of the self-attention layers in the LDMs' U-Net from cross-frame contexts to the current frame according to the denoising time steps. For frame with its query Qi previous methods [53, 5, 49, 28] with one overall text prompt retrieve the key and value from the former frame and the first frame to perform sparse spatio-temporal attention, based on the observation [53] that extending the spatial self-attention across images produces consistent content.
In scheme, frames are conditioned different prompts, requiring that not only maintaining temporal and identical coherence, but also semantic coherence with their respective prompts. To achieve identical and temporal coherence, we address the former and the first frame contextual frames and attend to contextual key-value pairs. In particular, the former frame helps to enhance temporal coherence, while the first frame acts as a benchmark shape, maintaining appearance consistency throughout the video. To align with the respective prompt in semantics, we shift attention the current frame itself as the time step increases, enabling the preservation of its specific characteristics and details. In summary, our inference strategy takes into account the time step during the diffusion process. The initial steps focus on contextual frames to form coarse-grained shapes and layouts. As we progress, complete images are generated, emphasizing producing accurate outlines conditioned on semantic information. Overall, our adaptation of the attention mechanism refers to:
Solf-Attention == Attention (Q,, [Ko, K,- Ki), (V. V2-1, Vil).
22T
(6)
Attention(Q2. K,, Vi),
t t<t
where 7 is the threshold of the time step for attention shift. The (Ko, Vo), (Ki-1, V 1), and (K;, V.) are the key-value pairs the first, former, and current frames, respectively.
4.3 Interpolation Empowerment
The interpolation empowerment module is proposed to further increase the frame rate without extra training resources. Similar to the insight of utilizing contextual information in the "Step- "Step-aware attention shift" proposed in Section 4.2, the interpolated intermediate frames should also consider contextual frames, i.e., the former and the latter frames, in the denoising process. One naïve approach is to directly derive the latent variable of the intermediate frame solely from the latent variables in contextual frames. However, this method overlooks the intermediate semantics indicated in text prompts in contextual frames, leading the semantic incoherence.
Therefore, we propose dual interpolation path for generating intermediate latent variables, where the contextual path interpolates the latent variables between the contextual frames to ensure temporal coherence, while the denoising path interpolates latent variables in DDIM [43] denoising process conditioned on interpolated text embedding to improve the semantic coherence. Specifically, to interpolate a new frame xf between x and xf we first sample the intermediate initial latent variable from the same distribution proposed in Section 4.2. For the conditional textual prompt T!, directly interpolate text embeddings of the previous and next frames.
1) Contextual path. To obtain the context-sensitive latent variable x we perform linear interpolation follows,
between the contextual frames x/- and x
x = Xx + (1 (1-1)x(**
2) Denoising path. Then, we perform a linear interpolation between the context-sensitive latent variable x and the latent variable obtained from DDIM conditioned on the interpolated prompt T'. We employ mtt) as the interpolation coefficient and use the notation of Pf 2+ and D mentioned in Section 3. The interpolation coefficient m(() varies according to the time step, with smaller values in the earlier denoising steps and increase in the latter steps. The interpolation formulated as follows,
x (1 m(t)x + (orp) + D/41 + o,2z).
(8)

---

To provide an intuitive conditional probability distribution of zz/, we present the distributions for the cases when m(t) and m(t) 0 respectively as:
N NK (11 + (1-
1 1- k) 1
1-11/4
k
m = 0
p(x x xx
m 1 1
(9)
(N 2)41):
where Σ/ 1/-1, is the covariance matrix of x and x/41 ff+1
4.4 Extensions
Our method exhibits high extensibility and seamless integration with existing L.DM-based methods. We demonstrate the versatility of our approach through some applications: (1) Personalization. By combining with LDM-based personalization methods [9, 34], our approach can generate videos with user-provided concepts. Here we choose DreamBooth [34] as an example, as shown in the top Figure 1, our method generates a video of "A prince is riding a horse [V]" with the concept [V] as "modern Disney style". (2) Making an image move. With the help of DDIM inversion [43], we can inverse the latent variable from an image, then generate sequence of frames that continues the image. For example, starting from a still image of the scenery of a river, our method further generates a video depicting the process of freezing, as illustrated in Figure 1.
5 Experiments
Implementation Settings. We develop our Free-Bloom based on LLM as ChatGPT [26] and LDM as Stable Diffusion [33] with its pre-trained v1.5 weights. We first generate a video of f f= 6 length with 512 x 512 resolution, then we iteratively interpolate between the most distinguished frames with A: 0.5 for the first interpolated frame. For mn(t), we m(t) 0.1 when the denoising time t 2 -" and m(t) = when & < r² Notice that our method actually can be adapted to generate longer videos. To enhance image quality in practice, we add fixed negative prompts and upscale our resulting frames with an image super-resolution network ESRGAN [50]. All experiments are performed on single NVIDIA GeForce RTX 3090Ti.
5.1 Baseline Comparisons
Qualitative Results. We showcase examples of Free-Bloom in Figure 3 and compare them with that generated by the zero-shot T2V-Zero [17] and VideoFusion [21], which demonstrate the most outstanding overall performance in the user study. More comparisons are included in the Supplemen- tary. Video generation shown in region demonstrates the following observations: (1) Our method vividly depicts the complete imagery of volcanic eruption the sequential motion of teddy bear jumping into the water, exhibiting the capacity to generate semantic meaningful frame sequences, in which the visual elements, actions, and events aligned with the input prompt as well as the contextual narrative. (2) In addition, our method shows temporal coherence and identical coherence while maintaining high fidelity for single frames. (3) Although the T2V-Zero method maintains overall content consistency between frames, it fails to depict sequential events. Furthermore, the the other hand, demonstrates impressive temporal coherency between frames as it is trained on large-scale datasets, and it also presents certain level of grasp of events. However, this training on the vast video dataset also significantly degrades the fidelity and quality of individual frames.
subject would easily be distorted as the length of the video increases. (4) VideoFusion,
For interpolation results shown in region B, present one latest state-of-art Video Frame Interpola- tion method AMT [18] tre-trained on Vimco90K [56] for comparison. AMT fails to comprehend the subject information between the two target frames. As result, it blurs the different parts of the teddy bear's body, failing to capture the intermediate motion. In contrast, our method fills in the content gap with serial of continuous motion of the bear from the air to the water, maintaining fidelity in the intermediate frames while ensuring content consistency.
Quantitative Results are reported with automatic metrics and the user study in Table 1. We adopt three publicly available diffusion-based methods, Text2Video-Zen [17], VideoFusion [21], and LVDM [10] as basclines. VidcoFusion and LVDM are both trained methods while the former is
7

---

AJ Video Geveration
"Teddy bear jumps into water."
Ours
T2N-Za0
"Volcano Eruption."
B) Iahaspaauaa
Ours
TZ--rro
Ours
AMI
VidkoFunii
OuES
AMT
Figure 3: Qualitative comparison. A) Our method can generate semantic meaningful frame sequences when conditioned on the same prompts. B) We interpolate two frames into the frame sequence by taking the two images within the same color box in part A as the start and the end.
trained on both large-scale image datasets ImageNet [6], LAIONSB [38] and a large-scale video dataset WebVid-10M [2], and the latter has been trained on a 2-million subset of the WebVid-10M
For automatic metrics, we use CLIP [29] to evaluate the similarity correlation between the input prompts and the visual content of generated frames. Recall that our method put an emphasis on the overall narrative semantic coherence, therefore we also compute the CLIP score between each frame and its corresponding prompt generated by LLM((*.. We can observe that although comparing each frame against the input prompt overlooks the potential of our approach, our method demonstrates good performance when frames are compared with the frame-level video prompts.
For the user study, the participants are instructed coherence on a scale of 1 to and give comprehensive ranking. According to the user study, although our method may not perform as well trained methods in terms of temporal continuity, has received high recognition in all other dimensions of video quality.
the fidelity, temporal coherence, and semantic
Table 1: Quantitative Results. for CL.IP score on serial prompts.
Automatic Metric
User Study
Method
Training-Fres
CLIP Metricst
Fidelity t Temporal t Semantic t
Rank |
VideoFusion [21]
0,483
3.436
3.889
3.267
2.317
LVDM [10]
0.480
3.289
3.650
3.242
2.567
T2V-Zero [17]
0.479
3.486
2,783
3.025
3.033
Ours
0.477 / 0.482*
4.133
3.267
3.867
2.083
5.2 Analysis of Our Pipeline
In Figure and Figure 5, we conduct comprehensive analysis of the modules our proposed pipeline. The top row showcases final results.

---

"Tronman surfing."
"The ice cube melting.
Ous
Figure 4: Analyze for the effects of Free-Bloc Bree-Bloom (A) Without joint noise sampling. (B) Without shifting self-attention, (C) Serial Prompting for Text2 Videovide- (D) Without attention to the current frame itself, and (E) Without the step-aware shift strategy.
A) Without joint noise sampling. In A, we replace the joint noise initialization with the independent initialization, resulting frame sequences with inconsistent image content. B) Without shifting self-attention. In B, we denoise the frames from the proposed joint noise initialization but use the raw LDM without modifying self-attention layers. The frames are similar naturally in some cases, as the joint noise contains portion of unified noise. However, without attention to contextual frames, is difficult to maintain identical coherence of both the foreground and background content, let alone temporal continuity. C) Serial Prompting for Text2 Vide2Video-Zar to enable the input of frame-level video prompts so that each frame is generated and conditioned on a distinct prompt. The results show that it is challenging for the current method comprehend serial prompts effectively, resulting in "moving images". D) Without atlention to the current frume itself. In D, we replace all self-attention layers with spatio-temporal attention layers proposed in TAV [53]. The resulting frames exhibit improved temporal coherence, demonstrating smoother transitions between frames. However, the frames appear almost identical, creating sequence of meaningless and jittery frames based solely on the first frame, which does not align with the intended temporal semantics. Additionally, prolonged contextual attention in long sequences can significantly compromise the fidelity of individual frames. As shown in the case of Iron Man, the last frame presents an incomplete leg. E) Without the step-aware shift strategy. Based on D, we further concatenate the attention the current frame with contextual attention but without shifting it along time steps. Although the results remain inconsistent with semantics due the absence of the inference strategy that varies across the denoising time step, the fidelity of images is improved.
In C, we adapt Text2Video-Zar [17]
Dual-path interpolation. We show the influence of bal- ancing timing coefficients between the dual interpolation path in Figure When 7" approaches 1, m(t) remains relatively small throughout the entire denoising process, indicating that the contextual path primarily determines the interpolated results. The resulting image presents blurring and ghosting artifacts, failing to generate frame with correct semantics. Conversely, heavily relying on the DDIM denoising path with larger values of m(t) causes substantial deviations in content compared to the contextual frames. Our method adopts smaller values rn(1) in the early steps to prioritize coarse -rasied shapes and layout and increases m()) in the latter steps to improve temporal consistency. a result, we achieve the benefits of both paths by appropriately setting m(t).
Promee
Figure 5: Analyze for interpolation.

---

6 Conclusion
In this paper, we devise a novel zero-shot and training-free text-to-video approach, which mainly focuses on improving the narrative of the progression of events. Our proposed pipeline effectively harnesses the knowledge from the pre-trained LLM and LDM and produces highly semantic coherent videos while also maintaining temporal coherence and identical coherence. Limitation: we look forward to further research on text-to-video generation, however, it should be acknowledged that there can be ethical impacts like other generative models. As we adopt ChatGPT and Stable Diffusion v1.5, our method may inherit the bias of those two models.
A Acknowledgment: This work was supported by the National Natural Science Foundation of China (No.62200 (74), Shanghai Pujiang Program No.21PJ1410900) Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), MoE Key Laboratory of Intelligent Perception and Hlmann-Machin Collaboration (ShanghaiTech University). and Shanghai Engineering Research Center of Intelligent Vision and Imaging. Also, we extend our heartfelt gratitude to the anonymous users who participated our user study.
References
[1] Jie An, Songyang Zhang, Harry Yang. Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.044 08477, 2023.
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint 2rXiv:2304.088 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[5] Duygu Ceylan, Chun-Hao Paul Huang, and Niloy Mitra. Pix2video: Video editing using image diffusion. arXiv preprint arXiv:2303. 12688, 2023.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
leee, 2009.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18/0.00803 2018.
[8] Richard A Friesner, Jay I. Banks, Robert B Murphy, Thomas A Halgren, Jasna J Klicic, Daniel Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, al. Glide: new approach for rapid, accurate docking and scoring. 1. method and assessment docking accuracy. Journal of medicinal chemistry, 47(7):1799-1749, 2004.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.012: 016618, 2022.
[10] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.132211 2022.
[11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promet image editing with cross attention control, arXiv preprint XXXi:22208001222 2022.
10

---

[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 13:6840-6851, 2020.
[13] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023000 2022.
[14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03458, 2022.
[15] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint 2.ii::2225.18666 2022.
[16] Hyeonho Jeong, Gihyun Kwon, and Jong Chul Ye. Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint arXiv:2302.03900 2023.
[17] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero Text-to-image diffusion models zero-shot video generators. arXiv preprint arXiv:2303. 2023.
[18] Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. arXiv preprint arXiv:2304.09999, 2023.
[19] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.0470 2023.
[20] Vivian Liu and Lydia Chilton. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1 1-23, 2022.
[21] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv e-prints, pages arXiv-2303, 2023.
[22] Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.00 1,9994, 2022.
[23] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint urXiv22302.08453 2023.
[24] Tuan-Phong Nguyen, Simon Razniewski, Julien Romero, and Gerhard Weikum. Refined commonsense knowledge from large-scale web contents. IEEE Transactions on Knowledge and Data Engineering, 2022.
[25] Jonas Oppenlaender. taxonomy of prompt modifiers for text-to-image generation. arXiv preprint arXiv: 13988, 2022.
[26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[27] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1909v 00006 2019.
[28] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09533 2023.

---

[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8888 8763. PMLR, 2021.
[30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(():5485-555), 2020.
[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[32] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint arXiv:2002.089) 08.00000.0...,.......00.00000000 2020.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[34] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv: / 12242, 2022.
[35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- tomi diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479 -6494,, 2022.
[36] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. is all about where you start: Text-to-image generation with seed selection. arXiv preprint arXiv:2304. 14530, 2023.
[37] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 1..12111111144 2021.
[38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402. 2022.
[39] Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision-language models. arXiv preprins arX1v:2309.01155, 2023.
[40] Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single video editing with object-aware consistency. arXiv preprint 07X49:2303.07945 2023.
[41] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 2499,49229, 2022.
[42] Jascha Sohl--Dcckstei,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv22010.025502, 2020.
[44] Jijjin Tang, Ge Zheng. Jingyi Yu, and Sibei Yang. Cotdet: Affordance knowledge prompting for task driven object detection. arXiv preprint arXiv:2309.0109. 2023.
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
12

---

[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[47] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02339, 2022.
[48] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the diffusion models. https: //gi tigtthu..cd www/hugginntccccccccccccccoooooa
hwtph//ggggggggeegeeeeeeeeoeeeee www.gngaee/iffiiee httf/dffffferrss.e 2022.
[49] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using fff-the-sheff image diffusion models. arXiv preprint arXiv:2303. 17599, 2023.
[50] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceed- ings of the European conference computer vision (ECCV) workshops, pages 0-0, 2018.
[51] Sam Witteveen and Martin Andrews. Investigating prompt engineering in diffusion models. arXiv preprint arXiv:22 15462, 2022.
[52] Chenfei Wo, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang. Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.1 14806, 2021.
[53] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot uning of image diffusion models for text-to-video generation. arXiv preprint arXiv:22 1565, 2022.
[54] Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn a motion pattern for few-shot-based video generation. arXiv preprint arXiv:2310.1076) 2023.
[55] Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5036-5045, 2022.
[56] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William Freemann Video enhancement with trientri flow. International Journal of Computer Vision, 127:1106- -1000 2019.
[57] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, arXiv preprint arXiv:2302.05544 2023.
[58] Ge Zheng, Bin Yang, Jiajin Tang. Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thoughh prompting for multimodal reasoning in language models. 2023.
[59] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint 211.122211.11018. 2022.
13

---

Appendix
In this section, we provide additional discussions, details, and experiments to further support our contributions. The content is organized as
Section A Discussions
Section Joint Noise Derivation
Section C- Implementation Details
Section C.1 Serial Prompting
Section C.2 Test Set for Quantitative Results
Section C.3 Details of User Study
Section C.4 Code Used and License
.
Section D Additional Experiments
Section D.1 Qualitative Results (comparisons and more visualizations)
Section D.2 - User Study Quantitative Comparisons
Section D.3 Analysis on Joint Noise Sampling
Section D.4 Extensions (long video story, personalization, making an image move)
A Discussions
Limitations and Future Work. We look forward further research this method. While method offers the advantage of being training-free and not requiring extra training data, it highly depends on the large foundation models LLMs [7, 26] and LDMs [33]. Consequently, would inherit the limitations of those large pre-trained models. For example, LDMs often struggle with generating images containing detailed faces and limbs, specific text, multiple objects, interactions between objects, etc, therefore our method has the same weakness. Moreover, LDMs are often sensitive to seed selections of initial noises [36]. when the initial frame of low quality, our method tends to result in relatively poor performance as well. Additionally, although our method demonstrates improved temporal consistency to other zero-shot methods, we found that still challenging to maintain high temporal coherency between frames in the zero-shot setting. However, leveraging video data proves be an effective solution for acquiring temporal priors. Therefore, how to combine the strengths of zero-shot methods and trained methods is promising direction for future research.
Societal Impacts. It should be acknowledged that there can be ethical impacts like other generative models. As adopt ChatGPT [26] and Stable Diffusion v1.5 [33], our method may inherit the bias of those two models. Also, although the results of our method of direct text-to-video generation are still step away from convincingly photo-realistic videos, the risk of abuse, for example, generating fake, harmful, or discriminating content, should be aware.
B Joint Noise Derivation
First, us consider the distribution unified noise. is composed by initial noise xi (O, I,) for each frame and can be represented as 1:: |x "TT The noise of any two frames in 1:f have the same values, and therefore their covariance is
x
cov x D(x;) In.
(10)
Thus the unified noise follows the distribution as
= I,),
(11)
where J represents the all-one matrix of size ʃ and (9 denotes the Kronecker product. The specific form of J, @ I, is
14

---

fxIn
J Ø In
(12)
Second, let us consider the distribution of individual noise, which each frame is independently sampled. Therefore, the covariance between any two frames is O, and the distribution still follows standard normal distribution:
p NN
(13)
According to Section 4.2, the mixed noise is defined as X l:f = cos( 1 XT I:f and / are independently sampled, the sum of the two still follows normal distribution, with a mean of 0 and a variance of
1:f + sin( 7 l:f
Since
sin"( 1in"(²A) A)ns cos ( In) sin²
0
+ cos (
(2
I.
I
cos ( A)In
cos² I) cos² (3
cos?
In
cos
I
cos² c
cos
I
2
π Inf + cos
(J
(14)
Thus, variable x2: follows the following distribution.
p(x)))= NO, sin? C A)InS + cos?1 A)J @ I,))
(15)
=N(O In cos² ())(s)) 1y)
I.))
In this distribution, without given noises of other frames, for any frame noise xy, it still follows standard normal distribution that p(x+)) (0, I.).
C Implementation Details
C.1 Serial Prompting
We first prompt ChatGPT [26] with the following instruction:
would like you play the role of the describer of each frame of the video director of movie. The content of each video should be concise and only clearly describe the subject. Each sentence in the video is independent. Every sentence needs include the subject's appearance and actions, please describe the main actions of the object and the extent of the actions in as much detail as possible. The sentences of each picture are independent, and each sentence should describe what exists in the picture. Each frame is described in only one sentence. Suppose there is video about "[INPUT PROMPT]" and there are "[F]" frames in the video. Describe the content of each frame separately. Please be straightforward and do not use a narrative style.
Then, the following prompt:
Now perform Coreference Resolution on the above sentence, replace reflexive pronouns with their original vocabulary, and eliminate the discourse cohesion. Keep the meaning the same. The sentence for each frame should be able to fully express all the visual information of the frame. Also, the linguistic structure of each sentence should be simple and similar.
15

---

C.2 Test Set for Quantitative Results
We list some prompts from our test set in Table 2, in which some prompts are from the webpage of Text2Video-Zern [17] and some designed by ourselves that incorporate more complex event content.
Table 2: Prompts for Test Set.
A cluster of flowers blooms
Astronaut riding a horse
Use pan to fire an egg Volcano eruption dog walking down the street Light a match then the match goes out
Iron man is surfing The icc cube melting A panda walking down the street
The Santa flying through the sky
River freezes
Two supermen are fighting A bear dancing on times square An astronaut waving his hands the moon
Two men shake hands
Teddy bear jumps into water
The growth of a sapling
egg hatch into chick
A dancing mickey
Teddy bear is greeting
C.3 Details of User Study
We conduct user study to understand how humans would evaluate the current text-to-video methods. The survey contains a total of 20 prompts with each prompt having 4 videos output from VideoFu- sion [21]. LVDM [10], Text2Video-Zero [17], and Ours. For each prompt, ask raters answer the following four questions:
How would you rate the temporal coherence and smoothness of the videos? Please assign score for their continuity. (Temporal Coherence)
How would you rate the quality and fidelity of the individual frames in the videos? Please assign score for the visual quality. (Fidelity)
How well does the video depict the content described in the text? Please assign a score for its content representation. (Semantic Coherence)
Based on your overall perception, please rank the videos. (Rank)
Figure 14 presents an example interface of our survey. We received valid responses from a total of 80 individuals from both industry and academia.
C.4 Code Used and License
Table 3: The used codes and license.
URL
Citation
License
http::/gghhbbb.oomm.bbwwaabbtaa Video
[53]
Apache License 2.0
tttp://gghhbbboooooooo////ootoot
[11]
Apache License 2.0
http://ggthhb..com/nnngiggggggch
[48]
Apache License 2.0
[17]
CreativeML Open RAIL-M
http://diiiiaaooonvvveeeettttttt http://ggiabbbcoommnnnneccccnnec
[10]
(Hugging Face Space) MIT
[21]
Apache License 2.0
All used codes and their licenses are listed in Table 3.
16

---

D Additional Experiments
D.1 Qualitative Results
We showcase more visualization of the generated video in this section. In Figure and Figure 7, we present the full comparison with Text2Video-eer [17], VideoFusion [21]. and LVDM [10]. In Figure 8, we randomly generate multiple videos with respect to the same prompts. In Figure 9, we demonstrate more results of our interpolation empowerment module.
D.2 User Study Quantitative Comparisons
Table 4: User Study Comparisons.
User Study
Method
Training-Fres
Rank
Fidelity
Temporal Semantic
Ours vs. LVDM [10]
55.00% 85.28%
53.33%
82.27%
Ours vs. VidcoFusion [21]
63 06.06666
82.50%
44.44%
78.33%
Ours vs. T2V-Zero [17]
73.61 %
87.78%
80.00%
85.28%
For the user study part the quantitative results, we also present the comparison- based results here in Table 4. Specifically, for the ranking column, the number denotes the percentage of participants who prefer our method and rank us before another. For the dimensions of fidelity, temporal coherence, and semantic coherence, the numbers indicate the percentage of participants who believe that our generated videos are better to that of another method that dimension.
D.3 Analysis on Joint Noise Sampling
We additionally analyze the effect of our noise sampling method. In part A of Figure 10, we sample 1:f + sin( A) using sin cos weighting. While the noise the same as equation 1. == cos( A)x
in part B, we modify the weight the unified noise and the individual noise as
-1- + A0
(16)
However, this would disrupt the single-frame noise from following normal Gaussian Distribution. As observe, in this way, LDM fails to generate reasonable images.
D.4 Extensions
In this section, we demonstrate the extensibility of our approach with more examples. Inspired by Phenaki [47] which generate story-based conditional videos based sequence of prompts, also apply our method to the same task, which is presented in Figure 11. Figure 12, we showcase some results of combining DreanBooth [34] to include personalized concepts in the generated videos. In Figure 13, we showcase the results of generating videos based on the given first frame by leveraging DDIM inversion [43].
17

---

"Two superman are fighting."
T2V-Zero
shutt
Shutt
shuut
Bokkssi
shu e
tock
tterstock
shulterstock
VideoPusion
LVDM
'The growth of a sapling."
T2V-Zero
VideoFusion
Figure 6: Additional Qualitative Comparisons. In the case of "Two supermen are fighting". the LLM vividly decomposes the process of fighting into frames, with the fifth frame depicting "colliding in dazzling display of sparks and force" which captured in our result. the case of "the growth of sapling" our result clearly presents the gradual sprouting of small sapling.
18

---

"Light match then the match goes out.''
T2V-Zero
erstock
Vidiononsio
LVDM
"A teddy bear is greeting."
Ten-
VideoFuuson
LVDM
Figure 7: Additional Qualitative Comparisons. In the case "light a match then the match goes out", our method successfully depicts the entire process of match from lightning. burning to extinguishing. the case of "a teddy bear is greeting we exploit the world knowledge of LLM [26] to translate greeting into series of specific actions such waving and smiling.
19

---

"A flower gradully blooms."
"Volcano eruption."
"A dancing mickey."
Figure Additional Qualitative Results. Multiple results based on the same prompts are shown.
20

---

Frame A
Interpolation
Frame B
Figure 9: Qualitative Results from Interpolation Module. We interpolate frames between each pair of original neighboring frames. Our interpolation module enables smooth transitions between two key states.
(A) cos(
//2 x)xx
1:f
+ sin( 2)
2
(B) (1 X 1:f -871
Figure 10: Analysis on Joint Noise Sampling. Without our proposed sampling, the initial noise at each single frame would not follow normal Gaussian distribution, resulting in corrupting frames.
21

---

1st prompt: "A flower is blooming"
2nd prompt: "The flower in the rain"
3rd prompt:
"The flower gradually freezes"
Figure 11: Extension Long Video Story. Our method can generate the long video story based on sequence prompts.
"cccorgi dog> is sitting down."
"A princess is waving her hands, <modern disney style>
"<sks mr potato head> is dancing."
Extension Personalization. Our method can generate videos with user-specific concepts. The tokens of "ccorgi dog". "modern disney style". and "sks potato head" are from their respective personalized models ccorgi-dog, Mo di Diffusion, and Mr Potato Head.
Figure 12:
22

---

DDIM Inversion - "Fuji Mountain."
"Fuji Eruption."
"Pyramid"
"Pyramid in the snow."
'Grassland'
'Grassland storm.'
Figure 13: Extension Making an Image Move. Our method can generate videos based on the first frame and its corresponding prompt by combining our method with DDIM inversion [43].
23

---

User Study on Text-to-Video Generation
Overview:
Thank you much for helping us with research! In this survey, will present text descríptions a along corresponding videos, which generated by four state-of-the- art text-to-video Al models. Each question will include text prompt and the corresponding generated videos. Your task dimensions. Additionally, pleese provide a final comprehensive ranking based on your preferences.
performance different
This survey consists of 5 questions and will approximately take 10 min of time.
Sign Google your progress, Learn
Indicates required question
Below are four videos generated from 'A cluster of flowers blooms'
How would you the temporal coherence and smoothness of the videos? Please assign score for their continuity.
O
O
O
O
O
0
0
O
0
0
0
0
Figure 14: Interface of surveys from the user study.
24


Reviews:
[{"review_id": "RuGkkAOpQO", "reviewer": "Reviewer_VS1k", "summary": "The paper proposes a zero-shot text-to-video generating pipeline called Free-Bloom, which first use LLM to generate prompt sequence decribing frames in a video, then generate frames according the prompts. To enhancing coherence, the authors proposed joint noise sampling, step-aware attention shift and dual-path interpolation. The authors compare their method with other video generator quantitatively (Clip metrics, user study) and qualitatively. ", "questions": "1. Please provide more clarification of experiments, including prompt engineering to get frame description, hyperparamters (such as \\tau, \\tau*)\n\n2. The coherence is not good, but the idea of leveraging LLM is interesting. Is it possible to combine LLM director with other existing pretrained text-to-video methods (e.g.  make-a-video) to enhance conherence?", "limitations": "see weakness and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. explore zero-shot text-to-video generation LLM director, which leverage the story generation ability of LLM to generate semantic meaningful frame sequences. \n2. Good per-frame quality. \n3. tried several methods to enhancing coherence of zero-shot video generation. ", "weaknesses": "1. insufficient ablation study. \n\n   a. there's only two qualitative cases in ablation study, which is not convincing enough. More cases and quantitative results (e.g. CLIP metrics) should be provided. \n\n   b. given that joint noise sampling, step-aware attention shift and dual-path interpolation are universal technics for zero-shot video generation, experiment result of combining these technics and past zero-shot methods (e.g. text2video-zero) is needed. \n\n2. The coherence is not good enough. e.g. sudden changes in the color / indentity can be observed. There is a fatal flaw in technical design: authors set m(t) to 1 and discard attention shift when t is small, which may lead to strong incoherence. \n\n3. Insufficient clarification of experiments, e.g. prompts to LLM, hyperparameters. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Insufficient ablation study (part 1)**: more cases and quantitative results should be provided.\n\nThanks for your valuable feedback. Per your advice, we conducted additional ablation studies and evaluated their performance in both human evaluation and automatic metrics. As presented in CLIP metrics and all dimensions in the user study, the absence of either the noise sampling module or the attention shifting module would significantly degrade the performance, and the lack of both would result in extremely poor results.\n|Serial Prompting|Joint Noise Smapling|Shifting Self-attention|CLIP Metrices↑|Temporal↑|Identical↑|Rank↓|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|√|-|-|0.4429 / 0.4661|1.62|1.95|3.66|\n|√|√|-|0.4482 / 0.4712|1.83|2.30|3.04|\n|√|-|√|0.4509 / 0.4736|2.52|2.71|2.19|\n|√|√|√|0.4521 / 0.4773|3.67|3.46|1.11|\n> **Insufficient ablation study (part 2)**: experiment result of combining these proposed technics and past zero-shot methods (e.g. text2video-zero) is needed.\n\nWe combine our technics with text2video-zero (T2VZ) and show the results in Figure 1 of the one-page PDF. \n\n- **Joint Sampling w/ T2VZ:** we substitute the initial noise of T2VZ with our sampling method. Compared with the T2VZ’s original noise that adds motion dynamics to the base latent code, our sampling method can ensure the quality of generated individual frames remains dependable even as the number of frames increases.\n- **Attention Shift w/ T2VZ:** we re-implemented the attention control in T2VZ. The figure shows that under the scheme proposed by T2VZ, our step-aware attention shift can achieve slightly better results than T2VZ's attention strategy. But, we still want to claim that our attention shift's contribution cannot be fully reflected in this experiment, that is, working much better for challenging cases with more complex motions and frame-wise prompts than \"moving image\" in T2VZ. When all the frames are generated based on one single prompt, it is more like creating \"a trembling gif\", leading attention to previous frames and current frames less critical.\n- **Interpolation w/ T2VZ:** we directly apply our interpolation algorithm to the generated video of T2VZ. Our interpolation module can supplement intermediate states between two frames in the original video, thereby increasing the frame rate.\n\n> **The coherence is not good enough (part 1).**\n\nWe have significantly improved the temporal coherence compared to previous zero-shot T2V methods, and our semantic coherence is much better than both zero-shot and trained T2V methods, according to our user study. \n\nWe appreciate the reviewer for pointing out the coherence concern, and the coherence in video generation is essential but incredibly challenging in a zero-shot setting without any reference videos. Many of the efforts in our work revolve around coherence, and our new technicals indeed improve coherence a lot and can be a good starting point for future research to enhance coherence further. \n> **Coherence (part 2)**: Sudden changes in the color/indentity can be observed. There is a fatal flaw in the technical design: authors set m(t) to 1 and discard attention shift when t is small, which may lead to strong incoherence.\n\nThis is not a design flaw. The $m(t)$ influences interpolated frames but is irrelevant to the contextual frames in the low-frame-rate stage. Thus, it is unrelated to the identical/coloring coherence. \n\nAlso, attention shift is not discarded in the interpolation process (equation 8), whether t is small or large. The term $\\tilde{\\mathbf x}_t^f$ after $(1-m(t))$ is derived from the latents of contextual frames, and the latents are obtained through their attention-shifting on their paths. And the term following $m(t)$ pertains to the frame denoising path, on which the attention-shift operation is also performed. Notice that smaller $t$ indicates the later denoising stage, thereby needing to focus more on the interpolated frame's denoising path. So, we set $m(t)$ as a step function empirically to simplify the pipeline and attain straightforward results. \n\nCertainly, if we define $m(t)$ as a more elegant function varying through time steps, the performance of interpolation may be further enhanced. \n> **Insufficient clarification of experiments**, e.g. prompts to LLM, hyperparameters (such as \\tau, \\tau*).\n\n**Prompts to LLMs:** Please refer to Appendix Section C.1.\n\nThe settings of the **hyperparameters** $τ=400$  and $τ^*=400$ (when timestep T=1000) are recommended for most practical use cases.  According to specific cases, they could be additionally adjusted. \n\n- For cases involving more semantic and motion changes, as exemplified by “Teddy bear jumps into water” in Figure 3, it is advisable to opt for larger values of $τ$ and $τ^*$. This choice put more emphasis on the current frame by shifting attention to the current frame at an earlier step ($τ$) and increasing the weight of the self-denoising path at an earlier step ($τ^*$).\n- For cases with basically similar frames, such as “Light a match then the match goes out” in Appendix Figure 2, smaller $τ$ and $τ^*$would be suitable.\n\n> Is it possible to **combine LLM director with other existing pretrained text-to-video methods** (e.g. make-a-video) to enhance coherence?\n\nWe agree that leveraging pre-trained text-to-video methods can improve the coherence of generated videos, as training with an immense amount of video samples allows the models to acquire knowledge about how to maintain coherence between frames. \n\nHowever, it is not easy to combine LLM director with pre-trained T2V methods in a training-free manner because videos generated by pre-trained T2V models are likely to conflict with the per-frame prompts. Addressing the conflict is a promising direction but is currently out of this work's scope. On the other hand, if requiring additional training, it is against the initial motivation of our present work considering the data- and cost-efficient."}, {"review_id": "D6z0MjnDeJ", "reviewer": "Reviewer_mgrm", "summary": "In this work, the authors propose a principled pipeline for text to video generation. Specifically, three techniques are proposed: (1) a LLM based per-frame prompt generation, so that the motion/dynamics of each frame can be better specified. (2) a noise joint sampling schedule, and a step-aware attention shift is proposed to enhance temporal consistency. (3) an interpolation module to generate longer videos. Extensive experiments and results have demonstrated the effectiveness of the proposed work.", "questions": "elaborated in the above sections.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The pipeline is technically sound and generally makes sense to me. It is straightforward to use the strong prior from pretrained LLM to give more temporal/motion/appearance information to each frame, to facilitate video generation. The design of noise scheme, sampling, and attention shift is technically sound.\n\n2. The exposition of this paper is very good and clear to me.", "weaknesses": "1. In my honest opinion, directing comparing this work w/ many baselines is kind of unfair, since this involves strong prior in LLM, and this is the key factor of improvement in video generation. I strongly recommend the authors to compare w/ some similar text2video pipelines that also factorize the unified single prompt condition into per-frame motion-aware conditions. For example, first generate a sequence of optical flow maps, then takes them as conditions to generate the video.\n\n2. Since I'm not the expert in NLP domain, I'm concerned about the ability of LLM to generate very reasonable or temporally-coherent per-frame prompts. I'm also curious about the feature trajectory of such text features. Say, you generate the per-frame prompt of p1, p2, p3, ..., pn, and use the text encoder to get text features f1, f2, ..., fn for cross-attention. Do they form a smooth trajectory in the high-dimensional text feature space? Do the authors think that, a smooth feature transition among frames can guarantee a smooth video? If so, a naive idea for improvement of this work is to add some regularizations to constrain the smoothness of text features across frames.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Prior in LLM (part 1)**: To my honest opinion, directing comparing this work w/ many baselines is kind of unfair, since this involves strong prior in LLM, and this is the key factor of improvement in video generation.\n\nAs the reviewer pointed out, using LLMs introduces prior knowledge about how the visual content evolves throughout the frames according to the unified prompt. One of our contributions is to propose this pipeline to leverage this prior in the text-to-video generation.\n\nHowever, language is sometimes not as exact as other conditions, such as pose. For example, it is hard to use language to precisely describe where a person's pose skeleton and the positions in a picture. From this aspect, we consider that the knowledge from LLMs could be a kind of weak prior. \n\nAlso, although LLM brings a prior, it also poses a hard challenge: given per-frame fine-grained prompts, how to adapt LDMs to produce a coherent and contextually fitting frame sequence. Per-frame prompts lead to significant incoherence in the LDM’s frame results, and language alone can not guide LDMs on how to make each adjacent frame coherent.  \n\n> **Prior in LLM (part 2)**: I strongly recommend the authors to compare w/ some similar text2video pipelines that also factorize the unified single prompt condition into per-frame motion-aware conditions. For example, first generate a sequence of optical flow maps, then takes them as conditions to generate the video.\n\nThank you for this advice. We reconducted a survey of recent works, but unfortunately, we did not find a zero-shot work capable of giving a text input and generating per-frame motion flow or per-frame optical flow. Actually, our pipeline achieves text-to-per-frame condition generation and then video generation. We humbly ask for your kind advice if we have missed some work. \n\nAs an alternative, we compare our method against Tune-a-Video, which introduces a reference video as a much stronger prior than ours. Figure 2 in the one-page PDF shows that Tune-a-Video benefits from the strong prior but is also highly limited by the strong bias from the reference video, failing to generate videos of specified subjects yet similar and consistent movements. On the contrary, our method can generate diverse videos corresponding to the input text.\n\n> **The ability of LLM (part 1)**: Since I'm not the expert in NLP domain, I'm concerned about the ability of LLM to generate very reasonable or temporally-coherent per-frame prompts.\n\nIn our practice, the LLMs can actually produce reasonable and temporally-coherent per-frame prompts because LLMs can learn common behaviors, phenomena, and events as well as their co-occurrence relationships, sequence patterns, and semantic associations from large-scale text data. All the latest LLMs we experiment with, such as GPT-3.5, GPT-4, bard, etc., are capable of decomposing actions like ”jumping into the water“ and translating \"say hello\" into waving hands and smelling. Our user study also suggests that raters perceive almost all video content trajectories as reasonable.\nCertainly, LLMs do have their limitations: they cannot provide guidance as precise as other strong conditions, such as pose sequences, as mentioned above. However, we claim this is also a benefit to keep the flexibility and generalization to generate a variety of videos from one given prompt. Coherence of details not provided by LLMs can be constrained and improved by our other three techniques, i.e., joint noise sampling, attention shift, and dual-path interpolation.\n\n> **The ability of LLM (part 2)**: I'm also curious about the feature trajectory of such text features. Do the authors think that, a smooth feature transition among frames can guarantee a smooth video? If so, a naive idea for improvement of this work is to add some regularizations to constrain the smoothness of text features across frames.\n\nThank the reviewer for this suggestion! We consider that all text features in one video only need to be relatively close (or similar) to each other. In our practice, most of the per-frame text features generated by ChatGPT remain relatively close to each other. Figure 3 in the one-page PDF shows average CLIP similarities (average on all cases in the test set) between six text features in each video are high. We have also applied this characteristic in our interpolation module, where we perform linear interpolation on the adjacent text features to acquire in-distribution text features for the interpolated frame. \n\nThanks to the high similarities and our proposed techniques on sampling and attention to encourage LDM to output smooth frame sequences, text features are not required to form a smooth trajectory."}, {"review_id": "nd8Ehu6CpC", "reviewer": "Reviewer_yWMb", "summary": "This paper proposes a novel approach to achieve Text-to-Video generation through Zero-Shot learning using LLMs and a diffusion-based image generation model. By using LLMs to generate detailed and varied descriptions for each frame of the video, the proposed method captures the changes in the video frames. The generated video frames are then constrained by self-attention module in SD and input noise to align with the input text and maintain a certain level of coherence. This paper also proposes a zero-shot frame interpolation algorithm based on a diffusion-based image generation model. While the current results may not be very satisfactory, this approach is a low-cost and Zero-Shot learning method that does not require additional training and can directly utilize existing SD models.", "questions": "1.What is the test set in Table 1? I did not find any relevant explanation in the paper.\n2.I didn't understand Figure 5, and what is the variable 𝜏* referring to?", "limitations": "1.The proposed pipline was not trained on video data, so compared to methods trained on videos, it is still weaker in terms of temporal consistency of the generated video frames.\n2.The proposed method is largely limited by the high-order reasoning capability of LLMs for video imagination and the image generation capability of the diffusion-based image generation model.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The entire process is Zero-Shot learning, without the need for any additional training, making the method low-cost. Moreover, compared to existing zero-shot video generation algorithms, the proposed approach has achieved significant improvements in performance.\n2.This algorithm effectively combines two pre-trained models that typically require significant computational resources for training and easy to follow.\n3.Since there is no fine-tuning or post-pretraining of the existing powerful image generation model, the proposed method preserves the ability of the model to generate high-quality images to the maximum extent. Compared to models trained on video data, the image quality of the generated frames is better.", "weaknesses": "1.Lacking a comparison of the quality of video generation guided by different large language models. Only ChatGPT was used in the experiments.\n2.The computational and time overhead during inference may increase when using LLMs.\n3.The proposed frame interpolation algorithm for video frames was not compared numerically with other state-of-the-art algorithms on the test set in the article.\n4.The ablation study only provides qualitative analysis and does not include quantitative comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Comparion with different LLMs**: Lacking a comparison of the quality of video generation guided by different large language models. Only ChatGPT was used in the experiments.\n\nThanks for your valuable feedback. Per your advice, we conduct experiments using other LLMs (GPT-4 and Bard), and the results validate that our methods are very robust to these LLMs. Due to limited space, we only show the case \"a flower is gradually blooming\" in Figure 4 in the one-page PDF. And six per-frame prompts of every LLM are given below. In practice, LLMs output prompt sequences to provide semantic guidance for frames. As long as the semantic guidance is reasonable and coherent, our proposed modules on the LDM can ensure the generation of sensible video.\n\n```\nGPT-3.5:\n> A close-up view of a green stem with buds tightly closed, waiting to bloom.\n> The buds slowly unfurl, revealing delicate petals in vibrant colors.\n> The flowers continue to bloom, forming a beautiful cluster of various shapes and hues.\n> A gentle breeze sways the blossoms, creating a mesmerizing dance of colors.\n> The cluster of flowers is now fully open, displaying a stunning arrangement of shapes and textures.\n> As the sunlight bathes the blooms, they glisten with dewdrops, exuding a sense of freshness and vitality.\nGPT-4:\n> A tight bud is suspended in the center of the frame against a background of soft pastels.\n> Small, delicate cracks appear on the surface of the bud, revealing a hint of white color within.\n> The bud begins to unfurl, petals delicately stretching outward, their hue a pure white.\n> The bloom is halfway open, the curling petals forming an intricate pattern, sunlight shimmering off their velvety surface.\n> The flower is almost fully bloomed, the vibrant petals dominating the frame, contrasting starkly with the greenery in the background.\n> The flower is in full bloom, petals splayed out in a radiant display, revealing a golden heart of stamens in the very center.\nBard:\n> A close-up of a white flower bud, tightly closed.\n> The flower bud begins to open, revealing a few white petals.\n> The flower continues to open, more petals unfolding.\n> The flower is now fully open, with a delicate white bloom.\n> The flower begins to wilt, the petals starting to curl.\n> The flower is now completely wilted, the petals falling to the ground.\n```\n\n> **The computational and time overhead** during inference may increase when using LLMs.\n\nThanks for pointing out this limitation. We agree that the cost of LLMs should be taken into consideration. However, LLMs' inference is fast currently (considering it can engage in real-time conversations with ChatGPT), which is a small overhead compared to the entire pipeline since the LDM is relatively slower for generating even a single image (20-30s). \n\n> **Quantitative comparison of interpolation**: The proposed frame interpolation algorithm for video frames was not compared numerically with other state-of-the-art algorithms on the test set in the article.\n\nPer your feedback, we conduct experiments on our frame interpolation algorithm on both human evaluation and automatic metrics. For the Image CLIP score, we average the CLIP similarity of the image features of every two adjacent frames. In the user study, we ask raters to compare the two videos from different methods. Our method surpasses another SOTA approach in all dimensions, particularly in terms of individual frame quality.\n\n||Image CLIP Score|Temporal|Fidelity|Rank|\n|:---:|:---:|:---:|:---:|:---:|\n|Ours Vs AMT [1]|74.55%|74.07%|81.48%|74.07%|\n\n[1] Li, Zhen, et al. \"AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n> **Quantitative ablation study**: The ablation study only provides qualitative analysis and does not include quantitative comparisons.\n\nThanks for your valuable feedback. Per your advice, we conducted additional ablation studies and evaluated their performance in both human evaluation and automatic metrics. As presented in all dimensions, the absence of either the noise sampling module or the attention shifting module would significantly degrade the performance, and the lack of both would result in extremely poor results.\n|Serial Prompting|Joint Noise Smapling|Shifting Self-attention|CLIP Metrices↑|Temporal↑|Identical↑|Rank↓|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|√|-|-|0.4429 / 0.4661|1.62|1.95|3.66|\n|√|√|-|0.4482 / 0.4712|1.83|2.30|3.04|\n|√|-|√| 0.4509 / 0.4736|2.52|2.71|2.19|\n|√|√|√| 0.4521 / 0.4773|3.67|3.46|1.11|\n> **Test set**: What is the test set in Table 1? I did not find any relevant explanation in the paper.\n\nThanks for pointing out this. We use prompts from the webpage of Text2Video-Zero, and we add more prompts that incorporate more complex event content. Our motivation is to validate whether our method achieves better results in the existing cases but also has an impressive effect on cases with rich semantic content (such as how to showcase flower blooming, saying hello, ice melting, etc.). The quantity and diversity of videos in our test set exceed that of the current methods for zero-shot video generation. \n\n> $τ^*$: I didn't understand Figure 5, and what is the variable 𝜏* referring to?\n\n$τ^*$ is the hyperparameter in the interpolation module, which is the step threshold of changing the weight from focusing on contextual frame paths to the self-denoising path. Figure 5 illustrates the influence of the selection of $τ^*$. We sincerely apologize for the oversight in Figure 5, where the image for $τ^*$ approaching 1000 and $τ^*$approaching 1 are switched. \n\nWe defined our interpolation step in Eq.8, in which $m(t)$ varies with denoising timesteps and controls the weight of two paths. To make the approach simpler, we set $m(t)$ as a step function empirically in the experiments. It can be formulated as $m(t)=0.1, t\\geqτ^*; m(t)=1,t<τ^*$ (larger t indicates earlier denoising stage)."}, {"review_id": "mX74kDagap", "reviewer": "Reviewer_6FyB", "summary": "This paper presents a novel zero-shot text-video generation method that leverages an LLM as a director to generate per-frame prompts fed into a pre-trained text-image model such as Stable Diffusion. In order to retain temporal coherence, the authors propose (1) a noise distribution consisting of an interpolation of global and local noise, and (2) attend to prior frames in the self-attention layers during the generation process. In addition, they propose a dual interpolation method to generate higher fps video.\n", "questions": "- How is $\\lambda$ chosen during generation? Is it fixed, or does it change over the video timestep?\n- How does this method behave in scenes with more content? (e.g. more people, more animals). Can the LLM directory + SD model distinguish specifications of motions between two humans doing different things?\n- How about scenes with larger shifts in motion (e.g. a camera panning to the left)? As in this case frame 0 (concatenated to the self-attention) would be less useful.\n- Or more complex motion that may be difficult to describe in words (e.g. each step in a dance)?\n", "limitations": "Yes, the authors discuss limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to understand\n- The presented method is novel and interesting, and outperforms SOTA zero-shot text-video methods, as well as SOTA trained video generation methods in general frame fidelity and visual semantics, while requiring not text-video training itself\n- This paper presents an interesting direction in leveraging an LLM’s understanding of the world through language for video generation\n", "weaknesses": "- Quantitative results / evaluation are a little weak, given the relatively small sample size (20 prompts, 4 videos each)\n- Results of ablations are shown qualitatively, and could be improved more through more rigorous quantitative evaluations\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Benchmark**:  Quantitative results/evaluation are a little weak, given the relatively small sample size.\n\nThanks for pointing out this. The standard benchmarks for video generation, especially for zero-shot video generation, are still building up in the community. The videos in our benchmark are diverse, including Text2Video-Zero prompts and prompts with rich semantic content (such as how to showcase flower blooming, saying hello, ice melting, etc.).\n\nHere, we enlarge our benchmark to 68 prompts with 4 videos each due to time limitations. Note that both the quantity and diversity of videos in our benchmark exceed that in previous methods.\n\n| Method | Training-Free | CLIP Metrics↑| Fidelity↑ | Temporal↑ | Semantic↑ | Rank↓ |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: |\n|VideoFusion [1]| - | 0.471| 3.386 | 3.914 | 3.209 | 2.368 |\n| LVDM [2] | - | 0.476 | 3.276 | 3.674 | 3.206 | 2.558 |\n| T2V-Zero [3] | √ | 0.482 | 3.469 | 2.721 | 3.003 | 3.033 |\n| ours | √ | 0.471/0.483* | 4.185 | 3.318 | 3.909 | 2.042 |\n\n[1] Luo, Zhengxiong, et al. \"VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n[2] He, Yingqing, et al. \"Latent video diffusion models for high-fidelity video generation with arbitrary lengths.\" *arXiv preprint arXiv:2211.13221* (2022).\n\n[3] Khachatryan, Levon, et al. \"Text2video-zero: Text-to-image diffusion models are zero-shot video generators.\" *arXiv preprint arXiv:2303.13439* (2023).\n\n> **Quantitative ablations**:  Results of ablations are shown qualitatively, and could be improved more through more rigorous quantitative evaluations \n\nThanks for your valuable feedback. Per your advice, we conducted additional ablation studies and evaluated their performance in both human evaluation and automatic metrics. As presented in all dimensions, the absence of either the noise sampling module or the attention shifting module would significantly degrade the performance, and the lack of both would result in extremely poor results.\n|Serial Prompting|Joint Noise Smapling|Shifting Self-attention|CLIP Metrices↑|Temporal↑|Identical↑|Rank↓|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|√|-|-|0.4429 / 0.4661|1.62|1.95|3.66|\n|√|√|-|0.4482 / 0.4712|1.83|2.30|3.04|\n|√|-|√| 0.4509 / 0.4736|2.52|2.71|2.19|\n|√|√|√| 0.4521 / 0.4773|3.67|3.46|1.11|\n> $λ$: How is $λ$ chosen during generation? Is it fixed, or does it change over the video timestep?\n\nThe hyperparameter setting of $λ=0.2$ is recommended for most cases. $λ$ is fixed and does not change over the video timestep. $\\lambda$ is used in the joint noise sampling module to regulate the ratio between the independent noise and the united noise. A video’s per-frame noise weighs from the frame-wise independent noise and one unified video noise, where the weights of the two noises are the same across different frames. \n>**Scenes with more content**: How does this method behave in scenes with more content? (e.g. more people, more animals). Can the LLM directory + SD model distinguish specifications of motions between two humans doing different things?\n\nWe supplemented some additional examples of scenes with more subjects in Figure 5 in the one-page PDF. As shown in the case “a dog is running and then another dog joins”, our pipeline performs well on simple scenarios involving two or three objects performing similar actions.\n\nHowever, when involving complex scenes with different subjects with diverse features performing distinct actions, such as shown in the case “A man is dancing and a woman is watching”, our method does indeed face limitations. In our practice, LLMs can separate and describe the detailed actions of subjects within each sentence in a reliable manner. In contrast, the current version of LDMs struggles to associate the action with the corresponding subject. If a more powerful LDM emerges in the future, we believe our pipeline holds the potential to achieve this as well. \n>**Scenes with larger shifts in motion**: How about scenes with larger shifts in motion (e.g. a camera panning to the left)? As in this case frame 0 (concatenated to the self-attention) would be less useful.\n\nThank you for considering more general applications and scenes. Unfortunately, this challenging camera panning lies beyond the scope and capability of our research, posing a significant challenge even for models trained on massive video data. \n\nThe reason is that when involving camera panning, the model is needed and expected to construct from or at least have the ability to image a 3D scene from one single frame, which is very challenging. Also, the current LDMs remain unresponsive to layout descriptions, hindering the realization of scene shifting even with detailed prompts like “subject on the left”, ”… in the middle”, ”… on the right”, etc. \n\n>**Complex motion in words**:  Or more complex motion that may be difficult to describe in words (e.g. each step in a dance)?\n\nOur pipeline is capable of generating corresponding results for some straightforward action sequences, such as raising and lowering the right hand, lifting the left foot, and similar motions. Figure 6 in the one-page PDF shows these cases.\n\nBut, our method indeed is limited in cases of intricate dance movements, where the pose details are required. On the one hand, LLMs often fail to provide prompts detailed at fine granularity enough, which is also the limitation of using natural language as the condition. For example, for the input “a man is hip hop dancing”, the LLMs are likely to provide vague and high-level state descriptions like “does the ‘wave’ by creating a ripple effect through his body”. On the other hand, the LDMs cannot generate images that match prompts with highly detailed descriptions. How to make LDMs comprehend complex prompts, including spatial directions, colors, and textual information, remains a prominent research focus in the T2I diffusion area."}]

Author rebuttal:
We thank all reviewers for engaging in the review process. Our code will be made public upon acceptance. 

We are deeply encouraged by positive comments from the reviewers. We appreciate the recognition and endorsement of our proposed zero-shot pipeline, such as acknowledging its qualities as interesting (VS1k and 6FyB), novel (6FyB and yWMb), effective (yWMb), and technically sound (mgrm). VS1k, yWMb, and 6FyB agree that our method generates videos with high per-frame quality (fidelity). Both mgrm and 6FyB agree that our paper is “clear”, “well-written”, and “easy to understand”. 

---

In our individual replies, we attempted to address specific questions and comments as clearly and detailed as possible. 

Moreover, we added several additional results to the one-page PDF and the individual replies to strengthen our work overall. Here, we briefly summarize these additional experiments and evaluations:

- Supplemented ablation study of (a) numerical comparison on the test set (b) combining our techniques with Text2Video-Zero.
- Quantitative comparison of interpolation module.
- Quantitative results in an enlarged test set.
- Qualitative results of Tune-A-Video under complex conditions.
- Visualization of CLIP similarities on cases between all per-frame prompts in our test set.
- Qualitative results of using other LLMs.
- Qualitative results of videos with multiple subjects.
- Qualitative results of videos with straightforward motions.

---

We hope that these additional results further strengthen Free-Bloom’s position as the state-of-the-art generative model of zero-shot text-to-video generation and demonstrate:

- Our pipeline is novel to combine the LLM director and the LDM animator and is the first T2V work capable of generating semantic-coherent videos (such as the whole process of flower blooming rather than a set of “moving images”) in a zero-shot and training-free manner.
- Our Free-Bloom can generate flexible and versatile videos according to one unified prompt in a general level of scenes.
- Our novel techniques of sampling, attention, and interpolation can greatly improve the temporal coherence of the generated videos while being faithful to the rich semantic content.
- Our Free-Bloom is robust to different LLMs and is compatible with the existing LDM-based extensions.

Meta-review: The paper received 5/7/5/7 ratings from reviewers. The initial review raised concerns regarding novelty/contribution of the paper as well as experimental evaluations. Those concerns were addressed through rebuttal discussion and two of the reviewers raised their scores. The AC acknowledges that the best practice of utilizing LLM and LDM still need exploration and this work benefit the research development over the direction. Taking all those into consideration, the AC recommends accepting the paper.
Decision: Accept (poster)