SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanzhuo Huang* Yufan Feng* Cheng Shi Lan Xu Jingyi Yu Sibei Yang†
School of Information Science and Technology, ShanghaiTech University https://github.com/SooLab/Free-Bloom
Text-to-video is a rapidly growing research area that aims to generate a semantic, identical, and temporal coherence sequence of frames that accurately align with the input text prompt. This study focuses on zero-shot text-to-video generation considering the data- and cost-efficient. To generate a semantic-coherent video, exhibiting a rich portrayal of temporal semantics such as the whole process of flower blooming rather than a set of "moving images", we propose a novel Free-Bloom pipeline that harnesses large language models (LLMs) as the director to generate a semantic-coherence prompt sequence, while pre-trained latent diffusion models (LDMs) as the animator to generate the high fidelity frames. Furthermore, to ensure temporal and identical coherence while maintaining semantic coherence, we propose a series of annotative modifications to adapting LDMs in the reverse process, including joint noise sampling, step-aware attention shift, and dual-path interpolation. Without any video data and training requirements, Free-Bloom generates vivid and high-quality videos, awe-inspiring in generating complex scenes with semantic meaningful frame sequences. In addition, Free-Bloom is naturally compatible with LDMs-based extensions.
Recent impressive breakthroughs [31, 33, 35] in text-to-image synthesis have been attained by training diffusion models on large-scale multimodal datasets [37, 38] comprising billions of text-image pairs. The resulting images are unprecedentedly diverse and photo-realistic while maintaining coherence with the input text prompt. Nevertheless, extending this idea to text-to-video generation poses challenges as it requires substantial quantities of annotated text-video data and considerable computational resources. Instead, recent studies [53, 19, 40, 28, 5, 49, 17] introduce adapting pretrained image diffusion models to the video domain in a data-efficient and cost-efficient manner, showing promising potential in one-shot video generation and zero-shot video editing.
This work takes the study of zero-shot text-to-video generation further, enabling the generation of diverse videos without the need for any video data. Instead of relying on a reference video for generation or editing [19, 28, 5, 40], the proposed approach generates videos from scratch solely conditioned on the text prompt. While a concurrent work, Text2Video-Zero [17], similarly focuses on zero-shot video generation, our study and Text2Video-Zero differ significantly not only in terms of technical contributions but also in motivation: we aim to generate complete videos that encompass meaningful temporal variations, which distinguish it from the generation with "moving images". As shown in Figure 1, with the text prompt of "a flower is gradually blooming", our approach generates a video that thoroughly depicts the entire process, seamlessly progressing from a flower bud to the
Video of "flower gradually blooming"?
Figure 1: The zero-shot Free-Bloom generates vivid and high-quality videos that are imbued with semantic significance according to the input prompt. On top of this, Free-Bloom is compatible with the existing LDM-based extensions and can be applied to other tasks such as personalization of user-specific concepts and video prediction from a start frame.
full blooming stage. This set ours apart from other methods [17, 21, 10] that often portray a single stage of the flower bloom. In other words, our resulting video is semantic-coherent that exhibits a rich portrayal of temporal semantics.
In this work, we introduce Free-Bloom, a zero-shot, training-free, semantic-coherent text-to-video generator that is capable of generating high-quality, temporally consistent, and semantically aligned videos. Our key insight is to harness large language models (LLMs) as the director, while pretrained image diffusion models as the frame animator. The underlying idea is that LLMs [7, 4, 30, 45], pre-trained on massive amounts of text data, encode general world knowledge [32, 27], thus being able to transform the text prompt into a sequence of prompts that narrates the progression of events over time. Simultaneously, pre-trained image diffusion models, such as latent diffusion models (LDMs) [33] generate the sequential frames conditioned on the prompt sequence to create a video. Moreover, considering the temporal resolution of the prompt sequence is difficult to fine-grained each frame of a video, thereby a zero-shot frame interpolation is employed to expand the video to a higher temporal resolution. Our insights shape Free-Bloom's progressive pipeline, which consists of three sequential modules: serial prompting, video generation, and interpolation empowerment.
The naive attempt of directly applying LLMs and LDMs respectively to the first two steps of our pipeline resulted in failure. Not surprisingly, the resulting images are completely independent of each other. Although they each match with their own prompt, the semantic content of these images is entirely disjointed (semantic coherence), the overall content varies greatly including the foreground and the background (identical coherence), and the adjacent frames cannot be smoothly connected (temporal coherence). To overcome these problems, we propose a series of novel and effective technical solutions. For semantic coherence, we instruct LLMs in a multi-stage manner, completing the missing information in the generated prompt sequence caused by discourse cohesion, and ensuring that each prompt in the sequence accurately describes the detailed visual content while maintaining a consistent linguistic structure across prompts. To ensure both identical coherence and temporal coherence, we propose two innovative modifications for jointly adapting LDMs for video generation: (1) Through modeling the joint distribution of initial noise latent across frames from both unified and individual noise latent distribution, we enhance temporal and content consistency while preserving suitable levels of perturbation. Notably, in order to conform to the LDMs' assumption [33], our joint distribution ensures that the noise latent at every single frame follows normal Gaussian distribution
when the noise latent of other frames is not provided. (2) Considering the trade-off between continuity and adherence to single-frame semantics, we carefully incorporate cross-frame attention [53], which focuses on contextual frames, into self-attention layers according to denoising time steps.
We further propose the training-free interpolation empowerment module to improve the temporal resolution of the generated videos. In order to maintain the aforementioned semantic coherence, identical coherence, and temporal coherence, we consider semantic contents of both contextual and current frames. We perform a novel dual interpolation on latent variables relying on both contextual frames and self-denoising paths, as the former path enables smooth transitions between neighboring frames and the latter path ensures high fidelity of single frame.
In summary, our contributions are multi-fold: (1) We introduce Free-Bloom, a novel pipeline to tackle the zero-shot text-to-video task. Our pipeline effectively harnesses the rich world knowledge of LLMs and the powerful generative capability of LDMs, proposing an insight into adapting text-to-image models for video generation. (2) We propose a video generation module incorporating joint noise sampling and step-aware attention shift, ensuring identical coherence and temporal coherence while expressing the semantic content. (3) We introduce a training-free dual-path interpolation strategy, ensuring consistency with context while maintaining fidelity. (4) Free-Bloom can generate impressive videos imbued with semantic significance corresponding with the contextual narrative.
Diffusion Models for Images. Diffusion models [42] and its variants DDPM [12] and DDIM [43], have achieved breakthroughs on text-to-image generation tasks [31, 8, 35, 33]. Moreover, diffusion models have a thriving research and application ecosystem, including multiple works [22, 11, 57, 23] as well as emerging open-source communities and libraries [48] with frameworks and plugins.
Open-Domain Text-to-Video Generation. Currently, both non-diffusion-based [15, 52, 15, 47] and diffusion-based [14, 41, 59, 21, 1, 13, 10, 3] T2V methods often conduct training on large-scale video datasets such as WebVid and HD-Vila-100M[2, 55], while leveraging T2I priors or jointly training with images to maintain the quality and the content diversity. However, even with datasets of millions of videos, the quality and quantity of the training set are still not comparable to images for training.
Zero/Few-shot Video Synthesis. Recently, tuning pre-trained T2I models under zero-shot/few-shot settings has also been found promising for video generation. Tune-A-Video [53] uses a reference video, generating videos conditioned on varied prompts while maintaining the original motion. Several works [19, 40, 28, 5, 49, 54] further explore the video editing task but are not able to either change the motion or generate videos with new events. Our concurrent work is Text2Video-Zero [17], which firstly proposes a zero-shot T2V pipeline started from pre-trained T2I models. While it models the motion flow by adding linear transformations to latent codes, we address that complex state transitions should be considered instead.
LLM-assisted Generations. Large Language Models (LLMs) [7, 4, 30, 45] have made significant impact by exhibiting remarkable performance across multiple Natural Language Processing tasks. From the immense training corpus, LLMs have captured open-world knowledge in various fields [24, 4, 27, 32, 44, 58]. To assist generation by LLMs, previous methods [51, 25, 20, 16, 39] are mainly based on prompt engineering, aiming to make the text prompts more expressive toward their goals. Our methods further exploit knowledge from LLMs to derive frame visual content in a complete video.
Denoising Diffusion Probabilistic Model. DDPM [12] is a type of probabilistic model that learns to approximate the probability distribution of the true data. The forward diffusion process over  $\mathbf{x}_0, \dots, \mathbf{x}_T$  gradually adds Gaussian noises in  $T$  time steps to corrupt an image. Then, the model is optimized to learn the denoising transitions  $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_t)$  in the reverse process to turn noises into images. The forward posteriors can be expressed as
$$
q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) = \mathcal {N} \left(\mathbf {x} _ {t - 1}; \tilde {\boldsymbol {\mu}} _ {t} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}\right), \tilde {\beta} _ {t} \mathbf {I}\right) \tag {1}
$$
where  $\tilde{\pmb{\mu}}_t\left(\mathbf{x}_t,\mathbf{x}_0\right)\coloneqq \frac{\sqrt{\bar{\alpha}_{t - 1}}\beta_t}{1 - \bar{\alpha}_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t(1 - \bar{\alpha}_{t - 1})}}{1 - \bar{\alpha}_t}\mathbf{x}_t$  and  $\tilde{\beta}_t\coloneqq \frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_t}\beta_t$
Denosing Diffusion Implicit Models. DDIM [43] is a generalized form of DDPMs by introducing the non-Markovian processes. DDIM can speed up inference by using fewer denoising steps with the same training objective of DDPMs, where the reverse process can be written as follows,
$$
\mathbf {x} _ {t - 1} = \sqrt {\alpha_ {t - 1}} \underbrace {\left(\frac {\mathbf {x} _ {t} - \sqrt {1 - \alpha_ {t}} \boldsymbol {\epsilon} _ {t} (\mathbf {x} _ {t} ; \theta)}{\sqrt {\alpha_ {t}}}\right)} _ {\text {" p r e d i c t e d x _ {0} " d e n o t e d a s P _ {t}}} + \underbrace {\sqrt {1 - \alpha_ {t - 1} - \sigma_ {t} ^ {2}} \cdot \boldsymbol {\epsilon} _ {t} (\mathbf {x} _ {t} ; \theta)} _ {\text {" d i r e c t i o n p o i n t i n g t o x _ {t} " d e n o t e d a s D _ {t}}} + \underbrace {\sigma_ {t} \mathbf {z} _ {t}} _ {\text {r a n d o m n o i s e}} \tag {2}
$$
where  $\mathbf{P}_t\coloneqq \frac{\mathbf{x}_t - \sqrt{1 - \alpha_t}\pmb{\epsilon}_t(\mathbf{x}_t)}{\sqrt{\alpha_t}}$  and  $\mathbf{D}_t\coloneqq \sqrt{1 - \alpha_{t - 1} - \sigma_t^2}\pmb {\epsilon}_t(\mathbf{x}_t)$
Latent Diffusion Models. LDMs [33] map data to a low-dimensional space, termed latent space, which possesses a strong representational capacity and can capture complex and abstract features. LDM employs a multi-layer attention-based U-Net architecture for noise prediction. The attention blocks [46] contain both self-attention and cross-attention, which focus on image content and textual content respectively. In the self-attention layer, the input feature  $\mathbf{z}_i$  is projected into query, key, and value through linear transformations, then they are utilized to compute the attention weights.
4.1 Free-Bloom: LLMs as The Director and LDMs as The Frame Animator
Figure 2: Overview of Free-Bloom. Our pipeline consists of three sequential stages. In Serial Prompting stage, the LLM is prompted to generate serial frame prompts. In Video Generation stage, modifications are made to the LDM to generate coherent frames by joint noise sampling and step-aware attention shift. In Interpolation Empowerment stage, a dual latent space interpolation conditioned on both contextual path and denoising path is proposed to generate intermediate frames.
We propose a zero-shot text-to-video generation pipeline named Free-Bloom, in which the generated videos are directed by LLMs and animated by LDMs. As shown in Figure 2, our pipeline consists of three sequential stages: serial prompting, video generation, and interpolation empowerment. Given a text prompt  $\mathcal{T}$ , Free-Bloom first generates a video  $\mathcal{V}_1 \in \mathbb{R}^{f \times 3 \times h \times w}$  with low frame rates at the first two stages, which include  $f$  frames with the frame size of  $h \times w$ . Then, the interpolation empowerment module fills in gaps between frames to improve continuity, resulting in the final video.
At the serial prompting stage, we first prompt the LLM with their general knowledge to transform the raw prompt into a series of prompts indicating the change of semantic content over time. Then, we instruct the LLM with referential resolution on the prompts to ensure that each prompt accurately describes the detailed visual content while maintaining a consistent linguistic structure across prompts, resulting in a prompt sequence,  $\mathcal{T}^{1:f} = \{\mathcal{T}^1,\dots ,\mathcal{T}^f\}$ . The prompt sequence accurately depicts the overall narrative and can effectively enlighten LDM [33] to generate semantic-coherence frames.
At the video generation stage, we employ two novel modifications to LDM, enabling it to generate semantic coherent, identical coherent, and temporal coherent video frames. We first simultaneously sample the noise at every frame from their joint Gaussian distribution, which is constructed by considering a unified noise at the video level and an individual noise at the frame level, which makes
it easier to generate coherence frames with certain variations. Then, we propose a strategy to modify the self-attention layer during the inference process. The modified attention layer adjusts attention to contextual information and self-consistent content according to the denoising time step.
At the interpolation empowerment stage, a dual latent space interpolation strategy is proposed to generate the intermediate frame between two neighboring frames. In addition to jointly interpolating the latent variables of neighboring frames for temporal coherence, we condition the latent variable generated by performing DDIM [43] on the interpolated text embedding to ensure semantic coherence. Also, the weights of the two paths vary over the denoising time step to ensure smoothly compatible.
4.2 Serial Prompting for Zero-Shot Text-to-Video Generation
Prompting a Video. When discussing the scenario of "a teddy bear jumping into the water", humans naturally imagine the series of events: the teddy bear crouches down, then propels itself into the air, and finally lands in the water with a splash. The prior knowledge of jumping water allows us humans to derive a semantic meaning of sequential events from a general description, which inspires us to ask: do LLMs also encode this kind of knowledge without further training? In our investigation of LLMs, we prompt the LLMs with instructions such as "describe each frame". We find that LLMs incorporate extensive world knowledge and can provide temporal transition knowledge.
However, the generated initial descriptions have free linguistic structure, and the text is fragmented in every single description due to discourse coherence, leading to falling short of adequately sufficient information for a single frame, as shown in Figure 2. Therefore, to ensure semantic coherence of prompts across frames, we further instruct the LLMs with the above-obtained initial descriptions to generate a sequence of  $f$  prompts  $\mathcal{T}^{1:f} = \{\mathcal{T}^1,\dots ,\mathcal{T}^f\}$  with consistent linguistic structures, where each prompt accurately describes the visual content in detail.
Zero-Shot Video Generation with LDMs. The video generation module aims to condition the LDMs on the prompt sequence  $\mathcal{T}^{1:f}$  to generate frames that are semantic, identical, and temporal coherence. However, simply generating  $f$  still images from different prompts results in a collection of unrelated images that cannot be sequenced into a coherent video. To address this issue, we propose two novel modifications for adapting LDMs to generate videos rather than a collection of images without the need for additional training.
- Joint noise sampling following LDMs' assumption. To model the coherence across frames, we propose to sample the initial noises in the diffusion process of frames from their joint probability distribution instead of independent distribution. To construct such joint distribution, we first investigate the effects of independent noise and united noise over sequential frames: (1) The same noise for every frame results in LDMs generating a sequence of images with similar content under similar textual conditions. While this feature may contribute to achieving temporal coherence among frames, it can potentially restrict the natural variation of the subject, significantly reducing the overall video quality. (2) Generating images from  $f$  independent noises issues in maintaining consistency across frames, although the generated images with naturally varying content showcase more diversity and independently captivating elements.
These observations motivate us to construct the joint distribution by considering the independent and united distribution jointly. Specifically, we propose to obtain the initial noisy latent variable by weighted summation over a unified noise across the video frames and an individual perturbed noise to maintain consistency across frames while introducing appropriate variation. Moreover, to conform to the LDMs' assumption, we design the weighting coefficient to ensure that without giving initial noise latent at other frames, the initial noise at every single frame follows normal Gaussian distribution. Let us denote the unified video noise as  $\mathbf{x}_T^*$  and the independent noise as  $\mathbf{x}_T^i$ , the joint distributions  $\mathbf{x}_T^{1:f}$  and  $\delta_T^{1:f}$  are formulated as follows,
$$
\mathbf {x} _ {T} ^ {1: f} = \left[ \mathbf {x} _ {T} ^ {*}, \dots , \mathbf {x} _ {T} ^ {*} \right] ^ {T}, \mathbf {x} _ {T} ^ {*} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {n}) \quad \Rightarrow \quad p \left(\mathbf {x} _ {T} ^ {1: f}\right) = \mathcal {N} \left(\mathbf {0}, \mathbf {J} _ {f} \otimes \mathbf {I} _ {n}\right) \tag {3}
$$
$$
\boldsymbol {\delta} _ {T} ^ {1: f} = \left[ \mathbf {x} _ {T} ^ {1}, \dots , \mathbf {x} _ {T} ^ {f} \right] ^ {T}, \mathbf {x} _ {T} ^ {i} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {n}) \quad \Rightarrow \quad p \left(\boldsymbol {\delta} _ {T} ^ {1: f}\right) = \mathcal {N} (\mathbf {0}, \mathbf {I} _ {n f})
$$
where  $\mathbf{J}_f$  denotes the all-ones matrix with size of  $f\times f$  and  $\otimes$  represents the Kronecker product. Then, we model the mixture noise by
$$
\tilde {\mathbf {x}} _ {T} ^ {1: f} := \cos \left(\frac {\pi}{2} \lambda\right) \mathbf {x} _ {T} ^ {1: f} + \sin \left(\frac {\pi}{2} \lambda\right) \boldsymbol {\delta} _ {T} ^ {1: f} \tag {4}
$$
where the  $\lambda$  is a coefficient of variance rate. This mixed noise follows the joint distribution as
$$
\begin{array}{l} p \left(\tilde {\mathbf {x}} _ {T} ^ {1: f}\right) = \mathcal {N} \left(\mathbf {0}, \sin^ {2} \left(\frac {\pi}{2} \lambda\right) \mathbf {I} _ {n f} + \cos^ {2} \left(\frac {\pi}{2} \lambda\right) \mathbf {J} _ {f} \otimes \mathbf {I} _ {n}\right)) \\ = \mathcal {N} \left(\mathbf {0}, \mathbf {I} _ {n f} + \cos^ {2} \left(\frac {\pi}{2} \lambda\right) \left(\left(\mathbf {J} _ {f} - \mathbf {I} _ {f}\right) \otimes \mathbf {I} _ {n}\right)\right). \tag {5} \\ \end{array}
$$
When  $\lambda = 0$ , the initial noise becomes unified noise, and when  $\lambda = 1$ , it becomes independent noise.
- Step-aware attention shift. To further maintain identical coherence while ensuring semantic coherence, we shift the attention of the self-attention layers in the LDMs' U-Net from cross-frame contexts to the current frame according to the denoising time steps. For frame  $i$  with its query  $Q_{i}$ , previous methods [53, 5, 49, 28] with one overall text prompt retrieve the key and value from the former frame and the first frame to perform sparse spatio-temporal attention, based on the observation [53] that extending the spatial self-attention across images produces consistent content.
In our scheme, frames are conditioned on different prompts, requiring that not only maintaining temporal and identical coherence, but also semantic coherence with their respective prompts. To achieve identical and temporal coherence, we address the former and the first frame as contextual frames and attend to contextual key-value pairs. In particular, the former frame helps to enhance temporal coherence, while the first frame acts as a benchmark shape, maintaining appearance consistency throughout the video. To align with the respective prompt in semantics, we shift attention to the current frame itself as the time step increases, enabling the preservation of its specific characteristics and details. In summary, our inference strategy takes into account the time step during the diffusion process. The initial steps focus on contextual frames to form coarse-grained shapes and layouts. As we progress, complete images are generated, emphasizing producing accurate outlines conditioned on semantic information. Overall, our adaptation of the attention mechanism refers to:
$$
\text {S e l f - A t t e n t i o n} _ {i} := \left\{ \begin{array}{l l} \operatorname {A t t e n t i o n} \left(Q _ {i}, \left[ K _ {0}, K _ {i - 1}, K _ {i} \right], \left[ V _ {0}, V _ {i - 1}, V _ {i} \right]\right), & t \geq \tau \\ \operatorname {A t t e n t i o n} \left(Q _ {i}, K _ {i}, V _ {i}\right), & t <   \tau \end{array} \right. \tag {6}
$$
where  $\tau$  is the threshold of the time step for attention shift. The  $(K_0, V_0)$ ,  $(K_{i-1}, V_{i-1})$ , and  $(K_i, V_i)$  are the key-value pairs in the first, former, and current frames, respectively.
4.3 Interpolation Empowerment
The interpolation empowerment module is proposed to further increase the frame rate without extra training resources. Similar to the insight of utilizing contextual information in the "Step-aware attention shift" proposed in Section 4.2, the interpolated intermediate frames should also consider contextual frames, i.e., the former and the latter frames, in the denoising process. One naive approach is to directly derive the latent variable of the intermediate frame solely from the latent variables in contextual frames. However, this method overlooks the intermediate semantics indicated in text prompts in contextual frames, leading the semantic incoherence.