SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} p \left(\tilde {\mathbf {x}} _ {T} ^ {1: f}\right) = \mathcal {N} \left(\mathbf {0}, \sin^ {2} \left(\frac {\pi}{2} \lambda\right) \mathbf {I} _ {n f} + \cos^ {2} \left(\frac {\pi}{2} \lambda\right) \mathbf {J} _ {f} \otimes \mathbf {I} _ {n}\right)) \tag {15} \\ = \mathcal {N} (\mathbf {0}, \mathbf {I} _ {n f} + \cos^ {2} \left(\frac {\pi}{2} \lambda\right) \left(\left(\mathbf {J} _ {f} - \mathbf {I} _ {f}\right) \otimes \mathbf {I} _ {n}\right)) \\ \end{array}
$$
In this distribution, without given noises of other frames, for any frame noise  $\tilde{\mathbf{x}}_T^i$ , it still follows a standard normal distribution that  $p(\tilde{\mathbf{x}}_T^i) = \mathcal{N}(\mathbf{0},\mathbf{I}_n)$ .
C.1 Serial Prompting
We first prompt ChatGPT [26] with the following instruction:
- I would like you to play the role of the descriptor of each frame of the video as a director of a movie. The content of each video should be concise and only clearly describe the subject. Each sentence in the video is independent. Every sentence needs to include the subject's appearance and actions, please describe the main actions of the object and the extent of the actions in as much detail as possible. The sentences of each picture are independent, and each sentence should describe what exists in the picture. Each frame is described in only one sentence. Suppose there is a video about "[INPUT PROMPT]" and there are "[F]" frames in the video. Describe the content of each frame separately. Please be straightforward and do not use a narrative style.
Then, we use the following prompt:
- Now perform Coreference Resolution on the above sentence, replace reflexive pronouns with their original vocabulary, and eliminate the discourse cohesion. Keep the meaning the same. The sentence for each frame should be able to fully express all the visual information of the frame. Also, the linguistic structure of each sentence should be simple and similar.
C.2 Test Set for Quantitative Results
We list some prompts from our test set in Table 2, in which some prompts are from the webpage of Text2Video-Zero [17] and some are designed by ourselves that incorporate more complex event content.
Table 2: Prompts for Test Set.
A cluster of flowers blooms
Use pan to fire an egg
Volcano eruption
A dog is walking down the street
Light a match then the match goes out
River freezes
Two men shake hands
Teddy bear jumps into water
The growth of a sapling
A dancing mickey	Astronaut riding a horse
Iron man is surfing
The ice cube is melting
A panda is walking down the street
The Santa flying through the sky
Two supermen are fighting
A bear dancing on times square
An astronaut is waving his hands on the moon
An egg hatch into a chick
Teddy bear is greeting
C.3 Details of User Study
We conduct a user study to understand how humans would evaluate the current text-to-video methods. The survey contains a total of 20 prompts with each prompt having 4 videos output from VideoFusion [21], LVDM [10], Text2Video-Zero [17], and Ours. For each prompt, we ask raters to answer the following four questions:
- How would you rate the temporal coherence and smoothness of the videos? Please assign a score for their continuity. (Temporal Coherence)
- How would you rate the quality and fidelity of the individual frames in the videos? Please assign a score for the visual quality. (Fidelity)
- How well does the video depict the content described in the text? Please assign a score for its content representation. (Semantic Coherence)
- Based on your overall perception, please rank the videos. (Rank)
Figure 14 presents an example interface of our survey. We received valid responses from a total of 80 individuals from both industry and academia.
C.4 Code Used and License
Table 3: The used codes and license.
URL	Citation	License
https://github.com/showlab/Tune-A-Video	[53]	Apache License 2.0
https://github.com/google/prompt-to-prompt	[11]	Apache License 2.0
https://github.com/huggingface/diffusers	[48]	Apache License 2.0
https://github.com/Picsart-AI-Research/Text2Video-Zero	[17]	CreativeML Open RAIL-M
https://github.com/VideoCrafter/VideoCrafter	[10]	(Hugging Face Space) MIT
https://github.com/modelscope/modelscope/	[21]	Apache License 2.0
All used codes and their licenses are listed in Table 3.
D.1 Qualitative Results
We showcase more visualization of the generated video in this section. In Figure 6 and Figure 7, we present the full comparison with Text2Video-Zero [17], VideoFusion [21], and LVDM [10]. In Figure 8, we randomly generate multiple videos with respect to the same prompts. In Figure 9, we demonstrate more results of our interpolation empowerment module.
D.2 User Study Quantitative Comparisons
Table 4: User Study Comparisons.
Method	Training-Free	User Study
Rank	Fidelity	Temporal	Semantic
Ours vs. LVDM [10]	55.00%	85.28%	53.33%	82.27%
Ours vs. VideoFusion [21]	63.06%	82.50%	44.44%	78.33%
Ours vs. T2V-Zero [17]	✓	73.61%	87.78%	80.00%	85.28%
For the user study part in the quantitative results, we also present the comparison-based results here in Table 4. Specifically, for the ranking column, the number denotes the percentage of participants who prefer our method and rank us before another. For the dimensions of fidelity, temporal coherence, and semantic coherence, the numbers indicate the percentage of participants who believe that our generated videos are better to that of another method in that dimension.
D.3 Analysis on Joint Noise Sampling
We additionally analyze the effect of our noise sampling method. In part A of Figure 10, we sample the noise the same as equation  $\tilde{\mathbf{x}}_T^{1:f} \coloneqq \cos\left(\frac{\pi}{2}\lambda\right)\mathbf{x}_T^{1:f} + \sin\left(\frac{\pi}{2}\lambda\right)\pmb{\delta}_T^{1:f}$  using sin cos weighting. While in part B, we modify the weight of the unified noise and the individual noise as
$$
\tilde {\mathbf {x}} _ {T} ^ {1: f} := (1 - \lambda) \mathbf {x} _ {T} ^ {1: f} + \lambda \delta_ {T} ^ {1: f} \tag {16}
$$
However, this would disrupt the single-frame noise from following normal Gaussian Distribution. As we can observe, in this way, LDM fails to generate reasonable images.
D.4 Extensions
In this section, we demonstrate the extensibility of our approach with more examples. Inspired by Phenaki [47] which can generate story-based conditional videos based on a sequence of prompts, we also apply our method to the same task, which is presented in Figure 11. In Figure 12, we showcase some results of combining DreamBooth [34] to include personalized concepts in the generated videos. In Figure 13, we showcase the results of generating videos based on the given first frame by leveraging DDIM inversion [43].
"Two superman are fighting."
"The growth of a sapling."
Figure 6: Additional Qualitative Comparisons. In the case of "Two supermen are fighting", the LLM vividly decomposes the process of fighting into frames, with the fifth frame depicting "colliding in a dazzling display of sparks and force", which is captured in our result. In the case of "the growth of a sapling", our result clearly presents the gradual sprouting of a small sapling.
"Light a match then the match goes out."
"A teddy bear is greeting."
Figure 7: Additional Qualitative Comparisons. In the case of "light a match then the match goes out", our method successfully depicts the entire process of a match from lightning, burning to extinguishing. In the case of "a teddy bear is greeting", we exploit the world knowledge of LLM [26] to translate a greeting into a series of specific actions such as waving and smiling.
"A flower gradually blooms."
"Volcano eruption."
"A dancing mickey."
Figure 8: Additional Qualitative Results. Multiple results based on the same prompts are shown.
Figure 9: Qualitative Results from Interpolation Module. We interpolate 4 frames between each pair of original neighboring frames. Our interpolation module enables smooth transitions between two key states.
(A)  $\cos \left(\frac{\pi}{2}\lambda\right)\mathbf{x}_T^{1:f} + \sin \left(\frac{\pi}{2}\lambda\right)\pmb{\delta}_T^{1:f}$
(B)  $(1 - \lambda)\mathbf{x}_T^{1:f} + \lambda \pmb{\delta}_T^{1:f}$
Figure 10: Analysis on Joint Noise Sampling. Without our proposed sampling, the initial noise at each single frame would not follow normal Gaussian distribution, resulting in corrupting frames.
1st prompt:"A flower is blooming"
2nd prompt: "The flower in the rain"
3rd prompt: "The flower gradually freezes"
Figure 11: Extension - Long Video Story. Our method can generate the long video story based on a sequence of prompts.
$<ccorgi\ dog>$  is sitting down."
"A princess is waving her hands, <modern disney style>."
"  $<$  sks mr potato head> is dancing."
Figure 12: Extension - Personalization. Our method can generate videos with user-specific concepts. The tokens of "ccorgi dog", "modern disney style", and "sks mr potato head" are from their respective personalized models ccorgi-dog, Mo di Diffusion, and Mr Potato Head.
Figure 13: Extension - Making an Image Move. Our method can generate videos based on the first frame and its corresponding prompt by combining our method with DDIM inversion [43].
User Study on Text-to-Video Generation
Thank you very much for helping us with our research! In this survey, we will present text descriptions along with corresponding videos, which are generated by four state-of-the-art text-to-video AI models. Each question will include a text prompt and the corresponding generated videos. Your task is to rank its performance across different dimensions. Additionally, please provide a final comprehensive ranking based on your preferences.
This survey consists of 5 questions and will approximately take  $10\mathrm{min}$  of your time.
Sign in to Google to save your progress. Learn more
* Indicates required question
Below are four videos generated from "A cluster of flowers blooms"
How would you rate the temporal coherence and smoothness of the videos? Please assign a score for their continuity.
Figure 14: Interface of surveys from the user study.