E MU V IDEO:
Factorizing Text-to-Video Generation by Explicit Image Conditioning
Rohit Girdhar†,∗Mannat Singh†,∗
Andrew Brown∗
Sai Saketh Rambhatla
Akbar ShahXi YinGenAI, Meta

Quentin Duval∗
Samaneh Azadi∗
Devi Parikh
Ishan Misra∗https://emu-video.metademolab.com/

Figure 1. E MU V IDEO can generate high quality and temporally consistent videos while using a text prompt as input (top two rows), oran additional user-provided image (bottom row). Prompts: (top-left) A fox dressed in a suit dancing in a park, (top-right) The sun breaks
through the clouds from the heights of a skyscraper, (middle-left): A bear is giving a presentation in the classroom, (middle-right): A
360 shot of a sleek yacht sailing gracefully through the crystal-clear waters of the Caribbean, (bottom-left): A ship driving off the harbor,(bottom-right): The dinosaur slowly comes to life. In the bottom two examples, a user-image is provided as an additional conditioning(shown in a blue border) and brought to life by E MU V IDEO. The first one is a historical picture of the RMS Titanic departing from Belfast,
Northern Ireland; and the second is a picture of a Tyrannosaurus rex fossil.

Abstract
1. Introduction

We present E MU V IDEO, a text-to-video generation
model that factorizes the generation into two steps: first
generating an image conditioned on the text, and then generating a video conditioned on the text and the generated
image. We identify critical design decisions–adjusted noise
schedules for diffusion, and multi-stage training–that enable us to directly generate high quality and high resolution
videos, without requiring a deep cascade of models as inprior work. In human evaluations, our generated videos arestrongly preferred in quality compared to all prior work–
81% vs. Google’s Imagen Video, 90% vs. Nvidia’s PYOCO,
and 96% vs. Meta’s Make-A-Video. Our model outperforms
commercial solutions such as RunwayML’s Gen2 and Pika
Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user’s text prompt,
where our generations are preferred 96% over prior work.

Large text-to-image models [17, 21, 28, 38, 55, 62] trainedon web-scale image-text pairs generate diverse and highquality images. While these models can be further adapted
for text-to-video (T2V) generation [7, 30, 38, 41, 68] by using video-text pairs, video generation still lags behind image generation in terms of quality and diversity. Comparedto image generation, video generation is more challenging
as it requires modeling a higher dimensional spatiotemporal output space while still being conditioned only on a text
prompt. Moreover, video-text datasets are typically an orderof magnitude smaller than image-text datasets [17, 38, 68].
The dominant paradigm in video generation uses diffusion models [38, 68] to generate all video frames at once.
† Equal first authors ∗ Equal technical contribution1Quality

In stark contrast, in NLP, long sequence generation is formulated as an autoregressive problem [11]: predicting one
word conditioned on previously predicted words. Thus, theconditioning signal for each subsequent prediction progressively gets stronger. We hypothesize that strengthening theconditioning signal is also important for high quality video
generation, which is inherently a time-series. However, autoregressive decoding with diffusion models is challenging
since generating a single frame from such models itself requires many iterations.
We propose E MU V IDEO to strengthen the conditioning
for diffusion based text-to-video generation with an explicit
intermediate image generation step. Specifically, we factorize text-to-video generation into two subproblems: (1) generating an image from an input text prompt; (2) generatinga video based on the stronger conditioning from the image
and the text. Intuitively, giving the model a starting image
and text makes video generation easier since the model only
needs to predict how the image will evolve in the future.
Since video-text datasets are much smaller than imagetext datasets, we also initialize [7, 68] our factorized text-tovideo model using a pretrained text-to-image (T2I) model
whose weights are kept frozen. We identify critical design decisions–changes to the diffusion noise schedule andmulti-stage training–to directly generate videos at a highresolution of 512px. Unlike direct T2V methods [38, 68],
at inference, our factorized approach explicitly generates
an image, which allows us to easily retain the visual diversity, style, and quality of the text-to-image model (examples
in Figure 1). This allows E MU V IDEO to outperform direct
T2V methods, even when accounting for the same amountof training data, compute, and trainable parameters.
Contributions. We show that text-to-video (T2V) generation quality can be greatly improved by factorizing the generation into first generating an image and using the generated image and text to generate a video. Our multi-stage
training enables us to directly generate videos at a high resolution of 512px, bypassing the need for a deep cascade ofmodels used in prior work [38, 68]. We design a robust
human evaluation scheme–JUICE–where we ask evaluators
to justify their choices when making the selection in the
pairwise comparisons. As shown in Figure 2, E MU V IDEO
significantly surpasses all prior work including commercial
solutions with an average win rate of 91.8% for quality and86.6% for text faithfulness. Beyond T2V, E MU V IDEO canbe used out-of-the-box for image-to-video where the model
generates a video based on a user-supplied image and a text
prompt. In this setting, E MU V IDEO’s generations are preferred 96% of the times over VideoComposer [76].

Emu Video
Emu Video
Emu Video
Emu Video
Emu Video
Emu Video
Emu Video
Emu Video

Text Faithfulness
Pika Labs
Gen2
78.5
100.0 CogVideo
100.0
87.0
95.7 Reuse & Diffuse
81.1
PYOCO
90.5
97.0 Align Your Latents
92.3
Imagen Video
81.8
85.1
96.8 Make-A-Video
87.7

56.4

0

25

50

96.9
98.5

75

100

Emu Video Win Rate Percentage

Figure 2. E MU V IDEO vs. prior work in text-to-video in terms ofvideo quality and text faithfulness win-rates evaluated by majority
score of human evaluator preferences. Since most models fromprior work are not accessible, we use the videos released by eachmethod and their associated text prompt. The released videos arelikely the best generations and we compare without any cherrypicking of our own generations. We also compare to commercial solutions (Gen2 [54] and PikaLabs [47]) and the open source
model CogVideo [41] using the prompt set from [7]. E MU V IDEO
significantly outperforms all prior work across both metrics.

and out-perform prior GAN [8, 43, 66] or auto-regressive
methods [1, 22, 29, 59]. Diffusion models learn a data distribution by gradually denoising a normally distributed variable, often called ‘noise’, to generate the output. Prior work
either denoises in the pixel space with pixel diffusion models [19, 36, 37, 56, 60, 63], or in a lower-dimensional latent
space with latent diffusion models [17, 62]. In this work,
we leverage latent diffusion models for video generation.
Video generation/prediction. Many prior works target theconstrained settings of unconditional generation, or video
prediction [45, 46, 53]. These approaches include training VAEs [4, 5, 18], auto-regressive models [25, 41, 42,
61, 81], masked prediction [27, 32, 86], LSTMs [67, 77],
or GANs [2, 9, 16, 75]. However, these approaches aretrained/evaluated on limited domains. In this work, we target the broad task of open-set T2V generation.
Text-to-Video (T2V) generation. Most prior works tackle
T2V generation by leveraging T2I models. Several works
take a training-free approach [40, 44, 49, 87] for zero-shot
T2V generation by injecting motion information in the T2I
models. Tune-A-Video [79] targets one-shot T2V generation by fine-tuning a T2I model with a single video. Whilethese methods require no or limited training, the quality anddiversity of the generated videos is limited.
Many prior works instead improve T2V generation bylearning a direct mapping from the text condition to the generated videos by introducing temporal parameters to a T2I
model [7, 30, 33, 39, 41, 48, 72, 74, 78, 82, 84]. Make-

2. Related Work
Text-to-Image (T2I) diffusion models. Diffusion models [69] are a state-of-the-art approach for T2I generation,
2

Mask
Mask
Mask
Concat enat e
A t urt le sw immingin t he ocean.

Figure 3. Factorized text-to-video generation involves first generating an image I conditioned on the text p, and then using stronger
conditioning–the generated image and text–to generate a video V. To condition our model F on the image, we zero-pad the image
temporally and concatenate it with a binary mask indicating which frames are zero-padded, and the noised input.

video V consisting of T RGB frames. Recent methods [7,30, 38, 68] directly generate the T video frames at once
using text-only conditioning. Our approach builds on the
hypothesis that stronger conditioning by way of both text
and image can improve video generation (cf . § 3.2).

A-Video [68] utilizes a pre-trained T2I model [60] andthe prior network of [60] to train T2V generation withoutpaired video-text data. Imagen Video [38] builds upon the
Imagen T2I model [63] with a cascade of diffusion models [37, 39]. To address the challenges of modeling thehigh-dimensional spatiotemporal space, several works instead train T2V diffusion models in a lower-dimensional
latent space [3, 7, 24, 30, 31, 34, 80], by adapting latentdiffusion T2I models. Blattmann et al. [7] freeze the parameters of a pre-trained T2I model and train new temporal
layers, whilst Ge et al. [30] build on [7] and design a noise
prior tailored for T2V generation. The limitation of theseapproaches is that learning a direct mapping from text tothe high dimensional video space is challenging. We instead strengthen our conditioning signal by taking a factorization approach. Unlike prior work that enhancing the conditions for T2V generation including leveraging large language models (LLMs) to improve textual description andunderstanding [24, 40, 50], or adding temporal informationas conditions [13, 76, 83, 87], our method does not require
any models to generate the conditions as we use the first
frame of a video as the image condition.
Factorized generation. The most similar works to E MU
V IDEO, in terms of factorization, is CogVideo [41] andMake-A-Video [68]. CogVideo builds upon the pretrained
T2I model [20] for T2V generation using auto-regressive
Transformer. The auto-regressive nature is fundamentallydifferent to our explicit image conditioning in both training and inference stages. Make-A-Video [68] leverages theimage embedding condition learnt from a shared image-text
space. Our factorization leverage the first frame as is, which
is a stronger condition. Moreover, Make-A-Video initializes from a pretrained T2I model but finetunes all the parameters so it cannot retain the visual quality and diversityof the T2I model as we do.

3.1. Preliminaries
Conditional Diffusion Models [36, 69] are a class of generative models that are trained to generate the output using
a conditional input c by iteratively denoising from gaussian noise. At training time, time-step t ∈ [0, N ] dependent
gaussian noise ϵt ∼ N (0, 1) is added to the√original input
signal X to obtain a noisy input Xt = αt X+ 1 − αt ϵt . αtdefines the “noise schedule”, i.e., noise added at timestep t
and N is the total number of diffusion steps. The diffusion model is trained√
to denoise Xt by predicting either ϵt ,X, or vt = αt ϵt − 1 − αt X (called v-prediction [64]).
The signal-to-noise ratio (SNR) at timestep t is given byαt)2 and decreases as t → N . At inference, samples( 1−αt
are generated by starting from pure noise XN ∼ N (0, 1)
and denoising it. Note that at inference time XN has no
signal, i.e., zero SNR which has significant implications forvideo generation as we describe in § 3.2.

3.2. E MU V IDEO
We factorize text-to-video generation into two steps (1) generating the first frame (image) given the text prompt p and(2) generating T frames of a video by leveraging the text
prompt and the image conditioning. We implement bothsteps using a latent diffusion model F, illustrated in Figure 3. We initialize F with a pre-trained text-to-image
model to ensure that it is capable of generating images atinitialization. Thus, we only need to train F to solve thesecond step, i.e., extrapolate a video conditioned on a text
prompt and a starting frame. We train F using video-text
pairs by sampling a starting frame I and asking the model
to predict the T frames using both the text prompt pxw andthe image I conditioning. We denote a video V consisting of T RGB frames of spatial dimensions H ′ , W ′ as a

3. Approach
The goal of text-to-video (T2V) generation is to constructa model f that takes as input a text prompt p to generate a3

4D tensor of shape T × 3 × H ′ × W ′ . Since we use latent diffusion models, we first convert the video V into alatent space X ∈ RT ×C×H×W using a image autoencoder
applied frame-wise, which reduces the spatial dimensions.
The latent space can be converted back to the pixel space
using the autoencoder’s decoder. The T frames of the video
are noised independently to produce the noised input Xt ,which the diffusion model is trained to denoise.
Image conditioning. We condition on the starting frame,
I, by concatenating it with the noise. Our design choice allows the model to use all the information in I unlike otherchoices [68, 76] that lose image information by using a semantic image embedding for conditioning. We represent I
as a single-frame video, i.e., T = 1 and zero-pad it to obtaina T × C × H × W tensor. We use a binary mask m of shape
T × 1 × H × W that is set to 1 at the first temporal position
to indicate the position of the starting frame, and zero otherwise. The mask m, starting frame I, and the noised video
Xt are concatenated channel-wise as the input to the model.
Model. We initialize our latent diffusion model F using thepretrained T2I model [17]. Like prior work [68], we add
new learnable temporal parameters: a 1D temporal convolution after every spatial convolution, and a 1D temporalattention layer after every spatial attention layer. The original spatial convolution and attention layers are applied toeach of the T frames independently and are kept frozen.
The pretrained T2I model is already text conditioned andcombined with the image conditioning described above, F
is conditioned on both text and image.
Zero terminal-SNR noise schedule. We found that the
diffusion noise schedules used in prior work [17, 62] havea train-test discrepancy which prevents high quality video
generation (reported for images in [12, 51]). At training,
the noise schedule leaves some residual signal, i.e., has
non-zero signal-to-noise (SNR) ratio even at the terminal
diffusion timestep N . This prevents the diffusion modelfrom generalizing at test time when we sample from random gaussian noise with no signal about real data. The
residual signal is higher for high resolution video frames,
due to redundant pixels across both space and time. We resolve this issue by scaling the noise schedule and setting thefinal αN = 0 [51], which leads to zero SNR at the terminal timestep N during training too. We find that this design
decision is critical for high resolution video generation.
Interpolation model. We use an interpolation model I,
architecturally the same as F, to convert a low frame-rate
video of T frames into a high frame-rate video of Tp frames.
The interpolation model operates on Tp × C × H × W inputs/outputs. For frame conditioning, the input T frames
are zero-interleaved to produce Tp frames, and a binary
mask m indicating the presence of the T frames are concatenated to the noised input (similar to the image conditioning for F). The model is trained on video clips of Tp

frames of which T frames are fed as input. For efficiency,
we initialize I from F and only train the temporal parameters of the model I for the interpolation task.
Simplicity in implementation. E MU V IDEO can be trainedusing standard video-text datasets, and does not require a
deep cascade of models, e.g., 7 models in [38], for generating high resolution videos. At inference, given a text
prompt, we run F without the temporal layers to generate
an image I. We then use I and the text prompt as input to
F to generate T video frames, directly at high resolution.
We can increase the fps of the video using I. Since the spatial layers are initialized from a pretrained T2I model andkept frozen, our model retains the conceptual and stylistic
diversity learned from large image-text datasets, and uses it
to generate I. This comes at no additional training cost unlike approaches [38] that do joint finetuning on image andvideo data to maintain such style. Many direct T2V approaches [7, 68] also initialize from a pretrained T2I modeland keep the spatial layers frozen. However, they do not
employ our image-based factorization and thus do not retain the quality and diversity in the T2I model.
Robust human evaluation (JUICE). Similar to recent
studies [17, 38, 57, 68], we find that the automatic evaluation metrics [73] do not reflect improvements in quality. We
primarily use human evaluation to measure T2V generation
performance on two orthogonal aspects - (a) video generation quality denoted as Quality (Q) and (b) the alignmentor ‘faithfulness’ of the generated video to the text prompt,
denoted as Faithfulness (F). We found that asking humanevaluators to JUstify their choICE (JUICE) when picking ageneration over the other significantly improves the interannotator agreement (details in Appendix C). The annotators select one or more pre-defined reasons to justify theirchoice. The reasons for picking one generation over theother for Quality are: pixel sharpness, motion smoothness,
recognizable objects/scenes, frame consistency, and amountof motion. For Faithfulness we use two reasons: spatial textalignment, and temporal text alignment.

3.3. Implementation Details
We provide complete implementation details in the supplement Appendix A and highlight salient details next.
Architecture and initialization. We adapt the text-toimage U-Net architecture from [17] for our model and initialize all the spatial parameters with the pretrained model.
The pretrained model produces square 512px images using
an 8 channel 64×64 latent as the autoencoder downsamples
spatially by 8×. The model uses both a frozen T5-XL [15]and a frozen CLIP [58] text encoder to extract features fromthe text prompt. Separate cross-attention layers in the UNet attend to each of the text features. After initialization,
our model contains 2.7B spatial parameters which are kept
frozen, and 1.7B temporal parameters that are learned.
4

The temporal parameters are initialized as identity operations: identity kernels for convolution, and zeroing the final
MLP layer of the temporal attention block. In our preliminary experiments, the identity initialization improved themodel convergence by 2×. For the additional channels in
the model input due to image conditioning, we add C + 1
additional learnable channels (zero-initialized) to the kernelof the first spatial convolution layer. Our model produces
512px square videos of T = 8 or 16 frames and is trainedwith square center-cropped video clips of 1, 2 or 4 seconds
sampled at 8fps or 4fps. We train all our models with abatch size of 512 and describe the details next.
Efficient multi-stage multi-resolution training. To reducethe computational complexity, we train in two stages - (1)for majority of the training iterations (70K) we train fora simpler task: 256px 8fps 1s videos, which reduces periteration time by 3.5× due to the reduction in spatial resolution; (2) we then train the model at the desired 512px
resolution on 4fps 2s videos for 15K iterations. The change
in spatial resolution does not affect the 1D temporal layers.
Although the frozen spatial layers were pretrained at 512px,
changing the spatial resolution at inference to 256px led to
no loss in generation quality. We use the noise schedulefrom [62] for 256px training, and with zero terminal-SNR
for 512px training using the v-prediction objective [64] with
N = 1000 steps for the diffusion training. We sample fromour models using 250 steps of DDIM [70]. Optionally, toincrease duration, we further train the model on 16 frames
from a 4s video clip for 25K iterations.
Finetuning for higher quality. Similar to the observation in image generation [17], we find that the motion of
the generated videos can be improved by finetuning themodel on a small subset of high motion and high quality
videos. We automatically identify a small finetuning subset of 1.6K videos from our training set which have highmotion (computed using motion signals stored in H.264 encoded videos). We follow standard practice [62] and also
apply filtering based on aesthetic scores [62] and CLIP [58]similarity between the video’s text and first frame.
Interpolation model. We initialize the interpolation modelfrom the video model F. Our interpolation model takes
8 frames as input and outputs Tp = 37 frames at 16fps.
During training, we use noise augmentation [37] where we
add noise to the frame conditioning by randomly samplingtimesteps t ∈ {0, ...250}. At inference time, we noise augment the samples from F with t = 100.

noted otherwise, we train the model on the full set, and do
not use the 1.6K high motion quality finetuning subset described in § 3.3.
Text prompt sets for human evaluation. We use the text
prompt sets from prior work (cf . Appendix Table 10) to generate videos. The prompts cover a wide variety of categories
that can test our model’s ability to generate natural and fantastical videos, and compose different visual concepts. We
use our proposed JUICE evaluation scheme ( Sec. 3) for reliable human evaluation and use the majority vote from 5
evaluators for each comparison.

4.1. Ablating design decisions
We study the effects of our design decisions using the 8frame generation setting and report human evaluation results in Table 1 using pairwise comparisons on the 307
prompt set of [68].
Factorized vs. Direct generation. We compare our factorized generation to a direct T2V generation model that generates videos from text condition only. We ensure that the
pretrained T2I model, training data, number of training iterations, and trainable parameters are held constant for thiscomparison. As shown in Table 1a, the factorized generation model’s results are strongly preferred both in Quality and Faithfulness.The strong preference in Quality is because the direct generation model does not retain the styleand quality of the text-to-image model despite frozen spatial parameters, while also being less temporally consistent(examples in Figure 4).
Zero terminal-SNR noise schedule. We compare using
zero terminal-SNR for the high resolution 512px training
against a model that is trained with the standard noise
schedule. Table 1b shows that generations using zeroterminal-SNR are strongly preferred. This suggests that thezero terminal-SNR noise schedule’s effect of correcting thetrain-test discrepancy as described in § 3.2 is critical forhigh resolution video generation. We also found that zeroterminal-SNR has a stronger benefit for our factorized generation compared to a direct T2V model possibly. Similar toimages [51], in the direct T2V case, this decision primarily
affects the color composition. For our factorized approach,this design choice was critical for object consistency andhigh quality as our qualitative results in Figure 4 show.
Multi-stage multi-resolution training. We spend mosttraining budget (4×) on the 256px 8fps stage compared tothe 3.5× slower (due to increased resolution) 512px 4fps
stage. We compare to a baseline that trains only the 512px
stage with the same training budget. Table 1c shows that
our multi-stage training yields significantly better results.
High quality finetuning. We study the effect of finetuning
our model on automatically identified high quality videos
in Table 1d. We found that this finetuning improves on both
metrics. In particular, finetuning improves the model’s abil-

4. Experiments
Dataset. We train E MU V IDEO on a dataset of 34M licensed video-text pairs. Our videos range from 5-60 seconds and cover a variety of natural world concepts. These
videos were not curated for any particular task and were notfiltered for any text-frame similarity or aesthetics. Unless
5

Method

Q

F

MethodQ

F

MethodQ

F

MethodQ

F

MethodQ

F

Factorized

70.5

63.3

Zero SNR

96.8

88.3

Multi-stage

81.8

84.1

HQ finetuned

65.1

79.6

Frozen spatial

55.0

58.1

(a)

(b)

(c)

(d)

(e)

Table 1. Key design decisions in E MU V IDEO. Each table shows the preference, in terms of the Quality (Q) and Faithfulness (F), on
adopting a design decision vs. a model that does not have it. Our results show clear preference to a) factorized generation that uses both
image and text conditioning (against a direct video generation baseline that is only text conditioned), b) adopting zero terminal-SNR noise
schedule for directly generating high resolution 512px videos, c) adopting the multi-stage training setup compared to training directly at
the high resolution, d) incorporating the high quality (HQ) finetuning, and e) freezing the spatial parameters. See § 4.1 for details.
Unicorns running along a beach.

w/ HQ Finetune

w/o Zero SNR

Factorized Gen.

Direct Gen.

Dolphins jumping in the ocean.

Figure 4. Design choices in E MU V IDEO. Top row: Direct text-to-video generation produces videos that have low visual quality and areinconsistent. Second row: We use a factorized text-to-video approach that produces high quality videos and improves consistency. Third
row: Not using a zero terminal-SNR noise schedule at 512px generation leads to significant inconsistencies in the generations. Bottom
row: Finetuning our model (second row) with HQ data increases the motion in the generated videos.

interpolate 16-frame videos with I.

ity to respect the motion specified in the text prompt as reflected by the strong gain in Faithfulness.
Parameter freezing. We test if freezing the spatial parameters of our model affects performance. We compare against
a model where all parameters are finetuned during the second 512px training stage. For fair comparison, we use thesame conditioning images I across our model and this baseline. Table 1e suggests that freezing the spatial parameters
produces better videos, while reducing training cost.

Human evaluation of text-to-video generation. Sincemany recent prior methods in text-to-video generation areclosed source [7, 30, 31, 38], we use the publicly released
examples from each of these methods. Note that the released videos per method are likely to be the ‘best’ representative samples from each method and may not capture
their failure modes. For Make-A-Video, we obtained noncherry-picked generations through personal communication
with the authors. For CogVideo [41], we perform T2V generation on the prompt set from [7] using the open source
models. We also benchmark against commercially engineered black-box text-to-video solutions, Gen2 [54] andPikaLabs [47], for which we obtain generations throughtheir respective websites using the prompts from [7]. We do
not cherry-pick or contrastively rerank [59, 85] our videos,

4.2. Comparison to prior work
We evaluate E MU V IDEO against prior work and train F to
produce 16 frame 4 second long videos and use the best design decisions from § 4.1, including high quality finetuning.
We use the interpolation model I on our generations to get
16fps videos. Please see Appendix A for details on how we
6Flying through an intense battle between pirate ships in a stormy ocean.

AYL [7]

IV [38]%age selected

E MU V IDEO vs.Make-A-Video [68]

E MU V IDEO vs.Imagen Video [38]

60
40
20
0
MotionObjectPixel
Smoothness Consistency Sharpness

VisualConcept

Amount of
MotionFigure 5. Qualitative comparison. E MU V IDEO produces higher
quality generations compared to Imagen Video [38] and AlignYour Latents [7] in terms of style and consistency.

and generate them using a deterministic random noise seedthat is not optimized in any way.
Since each method generates videos at different resolutions, aspect-ratios, and frame-rates, we reduce annotator bias in human evaluations by postprocessing the videos
for each comparison in Figure 2 so that they match inthese aspects. Full details on this postprocessing and thetext prompts used are in Appendix D. As shown in Figure 2, E MU V IDEO’s generations significantly outperform
all prior work, including commercial solutions, both interms of Quality (by an average of 91.8%) and Faithfulness (by an average of 86.6%). We show some qualitative
comparisons in Figure 5 and some additional generationsin Figure 1. E MU V IDEO generates videos with significantly higher quality, and overall faithfulness to both theobjects and motion specified in the text. Since our factorized approach explicitly generates an image, we retain thevisual diversity and styles of the T2I model, leading to far
better videos on fantastical and stylized prompts. Additionally, E MU V IDEO generates videos with far greater temporal consistency than prior work. We hypothesize that since
we use stronger conditioning of image and text, our model istrained with a relatively easier task of predicting how an image evolves into the future, and thus is better able to model
the temporal nature of videos. Please see Appendix E for
more qualitative comparisons. We include human evaluations where videos are not postprocessed in the supplement Appendix D, where again E MU V IDEO’s generations
significantly outperform all prior work. The closest modelin performance compared to ours is Imagen Video whenmeasured on Faithfulness, where we outperform Imagen
Video by 56%. Imagen Video’s released prompts ask forgenerating text characters, a known failure mode [17, 62] oflatent diffusion models used in E MU V IDEO.
We inspect the reasons that human evaluators prefer
E MU V IDEO generations over the two strongest competitors

MethodAutomated
FVD ↓ IS ↑MagicVideo [88]Align Your Latents [7]Make-A-Video [68]
PYOCO [30]655.0 550.6 33.5
367.2 33.0
355.2 47.8

E MU V IDEO

606.2 42.7

Human Evaluation
vs. Make-A-Video% Win Rate

E MU V IDEO

Figure 6. Percentage of each reason selected for samples
where E MU V IDEO wins against Make-A-Video [68] or Imagen Video [38] on Quality. Human evaluators pick E MU V IDEO
generations primarily because of their pixel sharpness and motion
smoothness, with an overall preference of 96.8% and 81.8% toeach baseline, respectively.

75
50
25
0
Q

F

Table 2. Automated metrics are flawed for zero-shot textto-video evaluation on UCF101. (Left) We present automatedmetrics and observe that E MU V IDEO does not outperform priorwork. (Right) We use human evaluations to compare E MU V IDEO
and Make-A-Video where E MU V IDEO significantly outperforms
Make-A-Video both in Quality (90.1%) and Faithfulness (80.5%).

in Figure 6. A more detailed inspection is provided in Appendix C. E MU V IDEO generations are preferred due totheir better pixel sharpness and motion smoothness. While
being state-of-the-art, E MU V IDEO is also simpler and has
a two model cascade with a total of 6.0B parameters (2.7B
frozen parameters for spatial layers, and 1.7B learnabletemporal parameters each for F and I), which is much
simpler than methods like Imagen Video (7 model cascade,
11.6B parameters), Make-A-Video (5 model cascade, 9.6B
parameters) trained using similar scale of data.
Automated metrics. In Table 2, we compare against priorwork using the zero-shot T2V generation setting from [68]on the UCF101 dataset [71]. E MU V IDEO achieves acomptetitive IS score [65] and a higher FVD [73]. Priorworks suggest that the automated metrics are flawed anddo not capture human preferences [6, 14, 17, 38, 57, 68].
We believe FVD penalizes our high quality generations thatare different from the UCF101 videos, while IS is biased
towards its training data [6, 14]. To confirm this, we use
human evaluations to compare our generations to Make-AVideo. We use a subset of 303 generated videos (3 random
samples per UCF101 class) and find that our generations arestrongly prefered ( Table 2 Right). Qualitative comparisons
7Method
E MU V IDEO vs. VideoComposer [76]
E MU V IDEO vs. PikaLabs I2V [47]
E MU V IDEO vs. Gen2 I2V [54]E MU V IDEO vs. VideoComposer [76]#Prompts
65 [7]
307 [68]

Q
96.9
84.6
70.8
97.4

Original: Low angle of pouring beer into a glass cup.

F
96.9
84.6
76.9
91.2

Table 3. Human evaluation of E MU V IDEO vs. prior work in
text-conditioned image animation. We compare E MU V IDEO
against three methods across two prompt sets using the generations
from [57] as the starting images. E MU V IDEO’s animated videos
are strongly prefered on both the prompt sets over all baselines.

Future prompt 1: The beer starts to pour over and spill on the table.

Future prompt 2: The beer in the glass catches fire.

can be found in Appendix E.
Animating images. A benefit of our factorized generation is that the same model can be used out-of-the-box to‘animate’ user-provided images by supplying them as theconditioning image I. We compare E MU V IDEO’s image
animation with three methods, concurrent work [76] andcommercial image-to-video (I2V) solutions [47, 54], on theprompts from [68] and [7]. All the methods are shown
the same image generated using a different text-to-image
model [57] and expected to generate a video according to
the text prompt. We use the API for [57] in our comparisons since the official training data and model is not available. We report human evaluations in Table 3 and automated metrics in the supplement (cf . Appendix Table 9).
Human evaluators strongly prefer E MU V IDEO’s generations across all the baselines. These results demonstrate
the superior image animation capabilities of E MU V IDEO
compared to methods specifically designed for the imageto-video task.

Figure 7. Extending to longer videos. We test a variant of
E MU V IDEO that is conditioned on all the frames from the originalvideo, and generates new videos conditioned on a future prompt.
For two different future prompts, our model generates plausible
extended videos that respect the original video and the future text.

the original video as well as the future text prompt.

5. Limitations and Future Work
We presented E MU V IDEO, a factorized approach to text-tovideo generation that leverages strong image and text conditioning. E MU V IDEO significantly outperforms all priorwork including commercial solutions. There is a differencein the image conditioning used for our model at train andinference: at training, we use a video frame sampled fromreal videos, while at inference we use a generated image(using the spatial parameters of the model). In practice, thisdifference does not affect the quality of the generated video
for most scenarios. However, in cases where the generated
image used for conditioning at inference is not representative of the prompt, our model has no way to recover fromthis error. We believe that improving the models ability torecover from such errors is an important direction for future
work. Strengthening the conditioning for video models using pure autoregressive decoding with diffusion models is
not currently computationally attractive. However, further
research may provide benefits for longer video generation.
Ethical considerations. We propose advancements in generative methods specifically to improve the generation ofhigh dimensional video outputs. Generative methods canbe applied to a large variety of different usecases which arebeyond the scope of this work. A careful study of the data,
model, its intended applications, safety, risk, bias, and societal impact is necessary before any real world application.

4.3. Analysis
Nearest neighbor baseline. We expect good and useful generative models to outperform a nearest neighbor retrieval baseline and create videos not in the training set.
We construct a strong nearest neighbor baseline that retrieves videos from the full training set (34M videos) by using the text prompt’s CLIP feature similarity to the training
prompts. When using the evaluation prompts from [68], human evaluators prefer E MU V IDEO’s generations 81.1% in
Faithfulness over real videos confirming that E MU V IDEO
outperforms the strong retrieval baseline. We manually inspected and confirmed that E MU V IDEO outperforms thebaseline for prompts not in the training set.
Extending video length with longer text. Recall that ourmodel conditions on the text prompt and a starting frame togenerate a video. With a small architectural modification,
we can also condition the model on T frames and extend thevideo. Thus, we train a variant of E MU V IDEO to generate
the future 16 frames conditioned on the ‘past’ 16 frames.
While extending the video, we use a future text prompt different from the one used for the original video and visualize
results in Figure 7. We find that the extended videos respect8

Acknowledgments. We are grateful for the support of multiple collaborators at Meta who helped us in this work.
Baixue Zheng, Baishan Guo, Jeremy Teboul, Milan Zhou,
Shenghao Lin, Kunal Pradhan, Jort Gemmeke, Jacob Xu,
Dingkang Wang, Samyak Datta, Guan Pang, Symon Perriman, Vivek Pai, Shubho Sengupta for their help withthe data and infra. We would like to thank Uriel Singer,
Adam Polyak, Shelly Sheynin, Yaniv Taigman, Licheng Yu,
Luxin Zhang, Yinan Zhao, David Yan, Emily Luo, Xiaoliang Dai, Zijian He, Peizhao Zhang, Peter Vajda, Roshan
Sumbaly, Armen Aghajanyan, Michael Rabbat, and Michal
Drozdzal for helpful discussions. We are also grateful to thehelp from Lauren Cohen, Mo Metanat, Lydia Baillergeau,
Amanda Felix, Ana Paula Kirschner Mofarrej, Kelly Freed,Somya Jain. We thank Ahmad Al-Dahle and Manohar
Paluri for their support.

[11] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al. Language models are few-shot learners. preprint
arXiv:2005.14165, 2020. 2[12] Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint arXiv:2301.10972, 2023. 4[13] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li,
Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video:
Controllable text-to-video generation with diffusion models.
arXiv preprint arXiv:2305.13840, 2023. 3[14] Min Jin Chong and David Forsyth. Effectively unbiased fid
and inception score and where to find them. In CVPR, pages
6070–6079, 2020. 7[15] Hyung Won Chung, Le Hou, Shayne Longpre, Barret
Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,
Mostafa Dehghani, Siddhartha Brahma, et al. Scaling
instruction-finetuned language models.
arXiv preprint
arXiv:2210.11416, 2022. 4[16] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2019. 2[17] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang
Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in ahaystack. arXiv preprint arXiv:2309.15807, 2023. 1, 2, 4, 5,
7[18] Emily Denton and Rob Fergus. Stochastic video generation
with a learned prior. In Proceedings of the 35th International
Conference on Machine Learning, pages 1174–1183. PMLR,
2018. 2[19] Prafulla Dhariwal and Alex Nichol. Diffusion models beatgans on image synthesis, 2021. 2[20] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.
Cogview2: Faster and better text-to-image generation via hierarchical transformers. NeurIPS, 2022. 3[21] J. Donahue, P. Krahenbühl, and T. Darrell. Adversarial feature learning. In ICLR, 2016. 1[22] Patrick Esser, Robin Rombach, and Björn Ommer. Tamingtransformers for high-resolution image synthesis. In CVPR,
2021. 2[23] Patrick Esser, Johnathan Chiu, Parmida Atighehchian,Jonathan Granskog, and Anastasis Germanidis. Structure
and content-guided video synthesis with diffusion models,
2023. 15[24] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and TatSeng Chua. Empowering dynamics-aware text-to-video diffusion with large language models, 2023. 3[25] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of the 30th International Conference
on Neural Information Processing Systems, page 64–72, Red
Hook, NY, USA, 2016. Curran Associates Inc. 2[26] Joseph L Fleiss and Jacob Cohen. The equivalence of
weighted kappa and the intraclass correlation coefficient asmeasures of reliability. Educational and psychological measurement, 33(3):613–619, 1973. 16

References[1] Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace
Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro
Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, andLuke Zettlemoyer. Cm3: A causal masked multimodal
model of the internet. ArXiv, abs/2201.07520, 2022. 2[2] Nuha Aldausari, Arcot Sowmya, Nadine Marcus, andGelareh Mohammadi. Video generative adversarial networks: A review. ACM Comput. Surv., 55(2), 2022. 2[3] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin
Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation, 2023. 3[4] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan,
Roy H. Campbell, and Sergey Levine. Stochastic variational
video prediction. In ICLR, 2018. 2[5] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj
Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan.
Fitvid: Overfitting in pixel-level video prediction. arXiv
preprint arXiv:2106.13195, 2020. 2[6] Shane Barratt and Rishi Sharma. A note on the inception
score. arXiv preprint arXiv:1801.01973, 2018. 7[7] A. Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn,
Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Alignyour latents: High-resolution video synthesis with latent diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575,
2023. 1, 2, 3, 4, 6, 7, 8, 14, 15, 16, 20[8] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large
scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. 2[9] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun
Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A
Efros, and Tero Karras. Generating long videos of dynamic
scenes. In NeurIPS, 2022. 2[10] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions.
In CVPR, 2023. 15

9[42] Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan,
Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray
Kavukcuoglu. Video pixel networks. In Proceedings of the34th International Conference on Machine Learning, pages
1771–1779. PMLR, 2017. 2[43] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park,Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling
up gans for text-to-image synthesis. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2[44] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant
Navasardyan, and Humphrey Shi. Text2video-zero: Text-toimage diffusion models are zero-shot video generators. arXiv
preprint arXiv:2303.13439, 2023. 2[45] Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational
Temporal Abstraction. Curran Associates Inc., Red Hook,
NY, USA, 2019. 2[46] Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan,
Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk
Kingma. Videoflow: A conditional flow-based model forstochastic video generation. In ICLR, 2020. 2[47] Pika Labs. Pika labs. https://www.pika.art/. 2, 6, 8
[48] Ivan Laptev and Tony Lindeberg. Space-time interest points.
In ICCV, 2003. 2[49] Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun
Kwak. Aadiff: Audio-aligned video synthesis with text-toimage diffusion, 2023. 2[50] Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, andBoyi Li. Llm-grounded video diffusion models. arXiv
preprint arXiv:2309.17444, 2023. 3[51] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang.
Common diffusion noise schedules and sample steps areflawed. arXiv preprint arXiv:2305.08891, 2023. 4, 5, 13[52] Ilya Loshchilov and Frank Hutter. Decoupled weight decay
regularization. arXiv preprint arXiv:1711.05101, 2017. 13[53] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep
multi-scale video prediction beyond mean square error,
2016. 2[54] Runway ML. Gen2. https://research.runwayml.com/
gen2. 2, 6, 8, 15[55] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav
Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, andMark Chen. Glide: Towards photorealistic image generationand editing with text-guided diffusion models. arXiv preprint
arXiv:2112.10741, 2021. 1[56] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav
Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, andMark Chen. Glide: Towards photorealistic image generationand editing with text-guided diffusion models, 2022. 2[57] Dustin Podell, Zion English, Kyle Lacey, Andreas
Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, andRobin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint
arXiv:2307.01952, 2023. 4, 7, 8, 15[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,[27] Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng-Yang Fu, JongChyi Su, William Yang Wang, and Sean Bell. Tell me what
happened: Unifying text-guided video completion via multimodal masked video generation. In CVPR, pages 10681–
10692, 2023. 2[28] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin,
Devi Parikh, and Yaniv Taigman. Make-a-scene: Scenebased text-to-image generation with human priors. arXiv
preprint arXiv:2203.13131, 2022. 1[29] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin,
Devi Parikh, and Yaniv Taigman. Make-a-scene: Scenebased text-to-image generation with human priors. In European Conference on Computer Vision, 2022. 2[30] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew
Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, MingYu Liu, and Yogesh Balaji. Preserve your own correlation:
A noise prior for video diffusion models, 2023. 1, 2, 3, 6, 7,
16, 20[31] Jiaxi Gu, Shicong Wang, Haoyu Zhao, Tianyi Lu, Xing
Zhang, Zuxuan Wu, Songcen Xu, Wei Zhang, Yu-Gang
Jiang, and Hang Xu. Reuse and diffuse: Iterative denoising
for text-to-video generation, 2023. 3, 6, 16, 20[32] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu,
Roberto Martı́n-Martı́n, and Li Fei-Fei. Maskvit: Masked
visual pre-training for video prediction. In ICLR, 2023. 2[33] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modelingof long videos. In NeurIPS, pages 27953–27965. Curran Associates, Inc., 2022. 2[34] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, andQifeng Chen. Latent video diffusion models for high-fidelity
long video generation, 2023. 3[35] Jonathan Ho and Tim Salimans. Classifier-free diffusion
guidance. arXiv preprint arXiv:2207.12598, 2022. 13[36] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239,
2020. 2, 3[37] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet,
Mohammad Norouzi, and Tim Salimans. Cascaded diffusion
models for high fidelity image generation. arXiv preprint
arXiv:2106.15282, 2021. 2, 3, 5[38] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang,
Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben
Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation withdiffusion models, 2022. 1, 2, 3, 4, 6, 7, 16, 20[39] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William
Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NeurIPS, pages 8633–8646. Curran Associates, Inc., 2022. 2, 3[40] Susung Hong, Junyoung Seo, Sunghwan Hong, Heeseong
Shin, and Seungryong Kim. Large language models areframe-level directors for zero-shot text-to-video generation,
2023. 2, 3[41] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and
Jie Tang. Cogvideo: Large-scale pretraining for text-to-video
generation via transformers, 2022. 1, 2, 3, 6

10

[73] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach,Raphaël Marinier, Marcin Michalski, and Sylvain Gelly.
Fvd: A new metric for video generation. 2019. 4, 7[74] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi
Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan.
Phenaki: Variable length video generation from open domain
textual descriptions. In International Conference on Learning Representations, 2023. 2[75] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba.
Generating videos with scene dynamics. In Advances in Neural Information Processing Systems 29: Annual Conference
on Neural Information Processing Systems 2016, December
5-10, 2016, Barcelona, Spain, pages 613–621, 2016. 2[76] Xiang* Wang, Hangjie* Yuan, Shiwei* Zhang, Dayou*
Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli
Zhao, and Jingren Zhou. Videocomposer: Compositional
video synthesis with motion controllability. arXiv preprint
arXiv:2306.02018, 2023. 2, 3, 4, 8, 15[77] Nevan Wichers, Ruben Villegas, D. Erhan, and Honglak Lee.
Hierarchical long-term video prediction without supervision.
In International Conference on Machine Learning, 2018. 2[78] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji,
Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. ArXiv,
abs/2104.14806, 2021. 2[79] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian
Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu
Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuningof image diffusion models for text-to-video generation. In
ICCV, 2023. 2[80] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang
Jiang. Simda: Simple diffusion adapter for efficient video
generation, 2023. 3[81] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind
Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021. 2[82] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv
preprint arXiv:2203.09481, 2022. 2[83] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang
Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained
control in video generation by integrating text, image, andtrajectory. arXiv preprint arXiv:2308.08089, 2023. 3[84] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang,
Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li,
Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming, Lijuan
Wang, Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl:
Diffusion over diffusion for extremely long video generation,
2023. 2[85] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv
preprint arXiv:2206.10789, 2022. 6[86] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han
Zhang, Huiwen Chang, Alex Hauptmann, Ming-Hsuan

Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. 2021. 4, 5[59] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray,
Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.
Zero-shot text-to-image generation, 2021. 2, 6[60] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu,
and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125,
2022. 2, 3[61] Marc’Aurelio Ranzato, Arthur Szlam, Joan Bruna, MichaëlMathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural
videos. ArXiv, abs/1412.6604, 2014. 2[62] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 1, 2, 4, 5, 7, 13[63] Chitwan Saharia, William Chan, Saurabh Saxena, Lala
Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed
Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi,Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J
Fleet, and Mohammad Norouzi. Photorealistic text-to-image
diffusion models with deep language understanding, 2022.
2, 3[64] Tim Salimans and Jonathan Ho. Progressive distillation forfast sampling of diffusion models, 2022. 3, 5[65] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki
Cheung, Alec Radford, and Xi Chen. Improved techniques
for training gans. Advances in neural information processing
systems, 29, 2016. 7[66] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, andTimo Aila. StyleGAN-T: Unlocking the power of GANs forfast large-scale text-to-image synthesis. 2023. 2[67] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung,Wai-kin Wong, and Wang-chun WOO. Convolutional lstm
network: A machine learning approach for precipitation
nowcasting. In NeurIPS. Curran Associates, Inc., 2015. 2[68] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An,
Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual,
Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.
Make-a-video: Text-to-video generation without text-video
data. In ICLR, 2023. 1, 2, 3, 4, 5, 7, 8, 14, 15, 16, 20[69] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan,and Surya Ganguli. Deep unsupervised learning using
nonequilibrium thermodynamics. In Proceedings of the
32nd International Conference on Machine Learning, pages
2256–2265, Lille, France, 2015. PMLR. 2, 3[70] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 5,
13[71] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
UCF101: A dataset of 101 human action classes from videos
in the wild. CRCV-TR-12-01, 2012. 7[72] Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, andMohit Bansal. Any-to-any generation via composable diffusion, 2023. 2

11

Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked
generative video transformer. In CVPR, 2023. 2[87] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng
Zhang, Wangmeng Zuo, and Qi Tian.
Controlvideo:
Training-free controllable text-to-video generation, 2023. 2,
3[88] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv,
Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video
generation with latent diffusion models, 2023. 7

12

Setting

Table Of Contents
A. Implementation Details

13

B. Additional experiments

14

C. Human evaluations
C.1. Robust Human Evaluations with JUICE . . . . .

16
16

D. Comparisons to Prior Work
D.1. Datasets used for Prior Work Comparisons . . .
D.2. Sampling from Commercial Models . . . . . . .
D.3. Postprocessing Videos for Comparison . . . . .
D.4. Prior Work at Original Dimensions . . . . . . .

18
18
19
19
20

E. Qualitative Results
E.1. Further E MU V IDEO qualitative Results . . . . .
E.2. Qualitative Comparisons to Prior Work . . . . .

20
20
20

Table 5. VAE architecture details. We use an image based VAE
and apply it to videos frame-by-frame. Our VAE encoder downsamples videos spatially by 8 × 8 and produces 8 channel latents.

and wtxt of 7.5 for both video generation and frame interpolation. We share more details about handling multiple
conditionings for Classifier Free Guidance next.
Setting

Diffusion settings:
Loss
Timesteps
Noise Schedule
Beta start
Beta end
Var type
Prediction mode
0-term-SNR rescale
Optimizer
Optimizer Momentum
Learning rate:
Schedule
Warmup Schedule
Peak
Warmup Steps
Weight decay
Dataset size
Batch size
Transforms:
Clip Sampler
Frame Sampler
Resizeinterpolation
size
Center Crop
Normalize Range

In this section we include details on the architecturesand hyper-parameters used for training the models in themain paper, and on the use of multiple conditionings forclassifier-free guidance. For both our text-to-video (F) andinterpolation (I) models we train with the same U-Net architecture. We share the exact model configuration for our
U-Net in Table 4, and the configuration for our 8-channel
autoencoder in Table 5.

input shape
output shapemodel channels
attention resolutions
num res blocks
channel multipliers
use spatial attention
use temporal attention
transformer config:
d headnum layers
context dim layer 1
context dim layer 2

AutoencoderKL [62]
8
3
3
128
[1, 2, 4, 4]2

typez channels
in channels
out channels
base channels
channel multipliers
num res blocks

A. Implementation Details

Setting

Value

Value
[17, T , 64, 64][8, T , 64, 64]384[4, 2, 1][3, 4, 4, 4][1, 2, 4, 4]
True
True
64
2
768
2048

Table 4. U-Net architecture details. Our U-Net contains 4.3B
total parameters, out of which 2.7B are initialized from our pretrained text-to-image model and kept frozen, resulting in 1.7B
trainable parameters. T is the total frames produced by the model.

256px
F

Training stage
512px HQ FT
F
F

FI
I

Mean Squared Error
1000
quad
quad∗
∗
8.5 × 10−4
8.5 × 10−4−2−2∗
1.2 × 10
1.2 × 10
Fixed small
eps-predv-predFalse
True [51]AdamW [52]β1 = 0.9, β2 = 0.999

1e-4
1K
0.0
34M
512

Constant
Linear2.5e-5
10K
1e-4
1.6K
64

1.5e-4
1.5K
0.0
34M
384

Uniform
Uniform

256px
256px

Box + Bicubic
512px
512px
[-1, 1]

Table 6. Training hyperparameters for various stages in our
pipeline: 256px training, 512px training, High Quality finetuning (HQ FT), and frame interpolation (FI). ∗ : noise schedules arechanged afterwards with zero terminal-SNR rescaling [51].

Table 6 shares the training hyperparameters we used forvarious stages of our training – 256px training, 512px training, High Quality finetuning, and frame interpolation. For
inference, we use the DDIM sampler [70] with 250 diffusion steps. We use Classifier Free Guidance (CFG) [35]with wimg of 7.5 for image generation, and wimg of 2.0

Multiple Conditionings for CFG. For video generation,
our model receives two conditioning signals (image I, text
prompt p), which we use in conjunction for Classifier Free
Guidance [35]. Eq 1 lists the combined CFG equation we
13Modelw/o HQ finetuning
w/o HQ finetuning
w/o HQ finetuning
w/o HQ finetuning
w/o HQ finetuning
w/o HQ finetuning
HQ finetuned
HQ finetuned
HQ finetuned
HQ finetuned
HQ finetuned
HQ finetuneduse.
X̃ = X + wi (X(I) − X(∅)) + wp (X(I, p) − X(I)) (1)
Eq 1 was chosen such that: (1) if the CFG scales for image wi and text prompt wp are both equal to 1, the resulting
vector X̃ should be equal to the prediction X(I, p) conditioned on the image and text, without Classifier Free Guidance. (2) if the CFG scales for image wi and text wp areboth equal to 0, the resulting vector X̃ should be equal tothe un-conditioned prediction X(∅).
In Eq 1 there is an ordering on the conditionings. We
also considered alternate orderings in which we start withthe text conditioning first instead of the image conditioning:

wp
2.0
8.0
16.0
8.0
8.0
8.0
2.0
8.0
16.0
8.0
8.0
8.0

wi
1.0
1.0
1.0
1.0
2.0
3.0
2.0
2.0
2.0
1.0
2.0
3.0

Motion Score
1.87
2.87
3.86
2.87
0.61
0.25
11.1
12.7
13.5
14.9
12.7
11.3

Table 7. We measure the amount of motion in the generated videos
using an automated motion score where a higher value reflects
more motion. We use the prompts from [68]. The ratio of text
CFG scale wp to image CFG scale wi influences the amount of
motion in the video. We also observe that, w/o HQ fine-tuning,
motion is much less and that the relative effect of CFG scales iseven more pronounced.

X̃ = X + wp (X(p) − X(∅)) + wi (X(I, p) − X(p))(2)
Eq 2 did not lead to improvement over Eq 1, but required
significantly different values for wi and wp to work equally
well. We also considered formulas without ordering between the two conditionings, for instance:X̃ = X + wi (X(I) − x(∅)) + wp (X(p) − x(∅))
andX̃ = X(I, p) + wi′ (X(I, p) − X(p)) + wp′ (X(I, p) − x(I))
where wi′ = (wi − 1) and wp′ = (wp − 1)
Similar to Eq 2, those formulas did not improve over Eq 1,
and in addition miss the useful properties listed above.
Selecting CFG scales. Eq 1 requires to find the guidance
factor wi for image and wp for text. We found that thesefactors influence the motion in the generated videos. To
quantify this, we measure a ‘motion score’ on the generated
videos by computing the mean energy of the motion vectors
in the resulting H.264 encoding. We found that the motion
score was a good proxy for the amount of motion, but did
not provide signal into consistency of the motion. Higher
motion as computed through motion vectors does not necessarily translate to interesting movement, as it could be undesirable jitter, or reflect poor object consistency. Table 7
shows how the CFG scales directly influence the amount of
motion in the generated videos.
After narrowing down a few CFG value combinations
by looking at the resulting motion score, we identified thebest values by visual inspection and human studies. Qualitatively, we found that the (1) higher wi for a fixed wp , themore the model stays close to the initial image and favors
camera motion; and (2) the higher wp for a fixed wi , themore the model favors movement at the expense of object
consistency.
Frame Interpolation Model. Here, we include extra details on the frame interpolation model, I. First we explain
our masked zero-interleaving strategy. Second we explain

how we interpolate 16-frame 4fps videos from F. § 3.3
in the main paper details how I is trained to take 8 zerointerleaved frames (generated from F at 4fps) as conditioning input and generate 37 frames at 16fps. One option for
training an interpolation model that increases the fps by 4fold is to generate 3 new frames between every pair of input
frames (as in [7]). However, the downside to this approachis that the resulting interpolated video has a slightly shorter
duration than the input video (since every input frame has 3
new generated frames after it, except the last input frame).
We instead take the approach of using I to increase the duration of the input video, and we design a zero-interleaving
scheme accordingly. Our interpolation model is trained togenerate 3 new frames between every pair of frames, andalso 4 new frames either side of the input video. As a result,during training I takes as conditioning input a 2s video, andgenerates a 2.3s video.
For interpolating 16-frame input videos from F (as described in § 4.2 in the main paper), we simply split thevideos into two 8-frame videos and run interpolation on
both independendly. In order to construct our final interpolated video, we discard the overlapping frames (the last
5 frames of the first interpolated video, and the first 4 of thesecond), and concatenate the two videos frame-wise. The
resulting interpolated video is 65 frames long at 16fps (4.06
seconds in duration – we refer to these videos as 4 seconds
long in the main paper for brevity).

B. Additional experiments
We detail additional experiments, viz. (i) an investigation
into the effect of the initial image on our video generations,(ii) a quantitative comparison to prior work in image anima14

Method
Gen2 vs. Gen2 I2V
E MU V IDEO vs. Gen2 I2V
E MU V IDEO vs. Gen2

#Prompts
65 [7]

Q
41.5
72.3
78.5

MethodDataset FC (↑) IC (↑) TC(↑)
VideoComposer [76]96.8 86.4 33.3
PikaLabs I2V
99.9 95.0 34.6
AYL [7]
Gen2 I2V
99.9 96.8 34.3
99.3 94.2 34.2
E MU V IDEO
VideoComposer [76]95.2 82.6 31.3
MAV [68]
E MU V IDEO
98.9 91.3 32.1

F
44.6
78.4
87.7

Table 8. Image conditioning for commercial T2V We compare
E MU V IDEO against two video generation variants of Gen2 API:
(1) Gen2 which accepts only a text prompt as input and (2) Gen2
I2V which accepts an input image (generated using [57]) and a text
prompt. We observe that the second variant (Gen2 I2V) outperforms the text-to-video Gen2 variant. E MU V IDEO’s generations
are strongly preferred to both the variants of the Gen2 API.

Table 9. Automatic evaluation of E MU V IDEO vs. priorwork in text-conditioned image animation. We compare E MU
V IDEO against three contemporary methods following the settingsfrom 3 using Frame consistency (FC), Image similarity (IC), andText consistency (TC). E MU V IDEO outperforms VideoComposer
across both the prompt sets and all three metrics. Automatic metrics favor static videos to ones with motion, resulting in lower
scores for E MU V IDEO compared to PikaLabs and Gen2.

tion with automated metrics, (iii) a joint investigation into
the effect of the number of training steps and data, and finally (iv) an analysis into the effect of the amount of training
data.
Image conditioning for commercial T2V systems. We
study the effect of image conditioning on the commercial
T2V solution from Gen2 [54] in Table 8. The Gen2 API
has two video generation variants: (1) A pure T2V API that
accepts a text prompt as input and generates a video; and(2) an ”image + text” API, denoted as Gen2 I2V, that takes
an image and a text prompt as input to generate a video. We
use images generated from [57] for the Gen2 I2V variant.
We observe that the Gen2 I2V variant outperforms theGen2 API that only accepts a text prompt as input. We
benchmark E MU V IDEO against both variants of the API
and observe that it outperforms Gen2 and the stronger Gen2I2V API. In Table 3, we also compare E MU V IDEO using
the same images as Gen2 I2V for “image animation” andobserve that E MU V IDEO outperforms Gen2 I2V in that setting as well.
Automated metrics for image animation. We follow thesetting from Table 3 and report automated metrics for comparison in Table 9. Following [23, 76], we report Frameconsistency (FC) and Text consistency (TC). We also report
CLIP Image similarity [10] (IC) to measure the fidelity ofgenerated frames to the conditioned image. We use CLIP
ViT-B/32 model for all the metrics. Compared to VideoComposer [76], E MU V IDEO generates smoother motion,
as measure by frame consistency, maintains a higher faithfulness to the conditioned image, as measured by the image score, while adhering to the text on both the prompt
sets. E MU V IDEO fares slightly lower compared to PikaLabs and Gen2 on all three metrics. Upon further inspection, E MU V IDEO (motion score of 4.98) generates moremotion compared to PikaLabs and Gen2 (motion scores of
0.63 and 3.29 respectively). Frame and image consistency
favour static videos resulting in the lower scores of E MU
V IDEO on these metrics.
Effect of the number of training steps and data. In Figure 8, we vary the number of training steps in the initialvs. 100%

50

Quality
Faithfulness

40
30
20
20

40

60

80

100

%age iterations during 256p stage
Figure 8. Performance vs. training iterations. On training the256px stage for fewer iterations, we compare the generations after
the same 512px finetuning to the 100% trained model via humanevaluations. We observe a gradual drop in performance, indicating
the importance of the low-resolution high-FPS pretraining stage.

vs. 100%

50
40
Quality
Faithfulness

30
20
20

40

60

80

100

%age data used for training
Figure 9. Performance vs. training data. We train our modelwith less data (for both 256px and 512px stages) while keeping
the training steps constant, and compare the generations with thethe 100% data model via human evaluations. We observe that evenwith 10% data, we only see a slight degradation in performance(∼ 43% on both Quality and Faithfulness), showcasing that our
method works well even with a fraction of the data.

low-resolution high-FPS pretraining stage. Note that since
we run one full epoch through the data during this training stage, reducing the steps correspondingly also reduces
the amount of training data seen. We finetune each of thesemodels at higher resolution/low FPS (512px, 4fps) for thesame (small) number of steps – 15K. We compare the modeltrained with 100% low-resolution pretraining with models
with less low-resolution pretraining using human evaluations. We observe a gradual drop in performance as we reduce the low-resolution pretraining iterations to 75%, 50%
15

and 25%, indicating the importance of that stage.
Effect of the amount of training data. In Figure 9, wevary the amount of training data, while keeping the training
iterations fixed for both the training stages, and perform asimilar comparison as in Figure 8. Here we find a muchsmaller drop in performance as we reduce the amount ofdata. This suggests that E MU V IDEO can be trained effectively with relatively much smaller datasets, as long as themodel is trained long enough (in terms of training steps).

generated videos from different prior works are superior to
E MU V IDEO and vice-versa. As detailed in the main paper,for each comparison between two videos from two different
models, we use the majority vote from 5 different humanevaluators. To further reduce annotator bias we make surethat the relative positioning of the generated videos being
shown to the human evaluators is randomized. For details
on how we ensure fairness in human evaluations when comparing videos with different resolutions, see Appendix D.
Next, we analyze quantitatively how JUICE improves
human evaluation reliability and robustness. To identify unbiased JUICE factors differentiating any two video generation models on Quality and Faithfulness, we made an initialpool of random video samples generated by a few models,and asked internal human raters to explicitly explain theirreasoning for picking one model over another. We thencategorized them into five reasons for Quality and two for
Faithfulness as mentioned in Section 3.2.
Effect of JUICE on improving evaluation reliability androbustness of human evaluations. We measure the reliability of our human evaluations when evaluators are required to justify their choice. For each pair of videos which
are compared, we look at the votes for model A vs. model
B and call the agreement between annotators either ‘split’(2|3 or 3|2 votes), ‘partial’ (4|1 or 1|4 votes), or ‘complete’(5|0 or 0|5 votes). We run human evaluations comparing
our generations vs. Make-A-Video, first using a naive evaluation template and then with JUICE, and show the resultsin Figure 11. We observe that the number of samples with‘split’ agreement is decreased significantly by 28%, and thenumber of ‘complete’ agreements is increased by 24%.
Next, we use Fleiss’ kappa [26] as a statistical measure for inter-rater reliability for a fixed number of raters.
This metric stands for the amount by which the observed
agreement exceeds the agreement by chance, i.e., when theevaluators made their choices completely randomly. Fleiss’
kappa works for any number of evaluators giving categorical ratings and we show the values in Figure 12. The valueof kappa is always in the range of [−1, 1], with positive
kappa values representing an agreement. To better understand its behavior and range of scores in our evaluation
setup, we perform an experiment on a simulated data representing our specific case of 304 tasks with two classes,
model A-vs-B, and five evaluators per task. We begin withcomputing the kappa value when we have a ‘complete’agreement among evaluators on all tasks, i.e. when all
five evaluators choose either model A or model B in eachtask. This run receives a kappa value of 1 (blue dot in Figure 12). We gradually decrease the number of samples withcomplete agreement by introducing samples with ‘partial’
agreement when four out of five evaluators picked model Aor model B (green line in Figure 12) Similarly, we decrease
the number of samples with complete agreement by replac-

Source#prompts
Make-A-Video [68]307
Imagen Video [38]55
65
Align Your Latents [7]
PYOCO [30]74
23Reuse & Diffuse [31]Table 10. Text prompt sets used for evaluation in our work. We
use the text prompt sets from prior work to generate videos.

C. Human evaluations
We rely on human evaluations for making quantitative comparisons to prior work. In Sec. 4 in the main paper, we introduce our method for robust human evaluations. We now
give extra details on this method, termed JUICE, and analyse how it improves robustness, and explain how we ensure
fairness in the evaluations. Additionally, in Table 10 wesummarize the prompt datasets used for evaluations.

C.1. Robust Human Evaluations with JUICE
When comparing to prior work, we use human evaluations
to compare the generations from pairs of models. Unlike thenaive approach, where evaluators simply pick their choicefrom a pair of generations, we ask the evaluators to selecta reason when making their choice. We call this approach
JUICE, where evaluators are asked to ‘justify your choice’.
We show an example of the templates used for human evaluations for both video quality and text faithfulness in Figure 10, where the different possible justifying reasons areshown. One challenge faced when asking evaluators to justify their choice is that human evaluators who are not experts in video generation may not understand what is meantby terms such as “Object/scene consistency” or “Temporal text alignment” or may have subjective interpretations,which would reduce the robustness of the evaluations. To
alleviate this challenge, for each justifying option we show
the human evaluators examples of generated video comparisons where each of the factors could be used is used in determining a winner. It is important that when giving humanevaluators training examples such as these that we do notbias them towards E MU V IDEO’s generations over those ofprior work. Thus, to ensure fairness in the comparisons, we
make sure that these training examples include cases where16Which video aligns better with the text prompt?

Which video do you prefer?
Video A

Video A

Video B

Video B

A giraffe underneath a microwave.
Which factors contributed towards making this choice?
(Select all that apply)

Which factors contributed towards making this choice?
(Select all that apply)

q Motion smoothness
q Object/scene consistency
q Pixel sharpness
q Recognizable objects/scenes
q Amount of motion(a) Video Quality

q Spatial text alignmentq Temporal text alignment(b) Video-Text Faithfulness

Figure 10. The JUICE template to compare two models in terms of (a) video quality and (b) video-text alignment. Here, human evaluators
must justify their choice of which generated video is superior through the selection of one or more contributing factors, shown here. To
ensure that human evaluators have the same understanding of what these factors mean, we additionally provide training examples of video
comparisons where each of the justifying factors could be used in selecting a winner.

Distribution of samples with different levels of agreement

1|4 or 4|1 ratings
2|3 or 3|2 ratings

1

200

150

0.6

28%

Kappa

Number of samples

Naive template
Juice

100

0.2
24%

50

−0.2

0

split

partialcomplete0Types of agreement

0.2

0.4

0.6

0.8

1

Proportion of samples with complete agreement

Figure 11. Human agreement in E MU V IDEO vs. Make-AVideo. Distribution of samples with ‘split’ (2|3 or 3|2 votes), ‘partial’ (4|1 or 1|4 votes), or ‘complete’ (5|0 or 0|5 votes) agreement
when using a naive evaluation vs. JUICE. Our JUICE evaluation
reduces ambiguity in the task and results in a 28% reduction in thenumber of samples with ‘split’ agreement and a 24% increase in
the number of samples with ‘complete’ agreement. This improves
Fleiss’ kappa from 0.004 to 0.31.

Figure 12. Analysis of Fleiss’ kappa for a simulated two-classfive-raters evaluation task. The blue dot shows the kappa value
when we have a complete agreement among evaluators on all thesamples. We progressively replace samples with 5|0 or 0|5 votes(complete agreeement) with either 1|4 or 4|1 or 3|2 or 2|3 votes
and compute the Fleiss’ kappa (shown in green and red). The
shaded region shows the kappa value for different proportions ofsamples with complete, partial or split agreements.

ing them with samples where three out of the five evaluators picked model A or model B, illustrated with a red line.
As shown in the plot, the kappa value ranges from −0.2
(ratings always being ‘split’) to 1.0 (ratings always having ‘complete’ agreement). Different proportions of sam-ples with ‘complete’, ‘partial’ or ‘split’ agreements resultin a kappa value in the shaded region. We compute andcompare kappa values for the naive evaluation and JUICE
evaluation–0.004 and 0.31, respectively–confirming the improvement in the inter-rater reliability of JUICE.
17

E MU V IDEO winning against Make-A-VideoE MU V IDEO winning against Imagen Video

Figure 13. Vertical bars show percentage of each reason and its co-occurrence with other reasons picked for E MU V IDEO against MakeA-Video (left) and Imagen Video (right). Horizontal bars depict the overall percentage of each reason, similar to Figure 6. Pixel sharpness
and motion smoothness are the two most contributing factors in the E MU V IDEO win against both baselines.
E MU V IDEO vs. Align Your Latents

E MU V IDEO vs. PYOCO

Make-A-Video vs. E MU V IDEO

E MU V IDEO vs. Reuse & Diffuse

E MU V IDEO vs. Gen2

Gen2 vs. E MU V IDEO

Imagen Video vs. E MU V IDEO

%age selected%age selected

E MU V IDEO vs. PikaLabs
60
40
20

40
20
0
MotionObjectPixel
Smoothness Consistency Sharpness

0
MotionObjectPixel
Smoothness Consistency Sharpness

VisualConcept

Amount of
MotionVisualConcept

Amount of
MotionFigure 15. Percentage of each reason selected for samples
where each baseline model wins against E MU V IDEO on Quality. Among the few preferred Make-A-Video generations from
Figure 2 against E MU V IDEO, object consistency has been the primary reason, while for Imagen Video generations, amount of motion has been an additional considerable reason. Gen2 generations
preferred over E MU V IDEO are mainly selected due to their motion smoothness and pixel sharpness.

Figure 14. Percentage of each reason selected for samples
where E MU V IDEO wins against each baseline model on Quality. Reasons that human evaluators pick E MU V IDEO generations over the baseline models from Figure 2 are primarily pixel
sharpness and motion smoothness of our videos for most models.
Amount of motion in E MU V IDEO generations is also an impactful winning factor against PYOCO and PikaLabs.

the prompt datasets that are used in comparisons to priorwork. In Appendix D.2 we detail how we sampled fromthe commercial models that we compare to in the main paper. In Appendix D.3 we give details on the postprocessing done for the human evaluations in Figure 2 in the main
paper. In Appendix D.4 we include further human evaluations conducted without postprocessing the videos fromE MU V IDEO or prior work.

Analyzing human evaluations. To clearly understand thestrengths of each model in our evaluations, we find the mostcontributing factors when E MU V IDEO generations are preferred to each baseline in Figures 6, 14. A more detailed
distribution of each reason and its co-occurrence with otherfactors is illustrated in Figure 13. We similarly, plot thepercentage of each reason picked for the best three baseline
generations preferred to E MU V IDEO in Figure 15.

D.1. Datasets used for Prior Work Comparisons

D. Comparisons to Prior Work

Since many of the methods that we compare to in Figure 2
are closed source, we cannot generate samples from all ofthem with one unified prompt dataset, and instead must construct different datasets via each method’s repsective publicly released example generated videos. In total, we use 5

In § 4.2 in the main paper, we conduct human evaluations
comparing E MU V IDEO to prior work. Here, we share further details and include human evaluation results using a
different setup. Specifically, in Appendix D.1 we outline
18

different prompt datasets. The human evaluations in Figure 2 for Make-A-Video, Imagen Video, Align Your Latents, PYOCO, and Reuse & Diffuse were conducted using
the prompt datasets from the respective papers (see Table 10
for details). Certain methods that we compare to are either
open-source (CogVideo) or can be sampled from throughan online interface (Gen2 and Pika Labs). For these, humanevaluations are conducted using the prompt set from AlignYour Latents.
Model
E MU V IDEO
Pika
Gen2
CogVideo
Reuse & Diffuse
PYOCO
Align Your Latents
Imagen Video
Make-A-Video
VideoComposer

Models Compared
E MU V IDEO vs. Pika LabsE MU V IDEO vs. Gen2
E MU V IDEO vs. CogVideoE MU V IDEO vs. Reuse & Diffuse
E MU V IDEO vs. PYOCOE MU V IDEO vs. Align Your Latents
E MU V IDEO vs. Imagen VideoE MU V IDEO vs. Make-A-VideoE MU V IDEO vs. VideoComposerVideo Dimensions
Frame Duration
T×H×W
Rate
(s)
65 × 512 × 512
72 × 768 × 768
96 × 1024 × 1792
32 × 480 × 480
29 × 512 × 512
76 × 1024 × 1024
112 × 1280 × 2048
128 × 768 × 1280
92 × 1024 × 1024
16 × 256 × 256

16
24
24
8
24
16
30
24
24
8

Dimensions after Postprocessing
Frame Duration
Rate
(s)

T×H×W

48 × 512 × 512
65 × 512 × 512
32 × 480 × 480
19 × 512 × 512
65 × 512 × 512
65 × 512 × 512
65 × 512 × 512
61 × 512 × 512
16 × 256 × 256

16
16
8
16
16
16
16
16
8

3.00
4.06
4.00
1.19
4.06
4.06
4.06
3.81
2

Table 12. Video Dimensions after postprocessing for humanevaluations.. To ensure fairness in the human evaluations inin Figure 2 in the main paper, we postprocess the videos for eachcomparison so that they have equal dimensions and hence are indistinguishable aside from their generated content. The top andbottom part of the table shows the specifications of Text-to-Videoand Image-to-Video models respectively.

4.06
3.00
4.00
4.00
1.21
4.75
3.73
5.33
3.83
2

Results for human evaluations conducted without any postprocessing are discussed in Appendix D.4.
As outlined in Appendix C, our human evaluations are
conducted by showing evaluators repeated comparisons ofvideos generated by two different models for the same
prompt, and asking them which model they prefer in termsof the metric being evaluated. It is key for the fairness ofthe human evaluation that the evaluator treats each comparison independently. It is hence important that the evaluator
does not know which model generated which video, otherwise they can become biased towards one model over theother. Since each method generates videos at different dimensions (see Table 11), conducting the human evaluations
without postprocessing the videos would lead to this annotator bias. Hence we decide to postprocess the videos being
compared such that they have the same aspect-ratios, dimensions and frame rates so that they are indistinguishable
aside from their generated content. For each pair of models
being compared, we downsample these dimensions to theminimum value between the two models (see Table 12 fordetails). Next, we detail how we postprocess the videos.

Table 11. Video Dimensions. The dimensions of the generated
videos from E MU V IDEO and each of the prior work. The top andbottom part of the table shows the specifications of Text-to-Videoand Image-to-Video models respectively. Each of the prior works
generates videos at different dimensions, making unbiased humanevaluation a challenge.

D.2. Sampling from Commercial Models
The commercially engineered black-box text-to-video models that we compare to (Pika Labs and Gen2) can be sampled from through an online interface. Here we include
details for how we sampled from these models. In bothcases, these interfaces allow for certain hyper-parameters
to be chosen which guide the generations.
We selected optimal parameters for each of the models
by varying the parameters over multiple generations andchoosing those that consistently resulted in the best generations. For Pika Labs, we use the arguments “-ar 1:1 -motion
2” for specifying the aspect ratio and motion. For Gen2, we
use the “interpolate” and “upscale” arguments and a “General Motion” score of 5. All samples were generated on
October 24th 2023.

Aspect Ratio. Since E MU V IDEO generates videos at a
1:1 aspect ratio, all videos are postprocessed to a 1:1 aspectratio by centre cropping.
Spatial Dimension. The height and width of videos areadjusted using bilinear interpolation.
Video Duration. The duration of videos is reduced via temporal centre cropping.

D.3. Postprocessing Videos for Comparison
Our goal with our main human evaluations in Figure 2 isto ensure fairness and reduce any human evaluator bias. To
ensure this fairness, we postprocess the videos from eachmodel being compared (as outlined in § 4.2 in the main paper). Here, we give further details on the motivation behind
this decision, and explain how this postprocessing is done.

Frame rate. The frame rate is adjusted using torchvision.
The number of frames is selected according to the desired
frame rate and video duration.
Next we discuss human evaluation results where videos
are compared without any postprocessing.
19

#Prompts
Quality
Faithfulness

Make-A-Video Imagen Video Align Your Latents PYOCO Reuse & Diffuse CogVideo Gen2 PikaLabs
307 [68]
55 [38]65 [7]
74 [30]23 [31]65 [7]
65 [7]
65 [7]
96.8
90.9
96.9
93.2
95.7
100.0
83.1
93.9
86.0
69.1
90.8
89.2
100.0
100.0
98.5
100.0

Table 13. E MU V IDEO vs. prior work where videos are not postprocessed. We evaluate text-to-video generation in terms of video
quality and text faithfulness win-rates evaluated by the majority votes of human evaluators for E MU V IDEO vs. Prior work methods. We
compare methods here with their original dimensions (aspect ratio, duration, frame rate). E MU V IDEO significantly outperforms all priorwork across all settings and metrics.

D.4. Prior Work at Original Dimensions

tative comparisons between E MU V IDEO and prior work(in Appendix E.2)

In this Section, we include further human evaluation results between E MU V IDEO and prior work where we do not
perform any postprocessing on the videos and conduct theevaluations with the original dimensions (as detailed in Table 11). In this system-level comparison, human evaluators
are comparing between videos that may have very different
aspect ratios, durations, and frame rates, and in turn may
become biased towards one model over another after seeing
repeated comparisons. We note that since the dimensions of
the videos here are so large, we must scale the height of each
video so that both compared videos can fit on one screen
for human evaluators. All other dimensions remain as in
the original sampled videos. The results are in Table 13.
Similar to the human evaluations conducted with postprocessed videos in Figure 2 in the main paper, E MU V IDEO
significantly outperforms prior work in terms of both text
faithfulness and video quality. Even when comparing E MU
V IDEO’s generated videos to generated videos with longer
durations (including PYOCO, Imagen Video), wider aspectratios (incliding Gen2, Align Your Latents), or higher frame
rates (including Pika, Gen2), human evaluators still prefer
E MU V IDEO’s generated videos in both metrics. We hypothesize that the vastly improved frame quality and temporal consistency of E MU V IDEO still outweighs any benefits
that come from any larger dimensions in the prior work’s
videos.
Interestingly, E MU V IDEO wins by larger margins here
than in the postprocessed setting (an average win rate of
93.8% in quality and 93.1% in faithfulness here, vs. 91.8%
and 86.6% in the postprocessed comparison). We conjecture that this improvement in win rates for E MU V IDEO
may be due to the potential evaluator bias introduced in thisevaluation setting. This introduced bias tends to favor E MU
V IDEO since our video generations are on average superior
in terms of quality and faithfulness than those of prior work.
Hence in this paper we primarily report and refer to the human evaluation scores from the fairer postprocessed setting.

E.1. Further E MU V IDEO qualitative Results
Examples of E MU V IDEO’s T2V generations are shown in
Figure 16, and E MU V IDEO’s I2V generations are shownin Figure 17. As shown, E MU V IDEO generates high quality video generations that are faithful to the text in T2V
and to both the image and the text in I2V. The videos havehigh pixel sharpness, motion smoothness and object consistency, and are visually compelling. E MU V IDEO generates
high quality videos for both natural prompts and fantastical
prompts. We hypothesize that this is because E MU V IDEO
is effectively able to retain the wide range of styles and diversity of the T2I model due to the factorized approach.

E.2. Qualitative Comparisons to Prior WorkWe include further qualitative comparisons to prior workin Figs. 18, 19, 20, 21, 22 and 23. This Section complements § 4.2 in the main paper where we quantatively
demonstrate via human evaluation that E MU V IDEO significantly outperforms the prior work in both video quality
and text faithfulness. E MU V IDEO consistently generates
videos that are significantly more text faithful (see Figs. 19
and 21), with greater motion smoothness and consistency(see Figs. 20 and 22), far higher pixel sharpess (see Figure 23), and that are overall more visually compelling(see Figure 18) than the prior work.

E. Qualitative Results
In this Section, we include additional qualitative resultsfrom E MU V IDEO (in Appendix E.1), and further quali20

(Ours - E MU V IDEO) Prompt: A hamster wearing virtual reality headsets is a dj in a disco.

(E MU V IDEO) Prompt: A massive tidal wave crashes dramatically against a rugged coastline.

(E MU V IDEO) Prompt: A majestic white unicorn with a golden horn walking in slow-motion under water.

(E MU V IDEO) Prompt: A grizzly bear hunting for fish in a river at the edge of a waterfall, photorealistic.

Figure 16. Example T2V generations from E MU V IDEO for a selection of diverse prompts (shown above each row of frames). E MU V IDEO
generates natural-looking videos which are faithful to the text and high in visual quality. The videos are highly temporally consistent, withsmooth motion. E MU V IDEO is able to generate high quality videos for both natural prompts (rows 2 and 4) depicting scenes from thenatural world, and also fantasical prompts including DJing hamsters (row 1) and underwater unicorns (row 3).

21

(Ours - E MU V IDEO) Prompt: The American flag waving during the moon landing with the camera panning.

(E MU V IDEO) Prompt: The sun sets and the moon rises.

(E MU V IDEO) Prompt: Satellite flies across the globe.

(E MU V IDEO) Prompt: horse moving its legs.

Figure 17. Example I2V generations from E MU V IDEO for a selection of diverse prompts (shown above each row of frames). E MU V IDEO
generates natural-loooking videos from the conditioning image (shown in a blue box on the left side of each row of frames) and the text
prompt, that have smooth and consistent motion.

22

(Ours - E MU V IDEO) Prompt: An astronaut flying in space, 4k, high resolution.

(Gen2) Prompt: An astronaut flying in space, 4k, high resolution.

(PikaLabs) Prompt: An astronaut flying in space, 4k, high resolution.

(Align Your Latents) Prompt: An astronaut flying in space, 4k, high resolution.

(CogVideo) Prompt: An astronaut flying in space, 4k, high resolution.

Figure 18. Example T2V generations from E MU V IDEO and a selection of prior work methods that we compare to in the main paper
for the same prompt, namely Gen2, Pika Labs, Align your latents, and CogVideo. E MU V IDEO generates higher quality videos that are
more faithful to the text, have realistic & smooth movement, and are visually compelling. In this example, CogVideo cannot generate anatural-looking video (see 5th row). PikaLabs is not faithful to the text and does not generate a realistic looking astronaut (see 3rd row),whereas Align Your Latents generates a video with low visual quality. Gen2’s video, although visually superior to other prior work, lacks
pixel sharpness and is not as visually compelling as E MU V IDEO.

23

(Ours - E MU V IDEO) Prompt: Teddy bear walking down 5th Avenue, front view, beautiful sunset, close up, high definition, 4k.

(Gen2) Prompt: Teddy bear walking down 5th Avenue, front view, beautiful sunset, close up, high definition, 4k.

(PikaLabs) Prompt: Teddy bear walking down 5th Avenue, front view, beautiful sunset, close up, high definition, 4k.

(Align Your Latents) Prompt: Teddy bear walking down 5th Avenue, front view, beautiful sunset, close up, high definition, 4k.

(CogVideo) Prompt: Teddy bear walking down 5th Avenue, front view, beautiful sunset, close up, high definition, 4k.

Figure 19. Example T2V generations from E MU V IDEO and a selection of prior work methods that we compare to in the main paper forthe same prompt, namely Gen2, Pika Labs, Align your latents, and CogVideo. CogVideo and PikaLabs’s videos are not faithful to the text
and lack on visual quality. Gen2 correctly generates a video of a bear on a street, but the bear is not moving, and there is limited motionin the vidoeo. Align Your Latents’s video lacks motion smoothness and pixel sharpness. On the other had, E MU V IDEO’s video has very
high visual quality and high text faithfulness, with smooth and consistent high motion.

24

(Ours - E MU V IDEO) Prompt: A clear wine glass with turquoise-colored waves inside it.

(Imagen Video) Prompt: A clear wine glass with turquoise-colored waves inside it.

(Ours - E MU V IDEO) Prompt: A panda bear driving a car.

(Imagen Video) Prompt: A panda bear driving a car.

Figure 20. Example T2V generations from E MU V IDEO and Imagen Video on two prompts (which are shown above each row of frames).
Imagen Video generates videos that are faithful to the text, however the videos lack in pixel sharpness and motion smoothness. Additionally
Imagen Video’s generations lack fine-grained high-quality details such as in the panda’s hair (see 4th row) and the water movements (see2nd row). E MU V IDEO on the other hand generates high quality videos that are faithful to the text, and with high pixel sharpness andmotion smoothness. E MU V IDEO accurately generates natural looking fine-grained details such as the hair on the panda (see 3rd row) andthe water droplets in the waves (see 1st row).

25

(Ours - E MU V IDEO) Prompt: A robot dj is playing the turntable, in heavy raining futuristic tokyo rooftop cyberpunk night,
sci-fi, fantasy, intricate, elegant, neon light, highly detailed, concept art, soft light, smooth, sharp focus, illustration.

(PYOCO) Prompt: A robot dj is playing the turntable, in heavy raining futuristic tokyo rooftop cyberpunk night,
sci-fi, fantasy, intricate, elegant, neon light, highly detailed, concept art, soft light, smooth, sharp focus, illustration.

(Ours - E MU V IDEO) Prompt: A cute funny robot dancing, centered, award winning watercolor pen illustration,
detailed, isometric illustration, drawing.

(PYOCO) Prompt: A cute funny robot dancing, centered, award winning watercolor pen illustration,
detailed, isometric illustration, drawing.

Figure 21. Example T2V generations from E MU V IDEO and PYOCO on two prompts (which are shown above each row of frames).
Whereas PYOCO’s videos lack motion smoothness or consistency and cannot generate fine-grained details, E MU V IDEO instead generates
highly realistic videos that are smooth and consistent. E MU V IDEO can generate high quality videos given fantastical prompts.

26

(Ours - E MU V IDEO) Prompt: There’s a dog with a harness on that is running through an open field and flying a kite.

(Make-A-Video) Prompt: There’s a dog with a harness on that is running through an open field and flying a kite.

(Ours - E MU V IDEO) Prompt: A person standing in the ocean fishing.

(Make-A-Video) Prompt: A person standing in the ocean fishing.

Figure 22. Example T2V generations from E MU V IDEO and Make-A-Video on two prompts (which are shown above each row of frames).
whereas Make-A-Video’s videos lack pixel sharpness and object consistency, E MU V IDEO generates high quality and natural-looking
videos. E MU V IDEO’s videos have high motion smoothness and object consistency.

27

(Ours - E MU V IDEO) Prompt: A sailboat is sailing on a sunny day in a mountain lake.

(Reuse & Diffuse) Prompt: A sailboat is sailing on a sunny day in a mountain lake.

(Ours - E MU V IDEO) Prompt: Waves are crashing against a lone lighthouse, ominous lighting.

(Reuse & Diffuse) Prompt: Waves are crashing against a lone lighthouse, ominous lighting.

Figure 23. Example T2V generations from E MU V IDEO and Reuse & Diffuse on two prompts (which are shown above each row offrames). whereas Reuse & Diffuse’s videos lack in visual quality both in terms of pixel sharpness, and temporal consistency, E MU V IDEO
instead generates visually compelling and natural-looking videos which accurately follow the prompt.

28

E MU V IDEO

Make-A-Video

Figure 24. Zero-Shot text-to-video generation on UCF101. The classes for these videos from top to bottom are: walking with a dog,
biking, handstand pushups, skiing. Our generations are of higher quality and more coherent than those from Make-A-Video.

29