Make-A-Video: Text-to-Video Generation without Text-Video Data
Authors: Uriel Singer
Adam Polyak
Thomas Hayes
Xi Yin
Jie An
Songyang Zhang
Qiyuan Hu
Harry Yang
Oron Ashual
Oran Gafni
Devi Parikh
Sonal Gupta
Yaniv Taigman
Venue: ICLR 2023 poster (2023)

Abstract: We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. 
We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

Published as a conference paper at ICLR 2023
MAKE-A-VIDEE TEXT-TO-VIDEO GENERATION WITHOUT TEXT- VIDEO DATA
Uriel Singer
Adam Polyak
Thomas Hayes
Xi Yin
Jie An
Songyang Zhang
Qiyuan Hu
Harry Yang
Oron Ashual
Oran Gafni
Devi Parikh
Sonal Gupta *
Yaniv Taigman"
Meta AI
ABSTRACT
We propose Make-A -iieeo- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to- Video (T2V). Our intuition is simple: learn what world looks like and how is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Videe has three advantages: (I) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's 's image generation models. We design simple yet effective way to build on T21 models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline generate high resolution and frame rate videos with video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Vides sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.
INTRODUCTION
The Internet has fueled collecting billions (alt-text, image) pairs from HTML pages (Schuhmann et al. (2022), enabling the recent breakthroughs in Text-to- Image (T2I) modeling. However, repli- cating this success for videos is limited since a similarly sized (text, video) dataset cannot easily collected. It would be wasteful to train Text-to-Video (T2V) models from scratch when there already exist models that can generate images. Moreover, unsupervised learning enables networks to learn from orders of magnitude more data. This large quantity of data is important to learn representa- tions of more subtle, less common concepts in the world. Unsupervised learning has long had great success in advancing the field of natural language processing (NLP) (Liu et al.) 22199a Brown al.) 2020). Models pre-trained this way yield considerably higher performance than when solely trained supervised manner.
Inspired by these motivations, we propose Make-A-Video. Make-A-Vides leverages T2I models learn the correspondence between text and the visual world, and uses unsupervised learning unlabeled (unpaired) video data, to learn realistic motion. Together, Make-A Video generates videos from text without leveraging paired text-video data.
Clearly, text describing images does not capture the entirety of phenomena observed in videos. That said, one can often infer actions and events from static images (e.g. woman drinking coffee, or an
Core Contributors. Corresponding author: urielsi s iinggremeta com. Jie and Songyang are from University of Rochester (work done during internship Meta).

---

Published as a conference paper at ICLR 2023
[a] A dog wearing superhero outfit with red cape flying through the sky.
(b) There table window with sunlight streaming through illuminating pile of books.
Robot dancing times square.
(d) running along beach, highly detailed.
Figure I: T2V generation examples. Our model can generate high-quality videos with coherent motion for a diverse set of visual concepts. In example (a), there are large and realistic motion for the dog. In example (b), the books are almost static but the scene changes with the camera motion. Video samples are available at imake-a-video.gituubb.
elephant kicking a football) as done in image-based action recognition systems (Girish et al. 2020). Moreover, even without text descriptions, unsupervised videos sufficient learn how different entities in the world move and interact (e.g. the motion of waves at the beach, or of an elephant's trunk). As result, a model that has only seen text describing images is surprisingly effective at generating short videos, as demonstrated by our temporal diffusion basedd method. Make-A- A-Video sets the new state-of-the-an in T2V generation.
Using function- preserving transformations, we extend the spatial layers at the model initialization stage, include temporal information. The extended spatial-temporal network includes new tention modules that learn temporal world dynamics from collection of videos. This procedure significantly accelerates the T2V training process by instantaneously transferring the knowledge from previously trained T2I network to a new T2V one. To enhance the visual quality, we train spatial super-resolution models as well as frame interpolation models. This increases the resolution of the generated videos, well as enables higher (controllable) frame rate.
Our main contributions are:
We present Make-A-Vidde an effective method that extends diffusion-basele T2I model to T2V through spatiotemporally factorized diffusion model.
We leverage joint text-image priors to bypass the need for paired text-video data, which in turn allows to potentially scale to larger quantities of video data.
We present super-resolution strategies in space and time that, for the first time, generate high-definition, high frame-rate videos given user-provided textual input.
We evaluate Make-A- A-didoo against existing T2V systems and present: (a) State-of-the-an the-art results in quantitative well as qualitative measures, and (b) A more thorough evaluation than existing literature in T2V. We also collect a test set of 300 prompts for zero-shot T2V human evaluation which plan release.
2

---

Published as a conference paper at ICLR 2023
2 PREVIOUS WORK
Text-to-Image Generation. (Reed et al. 2016) is among the first methods to extend uncondi-
tional Generative Adversairal Network (GAN) GGodfellow al., 2014) T2I generation. Later
GAN variants have focused on progressive generation (Zhang et al. 2017 Hong et al. 2018). or
better text-image alignment (Xu al. 2018 Zhang et al., 2021). The pioneering work of DALL- translation problem us-
E (Ramesh et al., 2 considers T2I generation as a Soqueence-o--equeeme
ing a discrete variational anto-enccodrr (VQVAE) and Transformer (Vaswani et al. (017)). Additional variants (Ding et al. (022)) have been proposed since then. For example, Make-A-Scaar
(Gafni
jet al., (022) explores controllable '21 generation using semantic maps. Parti (Yu et al.] (022a)
aims for more diverse content generation through an encoder-decoder architecture and an improved image tokenizer (Yu et al. 2021). On the other hand, Denoising Diffusion Probabilistic Models successfully leveraged for T2I generation. GLIDE (Nichol et al.
(DDPMs) (Ho al. 2020
2021 trained a T21 and an upsampling diffusion model for cascade generation. GLIDE's proposed classifier-free guidance has been widely adopted in T2I generation to improve image quality and
text faithfulness. DALLE-2 (Ramesh et al. 2022) leverages the CLIP (Radford et al. (202T) latent space and prior model. VQ-diffusion (Gu et al. 2022) and stable diffusion (Rombach et al. 2022)
performs T21 generation in the latent space instead of pixel space to improve efficiency.
Text-to-Vider Generation. While there is remarkable progress in T2I generation, the progress of T2V generation lags behind largely due to two main reasons: the lack of large-scale datasets with high-quality text-video pairs, and the complexity of modeling higher-dimensional video data. Farly works (Mittal et al.) 2017 Pan et al. 2017: Marwah et al.) 2017 Li et al., [0088] Gupta et al. 2018
Liu et 20T99) are mainly focused on video generation simple domains, such as moving digits or specific human actions. To our knowledge, Sync-DRAW (Mittal et al.) (0177) is the first T2V
generation approach that leverages a VAE with recurrent attention. (Pan et al. 2017) and (Li et al.
2018) extend GANs from image generation T2V generation.
More recently, GODIVA (Wu et al., 202 is the first to use 2D VOVAE and sparse attention for T2V generation supporting more realistic scenes. NÜWA (Wu et al. 2021bb extends GODIVA, and presents a unified representation for various generation tasks in a multitask learning scheme. To further improve the performance of T2V generation, Cog Video (Hong al.) 2022 is built top of a frozen Cog Viewi2 (Ding et al. (022) T2I model by adding additional temporal attention modules. Video Diffusion Models (VDM) (Ho et 2022) uses space-time factorized U-Net with joint image and video data training. While both Cog Video and VDM collected 10M private text-video pairs for training, our work uses solely open-source datasets, making it easier to reproduce.
Leveraging Image Priors for Video Generation. Due to the complexity of modeling videos and the challenges in high-quality video data collection, it natural to consider leveraging image priors for videos to simplifying the learning process. After all, an image is a video with single frame (Bain) et al. 202T). In unconditional video generation, MoCoGAN-HD (Tian et al., (02T) formulates video generation as the task of finding trajectorv in the latent space of a pre- trained and fixed image generation model. In T2V generation, NÜWA (Wu et al. 2021b) combines image and video datasets in multitask pre-training stage to improve model generalization for fine-tuning. Cog Video (Hong let al., 2022) uses a pre-trained and fixed T2I model for T2V generation with only a small number of trainable parameters to reduce memory usage during training. But the fixed autoencoder and T2I models can be restrictive for T2V generation. The architecture of VDM (Ho et al. (002) can enable joint image and video generation. However, they sample random independent images from random videos as their source of images, and do not leverage the massive text-image datasets.
Make-A- VVddeo differs from previous works in several aspects. First, our architecture breaks the dependency on text-video pairs for T2V generation. This is significant advantage compared to prior work, that has to be restricted to narrow domains (Mittal et al. 2017 Gupta et al., 2018 Ge et al., 2022; Hayes et al., (022). or require large-scale paired text-video data (Hong et al. 2022] Ho et 2022). Second, we fine-tune the T2I model for video generation, gaining the advantage of adapting the model weights effectively, compared to freezing the weights as in Cog Video (Hong 2022). Third, motivated from prior work on efficient architectures for video and 3D vision Ye et al., 2019; Qiu et al. 2017: Xie et al. 2018), our use of pseudo-3D convolution (Qiu ct al. 2017) and temporal attention layers not only better leverage a T2I architecture, it also allows
et al.
tasks
for better temporal information fusion compared to VDM (Ho et al. 2022
3

---

Published as a conference paper at ICLR 2023
Spatiotemporal Decoder
Spaciotemporal Super Resolation
Spatial Super Ressllttin SRn
Frame Interpolation
SR
Input Text
x
Fixoxd Volas
"A person
doing yoga
outdoor during
Prior
pps
sunrise
Generated Video
Fixed
Norse
Figure 2: Make-A-Video high-level architecture. Given input text translated by the prior into an image embedding, and a desired frame rate fps, the decoder D generates 16 64 x 64 frames, which are then interpolated to a higher frame rate by fr,, and increased in resolution 256 256 by SR; and 768 x 768 by SR,, resulting in a higgh-ppatotempprrrrrssstoluuu resouuttionn generated video ÿ.
3 METHOD
Make- Aiie-AA-iieo consists of three main components: (i) A base T2I model trained text-image pairs (Sec. [.T)) (ii) spatiotemporal convolution and attention layers that extend the networks' building
blocks to the temporal dimension (Sec. (32). and (iii) spatiotemporal networks that consist of both
spatiotemporal layers, as well as another crucial element needed for T2V generation a frame inter- polation network for high frame rate generation (Sec. 3.3
Nake-A-Video's final T2V inference scheme (depicted in Fig. 2 can be formulated as:
SRA SRÍ fF D o (τ, Cz())),
(1)
where ji. is the generated video, SRa, SR, are the spatial and spatiotemporal super-resolution net-
works (Sec. 3.2). F is frame interpolation network (Sec. 3.3 D is the spatiotemporal decoder
(Sec. 3.2) P is the prior (Sec. 3.J). & is the BPE-encoded text, C₂ is the CLIP text encoder (Rad-) Ford et al. 202T), and is the input text. The three main components are described in detail in the following sections.
3.1 TEXT-TO-IMAGE MODEL
Prior to the addition of the temporal components, we train the backbone of our method: a T21 model trained on text-image pairs, sharing the core components with the work of (Ramesh et al. 2022).
We use the following networks to produce high-resolution images from text: (i) A prior network P. that during inference generates image embeddings Ye given text embeddings T. and BPE encoded text tokens 2, (ii) decoder network D that generates a low-rcsolution 64 x 64 RGB image It. conditioned on the image embeddings Ye. and (iii) two super-resolution networks SR,RR, that increase the generated image in resolution to 256 x 256 and 768 x 768 pixels respectively, resulting in the fina generated image ÿ.
3.2 SPATIOTEMPORA LAYERS
In order to expand the awo-dimensional (2D) conditional network into the temporal dimension, modify the two key building blocks that now require not just spatial but also temporal dimensions in order generate videos: (i) Convolutional layers (Sec. (.2.I). and (ii) attention layers (Sec. 3.2.2) discussed in the following two subsections. Other layers. such as fully-connected layers, do not require specific handling when adding an additional dimension, as they are agnostic to structured spatial and temporal information. Temporal modifications are made in most U-Net-based diffusion networks: the spatiotemporal decoder D) now generating RGB frames, each of size 64 x 64, the
We then downsample to 512 using bicubic interpolation for cleaner aesthetic. Maintaining clean aes- thetic for high definition videos part of future work.
4

---

Published as a conference paper at ICLR 2023
Periban Rooiled Eapettogo
Frane Pretina Ranuday
Special
Altentia
Tomporal
4
Nest Rodiee Lawe topet
App Conuctios
Rarbekied bage Zike
Es Eakedee tabees Frmee Siy Coneciip
Figure 3: The architecture and initialization scheme of the Pseudo-3D convolutional and at- tention layers, enabling the seamless transition of pre-trained Text-to-Image model the temporal dimension. (left) Each spatial 2D conv layer is followed by a temporal ID conv layer. The temporal conv layer is initialized with an identity function. (right) Temporal attention layers are applied following the spatial attention layers by initializing the temporal projection to zero, resulting in an identity function of the temporal attention blocks.
newly added frame interpolation network fr, increasing the effective frame rate by interpolating between the 16 generated frames (as depicted in Fig. 2 and the super-resolution networks SR/.
Note that super resolution involves hallucinating information. In order to not have flickering ar- tifacts, the hallucination must be consistent across frames. As result, across spatial and temporal dimensions. In qualitative inspection we found this to significantly out- perform per-frame super resolution. It challenging to extend SRA the temporal dimension due to memory and compute constraints, as well as scarcity of high resolution video data. So SRA operates only along the spatial dimensions. But encourage consistent detail hallucination across frames, we use the same noise initialization for each frame.
SR module operates
3.2.1 PSEUDO-3D CONVOLUTIONAL LAYERS
Motivated by separable convolutions (Chollet 2017). we stack ID convolution following each 2D convolutional (conv) layer, as shown Fig. 3 This facilitates information sharing between the spatial and temporal axes, without succumbing to the heavy computational load of 3D conv layers. In addition, creates concrete partition between the pre-trained 2D conv layers and the newly initialized 1D) conv layers, allowing us to train the temporal convolutions from scratch, while retaining the previously learned spatial knowledge in the spatial convolutions' weights.
Given input tensor h E R BxCx Fx HxW where B, C, F, H, W are the batch, channels, frames, height, and width dimensions respectively, the Pseudo-3D convolutional layer defined as:
Conup3D (h) == Conelp o T) o
where the transpose operator eT swaps between the spatial and temporal dimensions. For smooth initialization, while the Conwep layer is initialized from the pre-trained T2I model, the Coney layer is initialized as the identity function, enabling seamless transition from training spatial-only layers, to spatiotemporal layers. Note that at initialization, the network will generate K different images (due to random noise), each faithful to the imput text but lacking temporal coherence.
3.2.2 PSEUDO-3D ATTENTION LAYERS
crucial component of T2I networks is the attention layer, where in addition to self-attending to ex- tracted features, text information is injected to several network hierarchies, alongside other relevant information, such as the diffusion time-step. While using 3D convolutional layers is computationally heavy, adding the temporal dimension to attention layers is outright infeasible in terms of memory consumption. Inspired by the work of (Ho et al. 2022). we extend our dimension decomposition strategy to attention layers as well. Following each (pre-trained) spatial attention layer, stack temporal attention layer, which as with the convolutional layers, approximates full spatiotemporal attention layer. Specifically, given input tensor h, we define flatten matrix operator that
5

---

Published as a conference paper at ICLR 2023
flattens the spatial dimension into h' RBXXXFXиW unflatten is defined as the inverse matrix operator. The Pseudo-3D attention layer therefore therefore defined as:
(T) = (TT
o T) T)-
(3)
Similarly to CONUPSD. to allow for smooth spatiotemporal initialization, the ATD layer is ini- tialized from the pre-trained T2I model and the ATTNNb layer is initialized as the identity function.
Factorized space-time attention layers have also been used VDM (Ho et al., 2022) and CogVideo (Hong et al., (022). CogVideo has added temporal layers to each (frozen) spatial layers whereas we train them jointly. order to force their network to train for images and videos inter- changeably, VDM has extended their 2D U-Net to 3D through unflattened 1x3x3 convolution filters, such that the subsequent spatial attention remains 2D, and added ID temporal attention through rel- ative position embeddings. In contrast, we apply an additional 3x1x1 convolution projection (after each Ix3x3) such that the temporal information will also be passed through each convolution layer.
Frame rate conditioning. In addition the T2I conditionings, similar to Cog Video (Hong al.) 2022). we add an additional conditioning parameter fps, representing the number of frames-per- second generated video. Conditioning on a varying number of trames-per-sccome enables an additional augmentation method to tackle the limited volume of available videos at training time, and provides additional control on the generated video inference time.
Objectives. We optimize the model by minimizing the hybrid loss following Nichol & Dhariwal (02T]:Rameste et al. (2022) train the video decoder. Specifically,, the loss consists of two terms: simple loss that learns to predict the added noise and a loss Loi that adds constraint the estimated variational lower bound (VLB). The Luds term is applied the same way as in Nichol &
Dhariwal (02II). Thus, only write the loss term that predicts the added noise
Ldecoder Ecytyol....pss [lec (ze. Cy(y)), fps, []][]]
(4)
where y is an input video and yo represents the first frame of this video. Cy (yr) denotes the extracted CLIP image embedding of the first frame. 2, is the noisy input added to y at time step t that is uniformly sampled from T during training. f'ss is the frame rate embedding as described above. E is the added noise that is to be estimated by the network represented as Eg.
3.3 FRAME INTERPOLATION NETWORK
In addition to the spatiotemporal modifications discussed in Sec. 3.2 we train new masked frame interpolation and extrapolation network 1r, capable of increasing the number of frames of the gen- erated video either by frame interpolation for a smoother generated video, or by pre/post frame extrapolation for extending the video length. In order to increase the frame rate within memory and compute constraints, we fine-tune a spatiotemporal decoder D on the task of masked frame inter- polation, by zero-padding pro--paddnn the masked input frames, enabling video upsampling. When fine-tuning on masked frame interpolation, we add additional 4 channels to the input of the U-Net: 3 chan- nels for the RGB masked video input and an additional binary channel indicating which frames are masked. We fine-tune with variable frame-skips and fps conditioning to enable multiple temporal upsample rates at inference time. The training objective is the same as Eq, $ exceep that we add the additional condition of the unmasked frames. We denote tr as the operator that expands the given video tensor through masked frame interpolation. For all of our experiments we applied iF with frame skip 5 to upsample 16 frame video 76 frames ((1--1)x5+)) Note that we can use the same architecture for video extrapolation or image animation by masking frames at the beginning or end of a video.
3.4 TRAINING
The different components of Make-A- described above are trained independently. The only component that receives text as input is the prior P. We train it on paired text-image data and do not fine-tune it on videos. The decoder and two super-resolution components are first trained on images alone (no aligned text). Recall that the decoder receives CLIP image embedding as input, and the super-resolution components receive downsampled images as input during training. After training on images, we add and initialize the new temporal layers and fine-tune them over unlabeled video data. 16 frames are sampled from the original video with random fps ranging from 1 to 30. We
6

---

the Published as a conference paper at ICLR 2023
Table 1: T2V generation evaluation MSR-VTT. Zero-Shot means no training is conducted on
MSR-VTT. Samples/Input means how many samples are generated (and then ranked) for each input.
Method GODIVA
Zero-Shot
Samples/Input
Resolution
FID (L)
CLIPSIM (t)
No
30
128 x 128
0.2402
NÜWA
No
336 x 336
47.68
0,2439
Cog (Chinese)
Yes
480 480
24.78
0.2614
CogVideo (English)
Yes
1
480 x 480
23.59
0.2631
Make- A-Video (ours)
Yes
256 x 256
13.17
0.3049
use the beta function for sampling and while training the decoder, start from higher FPS ranges (less motion) and then transition lower FPS ranges (more motion). The masked faaaeed--rameennepppllaaooo component is fine-tuned from the temporal decoder.
4
EXPERIMENTS
4.1 DATASETS AND SETTINGS
Datasets. To train the image models, use a 2.3B subset of the dataset from (Schuhmann et al.) where the text is English. We filter out sample pairs with NSFW images toxic words in the text, or images with a watermark probability larger than 0.5. We use Web and a 10M subset from HD-VILA-1000
(Bain et al.) 202T) (Xue et al. 2022 to train our video generation models. Note that only the videos (no aligned text) are used. The decoder D) and the interpolation model is trained on Web 10M. SR is trained on both Ved-id-I and HD-VID AED-VILA-ONS While prior work (Hong et al. 2022, Ho et al. (2022) have collected private text-video pairs for T2V generation, only public datasets (and no paired text for videos). We conduct automatic evaluation UCF-101 (Sommo et al. 2012) and MSR- VTT (Xu et al. 2016 in a zero-shot setting.
Automatic Metrics. For UCF-101, write one template sentence for each class (without generat- ing any video) and fix for evaluation. We report Frechet Video Distance (FVD) and Inception Score (IS) on 10K samples following (Ho et al. 2022). We generate samples that follow the same class distribution as the training set. For MSR-V VTT, we report Frechet Inception Distance (FID) (Parmar al., 2022) and CLIPSIM (average CLIP similarity between video frames and text) (Wu et al.j 2021a), where all 59, 794 captions from the test set are used, following (Wu et al. 20216).
Human Evaluation Set and Metrics. We collect an evaluation set from Amazon Mechanical Turk (AMT) that consists of 300 prompts. We asked annotators what they would be interested in gener- ating if there were T2V system. We filtered out prompts that were incomplete (e.g.. "jump into water"), too abstract (e.g.. "climate change"), or offensive. We then identified 5 categories (animals, fantasy, people, nature and scenes, food and beverage) and selected prompts for these categories. These prompts were selected without generating any videos for them, and were kept fixed. In addi- tion, we also used the Draw Bench prompts from Imagen (Saharia et al. 2022) for human evaluation. We evaluate video quality and text-video faithfulness. For video quality, we show two videos in ran- dom order and ask annotators which one is of higher quality. For faithfulness, we additionally show the text and ask annotators which video has better correspondence with the text (we suggest them to ignore quality issues). In addition, we also conducted human evaluation to compare video motion realism of our interpolation model and FILM (Reda al. 2022). For each comparison, majority vote from 5 different annotators as the final result.
4.2 QUANTITATIVE RESULTS
Automatic Evaluation MSR-VTT. In addition GODIVA and NÜWA that report on MSR- VTT. we also perform inference on the officially released CogVider model with both Chinese and English inputs for comparison.Fa Cog Video and Make-A-Video we only generate one sample for each prompt in a zero-shot setting. We only generate videos that are at 16 x 256 x 256 as the evaluation models do not expect higher resolutions and frame rate. The results are shown in Table
We used this model:
Thhse 100M clips are sourced from 3.1M 1M videos. We randomly downloaded 3 clips per video to form HD-VILAA-OM subset.
7

---

Published as a conference paper at ICLR 2023
Table 2: Video generation evaluation UCF-101 for both zero-shot and fine-tuning settings.
Method
Pretrain
Class
Resolution
IS (t)
FVD (J)
Zero-Shar Setting
CogVideo (Chinese) VogVideo (English) Make-A Bokdee- (ours)
No
Yes
480
480
23.55
751.34
No
Yes
480 x 480
25.27
701.59
No
Yes
256 256
33.00
367.23 23
Finetuning Setting
(Satoo et al. 2020
No
No
128 x 128
26.60 + 0.47
DIGAN Yu al. 2022b)
No
No
32.70 + 0.35
577 + 22
MoCoGAN-HDi Jan et al.
2021
No
No
256 x 256
33.95 0.25
700 + 24
Videi (Hong al. 2022
Yes
Yes
160 160
50.46
626
VDM (Ho èt al. 2022
No
No
64 x 64
57.80 + 1.3
TATS- AATS-base(Ge al. 2022)
No
Yes
128 128
79.28 + 0.38
278 + 11
Make-A A-ideo (ours)
Yes
Yes
256 256
82.55
81.25
Table 3:
Human evaluation results compared to CogVideo (Hong et al. 2022 on DrawBench and
our test set, and to VDM (Ho et al., 2022) on the 28 examples from their website. The numbers show the percentage of raters that prefer the results of Make-A-Video model.
Comparison
Benchmark
Quality
Faithfulness
Make-A- RakeeAAVVieo (ours) vs. VDM
VDM prompts (28)
84.38
78.13
Make-A A-Video (ours) vs. CogVideo (Chinese)
DrawBench (200)
76.88
73.37
Make-A Video (ours) vs. Cog Video (English)
DrawBench (200)
74.48
68.75
Mide-A-Video (ours) vs. Cog Video (Chinese)
Our Eval. Set (300)
73.44
75.74
Make-A-Video (ours) vs. Cog Video (English)
Our Eval. Set (300)
77.15
71.19
Make- A- Video's s zero-shot performance is much better than GODIVA and NÜWA which are trained on MSR-VTI. We also outperform CogVin Video in both Chinese and English settings. Thus, Make-A- Video has significantly better generalization capabilities than prior work.
Automatic Evaluation on UCF-101. UCF-101 a popular benehmark to evaluate video generation and has been recently used in T2V models. Cog Video performed finetuning of their pretrained model for caasscooniitinal video generation. VDM (Ho el. 2022) performed unconditional video generation and trained from scratch UCF-101. We argue that both settings are not ideal and is not direct evaluation the T2V generation capabilities. Moreover, the FVD evaluation model expects the videos to be 0.5 second (16 frames), which is too short to be used for video generation in practice. Nevertheless, in order to compare to prior work, we conducted evaluation on UCF-101 in both zero-shot and finetuning settings. As shown in Table Make-A- A-dieo''s 's zero-shot performance is already competitive than other approaches that are trained on UCF-101, and much better than Videi which indicates that Make-A- Video can generalize better even such a specific domain. Our finetuning setting achieves state-of-the-at results with a significant reduction in FVD, which suggests that Make-A-Videe A-iieo can generate more coherent videos than prior work.
Human Evaluation. We compare to Videideo (the only public zero-shot T2V generation model) on DrawBench and our test set. We also evaluate on the 28 videos shown on the webpage of VDM (Ho et al. 2022) (which may be biased towards showcasing the model's 's strengths). Since this is a very small test set, we randomly generate videos for each input and perform evaluation 8 times and report the average results. We generate videos at 76 x 256 x 256 resolution for human evalua- tion. For VDM, are shown in Table 3 Make-A-Video achieves much better performance in both video quality and text-video faithfulness in all benchmarks and comparisons. For Cog Video, the results are similar on DrawBench and our evaluation set. Additional experiments combining components of Cog Video & Make-A A- Video in order to measure the efficacy of different components are presented in Sec. [6.] without any cherry-picking. We also evaluate frame interpolation network in comparison to
worth noting that we have achieved significantly better results The results
8

---

Published as a conference paper at ICLR 2023
FILM (Reda et al., 2022). We first generate low frame rate videos (1 FPS) from text prompts in Brachenchhs and our evaluation set, then use each method to upsample to FPS. Raters choose our method for more realistic motion 62% of the time DrawBench. We observe that our method excels when there are large differences between frames where having real-world knowledge of how objects move is crucial.
evaluation set and 54% of the time
4.3 QUALITATIVE RESULTS
Examples of Make-A- generations are shown in Figure ] In this section, we will show T2V generation comparison to Cog Video (Hong al.] 2022) and VDM (Ho et al., (022), and video interpolation comparison to FILM (eeda et al. 2022). variety of other tasks such as image animation. video variation, etc. Due to space constraint, we only show a single example of each. Figure (a) shows the comparison of Make-A- to Cog Video and VDM. Make-A- A-iido can generate richer content with motion consistency and text correspondence. Figure 4 (b) shows an example of image animation where we condition the masked frame interpolation and extrapolation network iF on the image and CLIP image embedding to extrapolate the rest of the video. This allows a user to generate a video using their own image giving them the opportunity to personalize and directly control the generated video. Figure (c) shows a comparison of our approach to FILM (Reda et al. 2022) on the task of interpolation between two images. We achieve this by using the interpolation model that takes the two images as the beginning and end frames and masks 14 frames in between for generation. Our model generates more semantically meaningful interpolation while FILM seems to primarily smoothly transition between frames without semantic real-world understanding of what is moving. Figure (d) shows an example for video variation. We take the average CLIP embedding of all frames from a video as the condition to generate semantically similar video. More video generation examples and applications can be found here: maake-a-vided --iiiogggii
addition, our models can be used for
DISCUSSION
Learning from the world around us is one of the greatest strengths of human intelligence. Just we quickly learn to recognize people, places, things, and actions through observation, generative sys- tems will be more creative and useful if they can mimic the way humans learn. Learning world dy- namics from orders of magnitude more videos using unsupervised learning helps researchers break away from the reliance on labeled data. The presented work has shown how labeled images com- bined effectively with unlabeled video footage can achieve that.
As a next step we plan to address several of the technical limitations. discussed earlier, our approach not learn associations between text and phenomenon that only be inferred in videos. How to incorporate these (c.g., generating a video a person waving their hand left-to-right right-to-left), along with generating longer videos, with multiple scenes and events, depicting more detailed stories, is left for future work.
Also, the lack of standard benchmarks in the field of large-scale generative models makes it difficult for works to compare with each other and measure progress over time. To address this issue, we went beyond what was done in most existing works, with extensive human evaluation, including a comparison to examples shared by authors of existing approaches (or generated using models when publicly released). We hope the community will continue making progress towards better benchmarks for generative models.
As with all large-scale models trained on data from the web, our models have learnt and likely exaggerated social biases, including harmful ones. Our T2I generation model was trained on data that removed NSFW content and toxic words. All our data (image as well videos) publicly available, adding a layer of transparency to our models, and making it possible for the community to reproduce our work.
ACKNOWLEDGMENTS
Mustafa Said Mehmetoglu, Jacob Xu. Katayoun Zand, Jia-Bin-Huang Jiebo Luo, Shelly Sheynin, Angela Fan, Kelly Freed. Thank you for your contributions! Thank you as well to all the people internal to FAIR who helped enable this work by providing extra compute for experimentation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. ICCV, pp. 1728-1738, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Vorbertt-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Bemnr,, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165,, 2020. BtRLhttss:/ http::s//aaxxvv.oooo abs 2005.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204.1444 142/7, 2022.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman, Make-a- scene: -ased text-lo-image generation with human priors, 2022. URL htttttt///aaxxi org/abs/ 2203 13131
Songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. ECCV, 2022.
Deeptha Girish, Vineeta Singh, and Anca Ralescu. Understanding action recognition in still images. pp. 1523-1553 1529, 06 2020. doi: 10.1 10.10099/9R550498.22000000 2020000099
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. NIPS, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, pp. 10696- 10706, 2022.
Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions to videos. In ECCV, pp. 598-613, 2018.
Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Is- abelle Hu, and Devi Parikh. Mugen: playground for video-audio-text multimodal understanding and generation. ECCV, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https: http///wwi.gg////a.a.///.. wwwpooggg/bbsssoe 200p:///.////.//.///// 11239
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. 03458
Fleet. Video diffusion models, 2022. URL htttps::/aaxxv..o//////aaea
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hicrarchical text-to-image synthesis. In CVPR, pp. 7986-7994, 2018.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. 2022. URL
abs/2205. 15869
Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In AAAI, volume 32, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ 1692, 2019a. URL http:://aaxxii..rrr/////////////
10

---

Published as a conference paper at ICLR 2023
Yue Liu, Xin Wang, Yitian Yuan, and Wenwu Zhu. Cross Pross-modal dual learning for sentence-to- video generation. In Proceedings of the 27th ACM International Conference on Multimedia, pp- 1239-1247, 2019b.
Tanya Marwah, Gaurav Mittal, and Vineeth N Balasubramanian. Attentive semantic video genera- tion using captions. In ICCV, pp. 1226-1434, 2017.
Gaurav Mittal, Tanya Marwah, and Vineeth N Balasubramanian. Sync-draw: Automatic video gen- cration using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1096-11 1104, 2017.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 wt..iv:212..07722 10741, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162 -671- PMLR, 2021.
Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Generat- ing videos from captions. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1789-1798, 2017.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pp. 5533-5541, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL httpps://arxivorgg abs/ 2204. 06125
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. arXiv preprint arXiv:2202.0499 04901, 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In JCML, pp. 1060-1069. PMLR, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer: High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022. URL hhttps://arxitt wtt.:oorggg..... 2205. 11487
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory- unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, (28(10):586-266) 2020.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski. Srivatsa R Kundurthy. Katherine Crowson, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.

---

Published as a conference paper at ICLR 2023
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Theo Coombes, Cade Gor- don, Aarush Katta, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5B: laion- 5b: A new era of open large-scale multi-modal datasets. https://laion.al laion- nre -ww.enr--------------------eee-e Sampen-larrgee large sca Setale-me eelel--lllee 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:122..00 2012.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. ICLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https://arxxv. org/ erg/abs/ / 1 1706. 03762
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint vrXiv:2104.14806 2021a.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NÜwa: Visual synthesis pre-training for neural visual world creation, 2021b. URL https://arxiv.org) labs/ bbbs/211..0 12417
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. ECCV, pp. 305-321, 2018.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In CVPR, pp. 5288-5296 2016.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In CVPR. pp. 1316-1324, 2018.
Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. CVPR, pp. 5036-5045, 2022.
Rongtian Ye, Fangyu Liu, and Liqiang Zhang. 3d depthwise convolution: Reducing model parame- ters in 3d vision tasks. In Canadian Conference on Artificial Intelligence, pp. 186-199. Springer, 2019.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 10.04627, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022a. URL URL http::tttt///tt http::////xxxv..////..so wwwpo//////////.o../// 10789.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awart implicit generative adversarial networks. ICLR, 2022b.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistc image synthesis with stacked generative adver- sarial networks. In ICCV, pp. 5907-5915, 2017.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. CVPR, pp. 833-842, 2021.
12

---

Published as a conference paper at ICLR 2023
6 APPENDIX
6.1 DISENTANGLING EFFICACY OF THE T21 AND 12V COMPONENTS
We evaluate two additional baselines to disentangle the contributions of two components of Make- A- A-Video - Lexx-to-Imagge (T2I) and Image-to- Video (I2V). We do by using Make-A- A-Vido''s T2I with Cog Vogeo'ss ' 12V, and analogously Cog Video's ' T2I with Make-A- Video's 12V.
Note that Make-A- Video does not have two explicit T2I and 12V modules. That is, it does not gen- erate an image first and then use it's CLIP embedding to generate the 16 video frames. generates the 16 video frames directly from the image CLIP embedding which is predicted using the prior. As a result, in order to evaluate an ablation like this, we first generate a frame with Cog Vide's''s T2I module, extract the image CLIP embedding from it, and that generate the 16 video frames.
We perform our ablation in a zero-shot setting on the MSR-VTT dataset where we generate 5K samples using each of the approaches. We condition CogVidee English because English prompts performed better in our MSR-VTT evaluation (see Tab.
Please see Tab. 4 for the results of this ablation study. We report FID and CLIPSIM metrics com- puted on static frames generated by the approaches. In addition, to evaluate temporal quality, we perform human evaluation on 200 videos out of the generated 5K. Each video was rated by 5 human evaluators. and we take the majority vote as the final result. We report the percentage of raters that prefer the results of our Make-A- A-Vido model over the two baselines.
We see that Make-A- AVideo is favored across all metrics when compared to two of its CidVideo vari- ants. Human evaluators preferred generations that used Make-A- Videeo's spatial-temporal mecha- nism (12V) over Video's about 2 out of times. Similarly, evaluators preferred generations using Make-A- A-Vde'-Video's T2I module over Cog Video's out of times. We also report Cog Video performance (Cog Video T2I and I2V) reference.
Table 4: Evaluating the contribution of the T2I and I2V components in zero-shot generation
MSR-VTV We report FID and CLIPSIM scores computed on static frames. Quality shows the percentage of human raters that prefer the results of our Make-A- A-Video model over the baselines.
Method
FID (4)
CLIPSIM (1)
Quality (t)
Cog (English)
20.01
0.201
60%
Cog Video T21 Make-A-Vides 12V
18.42
0.251
74%
Make- Video T2I + Vigeideo 12V
14.09
0.302
66%
Make- Video (ours)
13.96
0.305
6.2 ABLATION STUDY
We perform an ablation study on several architecture and training choices. First, we ablate our architecture design. Specifically the contribution of temporal convolutional layers (Sec. 3.2.1 and temporal attention layers (Sec. 3.2.2) Second, we demonstrate the effectiveness of initializing our Text-to- Video model with praitrdined Text-to-Image model weights.
We perform our ablation in zero-shot setting on the MSR-VT VTT dataset where we generated videos for 6K sentences. We report the automatic CLIPSIM metric to evaluate text faithfullness. addi- tion, to evaluate temporal quality, we generate videos for our human evaluation set of 300 prompts and ask raters to select which model's generation higher quality. Each pair of videos was rated by 5 human evaluators, and we take the majority vote as the final result. We report the percentage of raters that prefer the results of our Make-A- A-Video model over the two baselines. Please see Tab. for the results of this ablation study.
In our architecture design ablation we trained two variants: (i) Make-A- Video architecture without temporal attention layers with the spatial attention layers kept is "No Attn", (ii) Make-A-Vide architecture without temporal convolutional layers with the spatial convolutional layers kept as is "No Cony", and, (iii) complete Make-A- Aake-A-Video architecture "Full". All models were trained
13

---

Published as a conference paper at ICLR 2023
Table 5: Ablation study on architecture and training design choices. Results are reported zero- shot generation on MSR-VTT. We report CLIPSIM scores computed on static frames. Quality shows the percentage of human raters that prefer the results of our Make- A-Video model over the baselines.
Method
CLIPSIM (1)
Quality (1)
From Scratch
0.246
63.31 %
No Conv
0.256
52.04%
No Attn
0.257
55.25%
Full
0.258
Table Human evaluation comparing the effects of different components. Results are evaluated on the human evaluation set with 300 prompts. Quality shows the percentage of the human raters who prefer the results of setting B in each comparison.
Comparison
Setting A
Setting B
Quality (t)
16 64 x 64
16 256 x 256
92.48%
16 256 256
76 256 256
68.30%
3
16 x 256 x 256
16 x 768 x 768
60.13%
16 256 256 (static SR)
16 256 x 256
54.25%
5 16 768 x 768 (random noise)
16 768 x 768
50.98%
for 100K iterations. As can seen, both the temporal convolutional layers and temporal attention layers are important to improve video quality and text faithfulness.
In addition to architecture design ablation, we justify our decision to initialize the T2V model with the weights of pre-trained T2I model. We begin by reporting CLIPSIM and subjective quality evaluation metrics on a T2V model trained from scratch 100K iterations "From Scratch". As can be seen in Tab.5) the model initialized with pre- pre-trained T2I weights ("Full")) outperforms the model trained from scratch when trained the same number of iterations. In addition, the model initialized with pre-trained T2I weights achieves the CLIPSIM score of the model trained from scratch after just 50K iterations, demonstrating the acceleration achieved by initializing with the weights of a T2I model.
6.3 EFFECTS OF DIFFERENT COMPONENTS
Our framework consists of several components that are independently trained and sequentially ap- plied during inference. These models include: 1) decoder that generates a video from the image embedding generated from a prior model; 2) interpolation model that improves the frame rate and generates a video of 76 x 64 x 64; a temporal super-resolution model that improves the video resolution by considering temporal information and generates video of 76 x 256 256; 4) a second super-resolution model that is applied independently on each frame with the same sampled frame noise and generates the final video of 76 768 768.
x 64 x 64
We study the contributions of each of these components through human evaluation. The results are shown in Table 6 We have the following observations. First, improving the resolution from 64 to 256 helps boost the video quality significantly (Comparison 1). Second, increasing the frame rate from fps (16 frames) to 19 fps (76 frames) also help to increase the quality quite a bit (Comparison 2). Third, further increasing the resolution from 256 to 768 can still boost the quality (Comparison
3). These three comparisons have demonstrated the effectiveness of our interpolation model, and two super-resolution models in improving the quality of the generated videos.
Furthermore, we also compare our temporal super-resolution model with a static super-resolution model. The latter is applied independently on each frame without considering temporal information. 4), the temporal super- super-resolution model shows better video quality
As shown in Tablc(((Compariss compared to the static super-resolution model. This justifies model at the 256 resolution level. Another comparison we have done to validate the effect of a fixed frame noise for the second super-resolution model. As shown in the last row of using fixed noise has a slightly better result compared to using random noise for each frame.
of temporal super-resolution
14

---

Published as a conference paper at ICLR 2023
(a) T2V Generation: comparison between VDM (top). CogVideo (mid), Ours (bottom) input "Busy freeway at night".
(b) Image Animation: leftmost shows the input image, and we animated it to be video.
(c) Image Interpolation: given two images (leftmost and rightmost), we interpolate frames. Comparing FILM (left) and Ours (right).
(d) Video Variation: we can generate a new video (bottom) variant the original video (top).
Figure 4: Qualitative results for various comparisons and applications.
15

---

Published as a conference paper at ICLR 2023
Table 7: Hyperparameters for the models
D: D'
iF
SR ; SR
SRA
Diffusion steps
1000
1000
1000
1000
1000
Noise schedule
cosine
cosine
cosine
cosine
linear
Objective
Sstert
E 100
Istart
Istart
Sampling steps Sampling variance method
64
50
50
50
analytic
DDPM
DDPM
DDIM
DDIM
Crop fraction
3 730M
Model size
1.3B
2.2B ; 3.1B 512
3.1B
1B 1.4B
Channels
512
320
320
Depth
3
3
3
3
Channels multiple
1,2,3,4
1,2,3,4
1,1,2,2,4,
1,2,3,4
Heads channels
64
64
Attention resolution
32, 16,8
32, 16,8
Text encoder context
128
Text encoder width
2048
Text encoder depth
24
Text encoder heads
32
Dropout
0.1
0.1
0.1
0.1
Weight decay
6.0c - 2
Batch size
4096
2048 512
512
1024 : 256
1024
Iterations
3M
2M 200K
100K
700K 150K
700K
Learning rate
1.1e
6.0e
6.0e 5
1.2e
4 6.0e
1.2e
Adam B8
0.96
0.999
0.999
0.999
0.999
Adam
1.0e
1.0e -
1.0e-
1.0c 8
1.0e
EMA decay
0.9999
0.9999
0.9999
0.9999
0.9999
Model Parameters (B)
1.3
3.1
3.1
1.4
0.7
16


Reviews:
[{"review_id": "sFb_IPJF5D5", "reviewer": "Reviewer_SMdY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Make-A-Video, a text-video generation approach trained based on a text-image model. The core idea of Make-A-Video is to take use of learned text-vision correlation from well-trained text-image model to accelerate the learning of text-video generation. It also claims that no paired text-video data is required for video generation. The quantitative results on MSR-VTT and UCF-101 are state-of-the-art and better than CogVideo and VDM, which are two main baselines to compare. More qualitative examples are provided to elaborate the motion consistency and richer content in the generated videos from Make-A-video.", "summary_of_the_review": "Overall, the paper introduces an idea to break the dependency on text-video pairs for text-video generation and achieves state-of-the-art performance. However, some training and inference details are missing; more analysis on the components of the model internally should be provided to elaborate why the model works, as it seems all quantitative and qualitative results are compared with other approaches but internal analysis is missing.", "strength_and_weaknesses": "Strength:\n1. Novel idea on training text-image and image-video separately to bypass paired text-video data;\n2. Well-motivated to use text-image model for text-video generation;\n3. State-of-the-art results;\n\nWeaknesses:\n1. The main idea is novel to combine text-image generation and image-video reconstruction; however, text-image pipeline is exactly same as previous work, and it's hard to say the components for image-video generation are novel as they are commonly used.\n2. Results and analysis are not sufficient. The results are state-of-the-art but it's not clear where the improvements come from. At least some quantitative results on ablation internally should be provided to make the model less blackbox. For example, how much improvement does FRAME INTERPOLATION NETWORK bring in terms of the evaluation on MSR-VTT and UCF-101, or more qualitative examples to show the difference in the generated videos w/ and w/o the module. Otherwise it's really hard to understand how and why the model works.\n3. Some claims are unclear and not well-supported: In Previous Work, \"Second, we fine-tune the T2I model for video generation, gaining the advantage of adapting the model weights effectively, compared to freezing the weights as in CogVideo\". This is not completely true: first, only some components of T2I model are tuned for image-video generation, starting from Dt; second, some more results should be provided to prove that in Make-A-video, fine-tuning the weights in T2I is better than freezing them, other than comparing with CogVideo.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jOQx8sC6N4q", "reviewer": "Reviewer_hyJV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Make-A-Video, an approach for generating short video clips conditioned on a given open domain text. The model does not need any text-video pairs and instead relies on learning dynamics from unlabeled videos and applying them to existing text to image systems. The approach also applies multiple levels of super resolution, both temporal and spatial, to increase the resolution of generated videos. ", "summary_of_the_review": "Overall, while the results of the paper is impressive, it lacks substantial details which makes it close to impossible to reproduce, no analysis to judge the limitations of the approach and exaggerated quantitative numbers. Therefore, the paper should be improved, scientifically, to be acceptable at a major scientific conference, imho. Particularly, more details, more analysis and fairer comparison. ", "strength_and_weaknesses": "===== Strengths\n+ The main idea behind the paper is well motivated and an interesting one. There are no massive dataset of labeled video datasets (yet!) while it is relatively easier to collect large amount of raw videos. \n+ The paper is easy to read and understand. The notations are clear to follow which makes it easier to follow what it going on. \n+ There are many qualitative examples on the appended website: https://gen-videos.github.io/ This is always a big plus for video papers.\n+ While the quantitative benchmarks for video generation are quite weak, authors did their best to compare with previous work in a fair setting.\n\n===== Weaknesses\n- Missing details. Unfortunately, there are *many* missing details in the paper which substantially reduces its impact on the field. The biggest one imho is the loss. While the system consists of many smaller parts, trained separately, it is not clearly what the loss for any of these components are. In fact, the term \"loss\" or \"objective\" which the authors heavily relied on to optimize, does not appear in the paper even once! This is really interesting for a deep learning paper. While I understand that creating such systems (unfortunately) is limited to big industrial labs, reproducibility still should be a big concern. \n\n- Quantitative results are exaggerated. It is no secret that the benchmarks for video generation are still quite weak. The metrics (e.g. FVD) are inconclusive and unrepresentative of the differences. While I appreciate the authors effort to provide a large set of videos in their website, as well as a short human eval, it seems the quantitative results in Table 1 and 2 are exaggerated and in absence of evaluation details it is hard to check that clearly.  Particularly, FVD is sensitive to resolution but this paper compares different resolution without mentioning how.  but how? Higher resolution usually results in better FVD (because of I3D resolution)  and therefore comparing Make-A-Video at 256x256 to e.g. CogVideo at 160x160 or VMD at 64x64 is not a fair comparison. Also for CogVideo, it is not clear where the videos came from, because the official published version is only in Chinese and as of writing this review, the official github page clearly mentions that \"Currently only Chinese input is supported.\" To make things even worse, the provided CogVideo videos in your website,  are significantly worse compared to CogVideo's official page. Again, missing details here severely affects the quality of the paper.\n\n- Lack of analysis. The main idea of learning dynamics from a large corpus of videos and then applying them to generated images is interesting and novel to the best of my knowledge. However, like any other system with multiple components, it has clear down sides compare to an e2e system. First of all, the dataset of videos that the authors used (WebVid) has labels. The paper clearly mentions that it did not use these labels but this raises the question of why? The paper does not provide any analysis on how using more videos improves the system and if 10M labeled videos was enough to learn dynamics then the motivation of the paper is questionable. Second, the paper does not provide any analysis on the weaknesses of the proposed method. I would assume a system that learns the dynamics separately will have weak correlation between dynamics and the terms to describe them (e.g. walking jumping dancing etc), is this the case? if not why not? Where does the alignment come from? Or as another example, how long the videos can be before becoming incoherent? The website includes a few examples of \"long\" videos however it's not clear how far the model can be pushed and what its limitations are. In absence of any such analysis and the fact the code/model is behind tall walls, it is hard to answer such questions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bocdoSA_YnE", "reviewer": "Reviewer_X9uC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on text-to-video generation, but without having text-video pairs. To achieve this, the proposed method relies on a pre-trained text-to-image generation model, and then further extend it to generate a video, semantically aligned with the given text.", "summary_of_the_review": "Although the proposed can produce a good results without using text-video pairs, it relies on many previous works and has limited differences between them. ", "strength_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow.\n2. The proposed method shows better quantitatively results, compared to the baselines, and also have a good human evaluation.\n3. The training of proposed method does not need text-video pairs, which can avoid the difficulty to have a text-video dataset.\n\nWeaknesses:\n1. Basically, the proposed method relies on many previous works: (1) the T2I model used in the proposed method is DALL-2, (2) the proposed Pseudo-3D convolutional layers is based on separable convolutions (Chollet, 2017), actually a similar idea to separate spatial and channel information to reduce computational cost have been widely adopted in different areas, and (3) as mentioned by authors, factorized space-time attention layers have also been used in VDM (Ho et al., 2022) and CogVideo (Hong et al., 2022). \n2. Compared to CogVideo, the better performance achieved by the proposed method might benefit from a powerful pretrained T2I model (i.e., DALLE-2). If authors use CogView as the T2I model, same as CogVideo, could the proposed method still have a better performance?\n3. I am confused about the quality of synthetic video results, as authors only show a small number of frames for each video results, and the frames in the same video only show limited differences between each other. Also, current text-to-image generation methods, such as DALLE-2, can produce loads of images with some differences, and combine these results together, we can also observe some meaningful story from them. \n4. Authors claim their method accelerates training of the T2V model, but there are no experiments to support this claim.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WNcdayfs1x", "reviewer": "Reviewer_5k2D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a text2video method, which builds on image2video model with spatial-temporal modules.\nThe method is straight-forward and the results are appealing.", "summary_of_the_review": "This work presents a new method for text to video generation, which is a really important problem. The method is effective and the results are compelling. In general, I like this paper and recommend for acceptance so far.\n\nThe main concerns are listed in Weakness above. Although some of the concerns are not specific for this paper but for this research domain, which lacks standard benchmark, It will be great to see the opinions and discussions for this problem in this paper.\n\nMany previous methods are more likely to show off the results rather than make it as a research problem. This paper tried to use open-source datasets, which has started to consider the reproducibility for the community. I really appreciate this awareness.\n\n", "strength_and_weaknesses": "**Strength**\n\n1) This paper is well-written and easy to follow.\n\n2) The method is simple but effective.\n\n3) The results are compelling.\n\n4) This work uses solely open-source datasets, making it easier to reproduce.\n\n**Weakness**\n\n1) A relatively comprehensive benchmarks. The automatic quantitative evaluation is only done in MSR-VTT and UCF-101. The class and coverage is somewhat limited. Since the results on general situation is more for fun rather than the practical application, some domain-specific evaluation is wanted, to evaluate the practical application ability of the proposed models. For example, text2face, text2human or text2scene, which could has broad application scenarios. Thus, a more comprehensive benchmarks, especially for domain-specific evaluations, is wanted.\n\n2) A absolutely fair comparison to SOTA. For the results on Table 1 and 2, the resolution, number of parameters are different for these compared models. It makes the comparison with SOTA of IS and FID seem to be weak, since other factors are neglected. A absolutely fair comparison benchmark is also wanted.\n\n3) The precise control of action in sequence. It will be better if authors can show the results for text2face/text2human, which is more challenging than general situations, since the more subtle action should be generated. Could the model control the action precisely, for example, with the text: a young man first smiles, then nod his head and finally turns left to speak to others. Will the subtle action and the sequential relation be faithfully maintained?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster