Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
Authors: Sihyun Yu
Weili Nie
De-An Huang
Boyi Li
Jinwoo Shin
Anima Anandkumar
Venue: ICLR 2024 poster (2024)
Keywords: video generation
diffusion models
Primary area: generative models
TL;DR: We propose a memory- and computation-efficient latent video diffusion model that leverages visual knowledge present in pretrained image diffusion models.

Abstract: Video diffusion models have recently made great progress in generation quality, but are still limited by the high memory and computational requirements. This is because current video diffusion models often attempt to process high-dimensional videos directly. To tackle this issue, we propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation. Specifically, we propose an autoencoder that succinctly encodes a video as a combination of a content frame (like an image) and a low-dimensional motion latent representation. The former represents the common content, and the latter represents the underlying motion in the video, respectively. We generate the content frame by fine-tuning a pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffusion model. A key innovation here is the design of a compact latent space that can directly utilizes a pretrained image diffusion model, which has not been done in previous latent video diffusion models. This leads to considerably better quality generation and reduced computational costs. For instance, CMD can sample a video 7.7$\times$ faster than prior approaches by generating a video of 512$\times$1024 resolution and length 16 in 3.1 seconds. Moreover, CMD achieves an FVD score of 238.3 on WebVid-10M, 18.5% better than the previous state-of-the-art of 292.4.

Published as a conference paper at ICLR 2024
EFFICIENT VIDEO DIFFUSION MODELS VIA CONTENT-FRAME MOTION-LATENT DECOMPOSITION
Sihyun Yu Weili Nie
De-An Huang
Boyi Li Jinwoo Shin'
Anima Anandkumar"
KAIST 2NVIDIA Corporation
UC Berkeley *Caltech
ABSTRACT
Video diffusion models have recently made great progress in generation quality, but are still limited by the high memory and computational requirements. This is because current video diffusion models often attempt to process high-dimensional videos directly. To tackle this issue, we propose content-motion latent diffusion model (CMD), novel, efficient extension of pretrained image diffusion mod- els for video generation. Specifically, propose an autoencoder that succinctly encodes a video as a combination dimensional motion latent representation. The former represents the common con- tent, and the latter represents the underlying motion in the video, respectively. We generate the content frame by fine-tuning pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffu- sion model. A key innovation here is the design of a compact latent space that can directly and efficiently utilize pretrained image model, which has not been done in previous latent video diffusion models. This leads considerably better qual- ity generation and reduced computational costs. For instance, CMD can sample a video 7.7 faster than prior approaches by generating a video of 512x 1024 res- olution and length 16 in 3.1 seconds. Moreover, CMD achieves an FVD score of 238.3 on WebVid-10M, 18.5% % better than the previous state-of-the-art of 292.4.
content frame (like an image) and a low-
INTRODUCTION
Recently, deep generative models have exhib- ited remarkable success in synthesizing photo- realistic and high-resolution images using dif- fusion models (DMs) (Ho et al., 2020; Nichol
360
450
LVDM
Mode Scape
400
& Dhariwal, 2021; Song et al., 2021b; Karras et al., 2022) and even achieving promising re- sults difficult text-to-image (T2I) generation (Rombach et al., 2022; Saharia et al., 2022; Bal- aji et al., 2022). Inspired by the success in the image domain, several works have focused solving a considerably more challenging task of video generation. In particular, they have at- tempted to design DMs specialized for videos and shown reasonable video generation qual- ity (Ho et al., 2022b; Yang et al., 2022; Yu et al., al., 2023). Nevertheless,
350
300
250
CMD (urrs)
200
6.5
7.5
Memary Consumptin (GBB
Figure 1: Existing (text-to-)video diffusion mod- els extended from image diffusion models often suffer from computation and memory inefficiency due to extremely high-dimensionalityy and tempo- ral redundancy of video frames. Compared with these methods, CMD requires ~16.7x less com- putation with only ~66 % GPU memory usage in sampling. while achieving significantly bet- ter video generation quality. FLOPs and memory consumption are measured with a single NVIDIA A100 40GB GPU to generate a single video of resolution 512x 1024 and length 16.
2023b; Blattmann unlike the image domain, there is still con- siderable gap in video quality between gener- ated and real-world videos. This is mainly due to the difficulty of collecting large training dataset of high-quality videos (Ho et al., 2022b; Ge et al., 2023) and the high dimensionality of video data cubic arrays, leading to heavy memory and computational burden (He et al., 2022; Yu et al., 2023b).
Work done during internship NVIDIA. Project page: httph:t/stttstteeeee MM//MM).

---

Published as a conference paper at ICLR 2024
Conventional appreaches
Our approsch
Temporal layars
Temporal Layers
Cantant diffusion [Fine- -uned from image dittusion
Pretrained image iffusion model
Metion
Content
Temporal Layers
DVT blook
Temporal layers
Video enooder
video
enczdding
generation Mutiiin representaal gainer
Figure 2: Comparison with (a) the conventional extension of image diffusion models for video gen- eration and (b) our CMD. We mark the newly added parameters as blue. Unlike common approaches that directly add temporal layers in pretrained image diffusion models for extension, CMD encodes each video age diffusion model (e.g., Stable Diffusion (Rombach et al., 2022)) for content frame generation and trains new lightweight diffusion model (e.g., DiT (Peebles & Xie, 2023)) for motion generation.
image-like content frame and motion latents, and then fine-tunes pretrained im-
To tackle the data collection issue, several video DM approaches leverage pretrained image DMs for video generation (He et al., 2022; Singer et al., 2023; Luo al., 2023; Ge et al., 2023). Due to the rich visual knowledge already learned from image datasets, the use of image DMs in video generation leads to better generation quality and faster training convergence compared to training video DM from scratch (An et al., 2023; Blattmann et al., 2023). However, since these video models directly generate high-dimensional videos as cubic arrays, they still entail high memory consumption and computational costs, especially for high-resolution and long videos.
Another line of video DM approaches focuses on alleviating memory and computational inefficiency by first projecting the video into a low-dimensional latent space and then training a DM in the latent space (Yu et al., 2023b). In particular, these approaches consider both the temporal coherency of videos well as frame-wise compression in video encoding to obtain the maximum efficiency. However, such latent video DMs are only trained on a limited amount of video data and do not incorporate pretrained image models, which limits their video generation quality.
Our approach. We address the aforementioned shortcomings by introducing content-motion la- tent diffusion model (CMD), memory- and computation-effiiien latent video DM that leverages visual knowledge present in pretrained image DMs. CMD is two-stage framework that first com- presses videos succinct latent space and then learns the video distribution in this latent space. A key difference compared to existing latent vidco DMs is the design of a latent space that directly incorporates a pretrained image DM. See Figure 2 for an illustration.
In the first stage, we learn low- cow-dimensional latent decomposition into a content frame (like an image) and latent motion representation through an autoencoder. Here, we design the content frame as weighted sum of all frames in video, where the weights are learned to represent the relative importance of each frame. In the second stage, we model the content frame distribution by fine- tuning pretrained image DM without adding any new parameters. It allows CMD to leverage the rich visual knowledge in pretrained image DMS. In addition, we design a new lightweight DM to generate motion latent representation conditioned us having to deal directly with video arrays, and thus, one can achieve significantly better memory and computation efficiency than prior video DM approaches built on pretrained image DMs.
the given content frame. Such designs avoid
We highlight the main contributions of this paper below:
We propose an efficient latent video DM, termed content- motienn latent diffusion model (CMD). We validate the effectiveness of CMD on popular video generation benchmarks, including UCF- 101 (Soomro et al., 2012) and Web Vid-10M (Bain et al., 2021). For instance, measured with FVD (Unterthiner al. 2018 8; lower is better), our method achieves 238.3 in text-to-video (T2V) generation on WebVid-108 18.5% better than the prior state-of-the-ee 292.4.
We show the memory and computation efficiency of CMD. For instance, to generate single video of resolution 512x 1024 and length 16, CMD only requires 5,56GB memory and 46.83 TFLOPs, while recent Modelscope (Wang et al., 2023a) requires 8.51GB memory and 938.9 TFLOPs, sig- nificantly larger than the requirements of CMD (see Figure l).
2

---

Published as a conference paper at ICLR 2024
"Fireworks"
"A video of the Earth rotating in space"
hutterste
Mutterste
ttersto
"A Teddy Bear skating in Times Square"
"A clownfish swimming through coral reef"
Figure 3: 512 x 1024 resolution, 16-frame text-to-video generation results from our CMD. We visualize video frames with stride of 5. We provide more examples with different text prompts in Appendix C, as well as their illustrations as video file formats in the supplementary material.
RELATED WORK
In this section, we provide a brief overview of some of the important relevant literature. For more extensive discussion with detailed explanation of other methods, see Appendix A.
Latent diffusion models. Diffusion models have suffered from memory and computation ineff- ciency because they require large number of iterations in high-dimensional input space for sam- pling (Ho et al., 2020). To mitigate this issue, several works have considered training diffusion models in low-dimensional latent space, learned by autoencoder (Zeng et al., 2022; Xu et al., 2023; Ben Melech Stan et al., 2023). In particular, this approach has shown remarkable success in the image domain (Rombach al., 2022) greatly improve efficiency well achieve high- quality synthesis results conditioned at a complex text prompt. Similarly, our work aims to design latent diffusion model for videos (He et al., 2022; Yu al., 2023b) alleviate the inefficiencies.
Video generation. Numerous works have actively focused on solving the challenging problem of video synthesis. Previously, generative adversarial network (GAN; Goodfellow al. 2014) based approaches (Gordon & Parde, 2021; Tian et al., 2021; Fox et al., 2021; Munoz et al., 2021; Yu et al., 2022: Skorokhodov et al., 2022; Singer et al., 2023) were proposed achieve the goal, mostly by extending popular image GAN architectures (Karras et al., 2020). Recently, there have been several works that encode videos as sequences of discrete tokens (van den Oord al., 2017), where they either generate tokens in an autoregressive manner (Kalchbrenner et al., 2017; Weissenborn et al., 2020; Rakhimov et al., 2020; Yan et al., 2021: Ge et al., 2022) or non-autoregressive manner (Yu et al., 2023a). In addition, with the success of diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021) in image generation, recent methods exploit diffusion models for videos (Ho et al., 2022b; Harvey et al., 2022; Yang et al., 2022; Höppe et al., 2022; Singer al., 2023; Lu et al., 2023), achieving promising results in modeling complex video distribution. Inspired by their success, we also aim build video diffusion model achieve better video synthesis quality.
3

---

Published as a conference paper at ICLR 2024
Text-to-video (T2V) generation. Following the success of text-to-image (T2I) generation (Rom- bach et al., 2022; Saharia et al., 2022: Balaji et al., 2022), several works have attempted to solve a more challenging task of T2V generation. The main challenge of T2V generation is to resolve the tremendous training costs of diffusion models and the difficulty in collecting large-scale and high- quality video data. Predominant approaches (Ho al., 2022a; Wang al., 2023b; An et al., 2023; Blattmann et al., 2023; Ge et al., 2023; He et al., 2022: Singer et al., 2023) have achieved this by fine-tuning pretrained T2I diffusion models by adding temporal layers (e.g., temporal attention and 3D convolution layers) to the 2D U-Net architecture (Saharia et al., 2022). However, they suffer from high memory consumption and computational costs due to the unfavorable increase of input dimension in high-resolution and long videos. To tackle this issue, few recent works have focused on alternative extension strategies that avoid dealing with entire raw video frames directly (Luo et al., 2023). Specifically, they achieve it by proposing frame-by-frame generation with an addi- tional lightweight diffusion model. However, our extension is based on a latent diffusion model approach to encoding vidcos content frames and motion latent representation to reduce the input dimension and learn video diffusion models on such compact latent representation.
3 CMD: CONTENT-MOTION LATENT DIFFUSION MODEL
Consider a condition- pair dataset D. where each sample (c, x E D drawn from an unknown data distribution Pdata (x c). Here, each e denotes a condition (e.g., video class text description) of the corresponding x1:L, and each xI- == (x',..., x²) is a video clip of length L >>> 1 with resolution H x W, i.e., x' e &Cxxxx with channel size C. Using D, We aim to learn a conditional model distribution (xoda(xx): to match the data distribution Pastaa (x (c).
Our main idea is encode each video into an "image-like like" content frame and succinct motion latent representation, where pretrained image diffusion models can be used to generate content frames due to the similarity between natural images and content frames. By doing so, rich visual knowledge learned from image data is leveraged for video synthesis, leading to better generation quality as well as reduced training costs. Given content frames, the video generation task thus reduces to designing a motion diffusion model to generate much lower-dimensional motion latent representation.
In the of this section, we explain our content-motion latent diffusion model (CMD) in detail. In Section 3.1, we provide an overview of diffusion models. In Section 3.2, we describe our video encoding scheme and design choices of diffusion models for video generation.
3.1 DIFFUSION MODELS
The main concept of diffusion models is to learn the target distribution Pdata Pdata (x) via a gradual denois- ing process from Gaussian distribution (O , Ix) to (x). Specifically, diffusion models learn reverse process p(x,-1|x; of the pre-defined forward process g(x; Xx) that gradually adds the Gaussian noise starting from Paau (x) for 1 S S T with a fixed T > 0. Here, for Xg ~ Paag(()) q(x, (x+ can be formalized as q(xx/xx1)) = >xxx 0xxo, Ix), where o, and ag == - of pre-defined hyperparameters with 0 < 01 < < 0r-1 < o = 1. If T is sufficiently large, the reverse process p(x.-1 1 (x.) can be also formalized the following Gaussian distribution:
o² colx, 1)), oiTx))
p(x, xx) = No (3--1:
(1)
VI
where ā. = IT' (1-o of) for 1 < T. Here, (x,, E) can be trained as a denoising autoencoder parameterized by θ using the c-prediction objective with a noise e ~ ~Nx0.. Ix) (Ho et al., 2020):
Exp.e,t ||E - c (xe, 1 where x; = Va + v1 a.E.
As the sampling process of diffusion models usually requires a large number of network evaluations p(x)-1[x:) (e.g., 1,000 in DDPM; Ho et al. 2020), their generation cost becomes especially high if one defines diffusion models in the high-dimensional data space. To mitigate this issue, several works have proposed latent diffusion models (Rombach et al., 2022; He et al., 2022): they train the diffusion model in -iwdeinenioonal latent space that encodes the data, thus reducing the computa- tion and memory cost. Inspired by their success, our work follows a similar idea of latent diffusion models to improve both training and sampling efficiency for video synthesis.
4

---

Published as a conference paper at ICLR 2024
(N) Encoder
Contant Trame
Motian latent rupres
(b) Decoder
Vidco network
(e.g. VVVTI
Sum.
Lin,
Average
Average
proj
Repeat
Repeat
Repeat
Video network
(e.g.. ViVIT)
Lin, proj.
Lin, proj
Lin, proj
Figure 4: Illustration of our autoencoder. Encoder: We compute relative importance all frames (blue) for content frame and motion latent representation. Decoder: Using the content frame and motion latent representation, we construct a cubic tensor for video network to reconstruct the video.
3.2 EFFICIENT EXTENSION OF IMAGE DIFFUSION MODELS FOR VIDEOS
CMD consists of an autoencoder and two latent diffusion models. First, we train an autoencoder that encodes a video x I:L. as a single content frame X and low-dimensional motion latent representation z. After that, learning the target distribution Paaa (x (x=:1c) becomes to learn the following distribution: p(x, 2 c) p(z) X, c)p(X(c), c). We model each distribution through two diffusion models, where we utilize a pretrained image diffusion model for learning the content frame distribution p(x/c).
Autoencoder. We train our autoencoder using a simple reconstruction objective (e.g. & loss) to encode video input xL:L We provide illustration of the encoder and decoder in Figure 4.
Our encoder f& consists of a base network f and two heads f for computing X, 2 (respec- tively) with parameterization Φ = (ΦB. Φ1, Φas). Here, the base network fęn RxXLx Hx.xxxxW CC' xL xw" (with fØs f (x u) maps video xI: to hidden feature with channel size C², where we adopt video transformer (e.g., ViViT; Arnab et al. 2021) as føg- Next, the head Se, pCCx xxxxxxx'xx xw' + RCx returns relative importance among video frames x² ,.....,x" x compute the content frame x. Specifically, compute using Jø, as:
Σ xx' (Jor t=1
x
where @ denotes an element-wise product and o(-) is a softmax function across the temporal axis. Consequently, the content frame & has the same dimension with each frame and lies in the space of interpolating consecutive frames, thus looking very similar to them (see Figure/ 7).
For motion latent representation z, we design it as concatenation of two latents, i.e., (Zxx Zy)
with 2x E DxLX and zy IR Dxx'' where 2x, Zy are computed from u using for as follows:
(Zz, zy) = (f (ūx). f (ūy)).
(4)
Here, üL E IR C'LH'' x-axis and y-axis, respectively, and føx is a 1x convolutional layer that maps an input tensor from channel size C' to D. Such -D-projectiion-asee motion encoding is motivated by recent triplane video encoding (Kim et al., 2022; Yu et al., 2023b) that project videos to each x, y, axis.
uy E C' two projected tensors of by simply averaging across
Similarly, design decoder network 9y two embedding layers 9 for (respectively) and a video network Iv' that returns the reconstruction of I!: from the outputs of 9vr 9 Specifically, I Iwn maps X, z to have the same channel size C":
= -ep, C'''''
v = Iepas (zx) E BC' X''H' v² = Iep (zz) E C''WXX'' (5)
where we denote y² = [uhuol v* = T [uy
ynd with oh Vihn Di E RC for LE [1, L], h E H"), w E [1, W'). After that, compute the input of video network Jøg. denoted by (Utho) RC' xLxX'x''xx xW' by taking the sum of the corresponding vectors of each vt, vX, vy namely:
Vshw = vi. + V + V/e for 1 L,
H', 1 w S W
(6)
and then v is passed to 9wn CX + xxXXXxxW to reconstruct the input video x 1: For J the same video transformer architecture føg-
5

---

Published as a conference paper at ICLR 2024
Content frame diffusion model. Recall that the content frame x is computed weighted sum of video frames x ,...,,''' x
and thus it resembles natural images. Hence, for training the content frame diffusion model to learn p(x|c), we directly fine-tune the pretrained image diffusion model c0, (Xe, c, i) without adding additional parameters. In particular, we use condition-content-rrmm pairs (c, x) computed from the dataset D and use the denoising objective for fine-tuning:
Exo,e, [Ile - Eie (X,, C, 0)1 where &, = vā, + VI OiE.
Note that this fine-tuning is -effcietefficient since it does not increase input dimension, and it can be trained efficiently due to the small gap between content frames and natural images.
Motion diffusion model. To learn the conditional distribution p(z) &, c). we train a lightweight dif- fusion model co,, (ze, C, X, t). For the network architecture, we exploit DiT (Peebles & Xie, 2023), recently proposed Vision Transformer (ViT) backbone (Dosovitskiy et al., 2020) for diffusion mod- els, due to its better performance and efficiency. Accordingly, for denoising target Z₁ we pass it to the model as a sequence of patch embeddings. Next, for an input condition c, we follow the same conditioning scheme of the original DiT that passes through the AdaIN layers (Huang & Belongie, 2017). For the conditioning content frame x, rather than passing it through the AdalN layers, we feed as input-level patch embeddings like z, to provide "dense conditions" to the model for predicting motion latent representation z (see Figure 2). Using these inputs, we train the model via the denoising objective:
Exg.e,t [||e 6 (Ze, c, &, 1 where Z, Vāizo + /1 dGE.
(8)
We observe that a lightweight model can quickly converge to well-predicting motion latent repre- sentation z, mainly due two factors: (a) the rich information provided by the conditions (c, (c, and (b) the low dimensionality of motion latent representation z. Moreover, one can use a larger patch size for x (condition) than z (prediction target) reduce the total sequence length of input patches to the DiT network, thus further decreasing the computational cost (see Section 4.3).
EXPERIMENTS
In Section 4.1, we provide setups for our experiments. In Section 4.2, we present the main results, including qualitative results of visualizing generated videos. Finally, in Section 4.3, we conduct extensive analysis validate the effect of each component as well as to show the efficiency of CMD) in various aspects, compared with previous text-to-video generation methods.
4.1 SETUPS
Datasets. We mainly consider UCF-101 (Soomro al., 2012) and Web 10M (Bain et al., 2021) for the evaluation. We also use MSR-VTT (Xu et al., 2016) for a zero-shot evaluation of the text-to- video models. For model training, we use only train split and exclude test (or validation) sets for all datasets. We provide more details, including how they are preprocessed in Appendix B.
Baselines. For class-conditional soo-z generation on UCF-101, we consider recent DI- GAN (Yu et al., 2022), TATS (Ge et al., 2022), Cigeideo (Hong et al., 2023), Make-A-Vides (Singer al., 2023), and MAGVIT (Yu et al., 2023a) as baselines. For zero-shot evaluations, we compare with recent VogVideo, LVDM (He et al., 2022). ModelScope (Wang et al., 2023a), VideoLDM (Blattmann et al., 2023), VideoFactory (Wang et al., 2023b), PYoCo (Ge et al., 2023), GODIVA (Wu et al., 2021), and NUWA (Wu et al., 2022). See Appendix B.2 for more details.
Training details. In experiments, videos are clipped to 16 frames for both training and evalua- tion. For video autoencoder, we use TimeSFormer (Bertasius et al., 2021) as backbone. For the content frame diffusion model, we use pretrained Stable Diffusion (SD) 1.5 and .bbse (Rombach et al., 2022), where each video frame is first encoded by SD image autoencoder into a latent frame with an 8x downsampling ratio and output channel size C = 4. For the motion diffusion model, we use DiT-L/2 (for UCF-101) and DIT-XL/2 (for Web 10M) in the original DiT paper (Peebles & Xie, 2023), where "L" and "XL" specify the model sizes and "2" denotes patch size of 2x2 when converting input into sequence of patches. We provide all other details in Appendix B.3.

---

Published as a conference paper at ICLR 2024
Table 1: Class-conditional video generation on UCF-101. # denotes the model also uses the test split for training. + indicates lower val- ues are better. Bolds indicate the best results, and we mark our method by blue. We mark the method by if the score is evaluated with 10,000 real data and generated videos, other- wise we use 2,048 videos. For zero-shot setup, report the dataset size used for training.
Table 2:
T2V generation on MSR-VTT. indi- cates higher scores are better. Bolds indicate the best results, and we mark our method by blue, We report the dataset size. denotes LAION- 5B (Schuhmann et al., 2022) is jointly used.
Method
Zero-shot
CLIPSIM
GODIVA NÜWA
No
0.2402
No
0.2409
VideoFactory
Yes (130M)
0,3005
Method
Zero-shot
FVD 4
Make-A-Video
Yes (100M)
0.3049
DIGAN
No
465+112
CogVideo
Yes (5,4M)
0.2631
TATS
No
332+18
LVDM
Yes (10.7M)
0.2381
CogVideo
No
305
VideoL.MM
Yes (10.7M)
0.2929
VidcoFusion
No
173
ModelScope
Yes (10.7M)
0.2930
CMD (Ours)
No
10749
CMD (Ours)
Yes (10.7M)
0.2894
Make-A-Video*
No
367
Table T2V generation on WebVid-10ML 4
MAGVIT*
No
76.12
and indicate lower and higher scores are bet- ter, respectively. Bolds indicate the best results, and we mark method by blue. cfg denotes classifier-free guidance scale.
CMD (Ours)*
No
7312
VideoFactory
Yes (130M)
410
PYoCo
Yes (22.5M)
355
Video
Yes (5.4M)
702
Method
FVD
CLIPSIM 1
LVDM
Yes (10.7M)
642
ModelScope
Yes (10.7M)
640
LVDM
455.5
0,2751
VidcoLDM
Yes (10.7M)
551
ModelScope
414.1
0.3000
VideoGen
Yes (10.7M)
554
VideoFactory
292.4
0.3070
CMD (Ours)
Yes (10.7M)
504
CMD (Ours); efg=9.0
238.3
0.3020
Metrics. Following the experimental setup in recent representative video generation literature (Sko- rokhodov et al., 2022; Yu et al., 2023a), we mainly use Fréchet video distance (FVD; Unterthiner et al. 2018, lower is better) for evaluation. To measure text-video alignment, we additionally mea- sure CLIPSIM (Wu et al. 2021, higher is better) and compare the values with the baselines. We provide more details of evaluation metrics and how they computed in Appendix B.4
4.2 MAIN RESULTS
Qualitative results. We visualize several -ix-eto-vidoo generation results from CMD in Figure 3. As shown in this figure, generated videos contain the detailed motion and contents provided by text prompts and achieve great temporal coherency, leading to a smooth video transition. In particular, the background is preserved well between different video frames in the generated video with the prompt. For instance, "A Teddy Bear Skating in Times Square" maintains details of Times Square well across different video frames. Note that each frame has resolution of 512x 1024, where achieve such a high-resolution video generation without requiring any spatiotemporal upsamplers. We provide more qualitative results with other text prompts in Appendix C.
Quantitative results. Table 1 provides the non-zero-shot generation result on UCF-101 by training all models from scratch UCF-101 (including the content frame diffusion model). As shown in this table, CMD outperforms all other video generation methods, indicating our framework design itself an effective video generation method irrespective of the exploitation of pretrained image diffusion models. Moreover, we consider text-to-vides vi-eo generation by training CMD on VebVid-10N with the pretrained SD backbone fine-tuned for content frame generation. As shown in Table and 3, our model shows better FVD scores than previous approaches if the same amount of data is used. Moreover, our model achieves comparable or even better CLIPSIM scores, compared with state-of- the-art as shown in Table 2 and 3, indicating good text-video alignment. CMD shows a slightly worse CLIPSIM score than ModelScope and VideoLDM on MSR-VTT, but note that our model (1.6B) is 1.99 smaller than VideoLDM (3.1B). Moreover, ModelScope jointly trains on 5 billion image-text pairs along with video data to avoid catastrophic forgetting, in contrast to CMD that does not any image data for training once provided pretrained image diffusion models.
7

---

Published as a conference paper at ICLR 2024
10
1.0
80
Mode Scepa
LVDM
D UXDM
ModerSoop
70-
G MideBoospe
LVDM
CMD (Cont)
(AE)
60
0.6
(Cont)
(Som)
CMD (kroe)
(Mcl
BB (Wet)
0.4
Single
0.0
(a) FLOPs
(b) Time
(c) Memory
Figure 5: Training efficiency. (a) FLOPs, (b) sec/step, and (c) memory (GB) of different methods that are trained on 16-frame videos with resolution of 512 x 512 and batch size of 1. All values are measured with a single NVIDIA A A100 80GB GPU with mixed precision. For fair comparison, we do not apply gradient checkpointing for all models. See Appendix D for details.
1750
ModesSoppe
70-
ModelEczee
20-
ModweScpp
LVDM
1900
(Dun))
60
(Cum)
CMD (Ouns)
1850
60
Menocy IEEI
1000
40
790
80
500
20-
250
10
1100-100
5000+50
stsp-100
2000h_52001
FL.PPs
(b) Time
(c) Memory
Figure 6: Sampling efficiency. (a) FLOPs, (b) time (s), and (c) memory (GB) different methods that sample 16-frame video with resolution of 512 1024 (i.e., batch size by default). All values are measured with a single NVIDIA A100 40GB GPU with mixed precision. Note that we exclude the of Stable Diffusion decoder for all measurements. See Appendix D for details.
4.3 ANALYSIS
Training efficiency. Figure summarizes the computation (floating point operations; FLOPs), time, and memory consumption in training each component of CMD and compares the values with other CMD require less
public text-to-video diffusion models. As shown these plots, all components memory and computation for training due to the decomposition of videos as two low-dimensional latent variables (content frame and motion latent representation). Notably, CMD shows significantly fewer FLOPs than prior methods: the bottleneck is in the autoencoder (0.77 TFLOPs) and is ~12 more efficient than 9.41 TFLOPs of ModelScope. Note that one sums up the FLOPs or training time of all three components in CMD, they are still significantly better than existing text-to-video diffusion models. We also note that the training of content frame diffusion models and motion diffusion models can be done in parallel. Thus, the training efficiency (in terms of time) can be further boosted. We also provide a model parameter size comparison in Appendix D.
Sampling efficiency. Figure 6 reports FLOPSs time, and memory consumption sample videos. As shown in Figure 6a, existing text-to-video diffusion models require tremendous computations for sampling since they directly input videos as nigh-dimensional cubic arrays. particular, they over- look common contents in video frames (e.g., static background), and accordingly, many spatial layer operations (e.g., 2D convolutions) become unfavorably redundant and tremendous. However, CMD avoids dealing with giant cubic arrays, and thus, redundant operations are significantly reduced, resulting in a computation efffftatton-effccinnt video generation framework. The sampling efficiency is also reflected in sampling time (Figure 6b); CMD only requires ~3 seconds with a DDIM sampler (Song al., 2021a) a) using steps, which 10x faster than existing text-to-video diffusion models.
Not only improving computation efficiency, our method also exhibits great memory efficiency com- pared with existing methods due to the significantly reduced input dimension. Note that the im- provement becomes more significant if the models sample multiple videos at once (i.e., a batch size larger than 1) because, in that case, the memory bottleneck mainly stems from the computation of intermediate features for sampling rather than the memory allocation of the model parameters. For instance, as shown in Figure 6c, our model uses about 8.6GB GPU memory to generate 4 videos in parallel, 2.5x less consumption than the recent ModelScope model that requires more than 20GB.
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablation studies. (a) FVD on UCF-101 evaluate each component. Reconstruction: FVD between real videos and their reconstructions from our autoencoder. Motion prediction: FVD between real videos and predicted videos with the motion diffusion model conditioning on (ground- truth) content frames encoded by our autoencoder. Content generation: performance of CMD, where content frames are generated by our content frame diffusion model. (b) R-FVD of autoencoders on WebVid-10M with different channel sizes D, video lengths L, and the usage of weighted sum or not. (c) FVD of motion diffusion models on UCF-101 with different content frame patch sizes.
(a) Performance of each component
(b) Autoencoder
(c) Motion diffusion
Task
FVD
D
L
Weight.
R-FVD
Config.
x patch.
FVD
Reconstruction
7.72
16
16
56.8
DİT-L/2
16
40.4
Motion prediction
19.5
8
16
69.5
DIT-L/2
32.9
Content generation
73.1
8
16
X
76.1
DIT-L/2
19.5
8
24
81.3
Content frame
Video clip
Figure 7: Content frame visualization with the corresponding video frames.
Ablation studies. In Table 4a, we report the FVD values by using only some of the components in CMD. As shown in this table, each module in CMD shows reasonable performance, which validates our design choices for the overall framework. Morcover, in Table 4b, we analyze the performance of the autoencoder under various setups; one can observe that the of weighted sum in content frame design helps to achieve better reconstruction, and our autoencoder can encode videos with longer length than 16 (e.g.. =24) with reasonable quality as well. Finally, Table 4c shows that motion diffusion models exhibit a reasonable performance with large patch sizes, so one can control tradeoff between computation efficiency and memory efficiency by adjusting the patch size.
Content frame visualization. Figure 7 visualizes videos in WehVid-10M and the corresponding content frames. As shown in this figure, the content frames resemble the original video frames, i.e. in the content frames, the background (e.g.. buildings) and objects (c.g., bicycle) appear similarly to the video frames. Moreover, one can observe that only the region with moving objects is corrupted, e.g., for the content frame of a video with moving arm, where the arm appears corrupted.
CONCLUSION
We proposed CMD, an efficient extension scheme of the image diffusion model for video generation. Our key idea is based on proposing a new encoding scheme that represents videos as content frames and succinet motion latents to improve computation and memory efficiency. We hope our method will facilitate lots of intriguing directions for efficient large-scale video generation methods.
Limitation and future work. In this work, we primarily focused on generating a video fixed length (e.g., L 16). One of the interesting future directions would be extending our method for long video synthesis, similar to PVDM (Yu et al., 2023b) which considers Another interesting direction is develop better form of content frame and motion latents to encode video with higher quality but still enable exploiting pretrained image diffusion models. We provide more detailed discussion of limitation and future work in Appendix I.
generation.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We believe CMD can provide a positive impact in real-world scenarios related to content-creation applications. Since CMD can instantly synthesize videos from arbitrary user text prompts, it can save time for designers (Villegas al., 2023) who want to generate new content by providing them with an initial shape of such desired result. Moreover, given that the success of large text-to-image generation models (Rombach et al., 2022; Saharia et al., 2022; Balaji et al., 2022) has facilitated intriguing applications such as image editing (Brooks et al., 2023; Kim et al., 2023; Meng et al., 2022) and personalized generation (Ruiz et al., 2023; Gal et al., 2023), we expect developing large-scale video generation framework will promote similar applications in the video domain as well (Molad et al., 2023).
In contrast, there also exists some potential negative impact of developing a large-scale genera- tion framework generate sensitive and malicious content, e-g., DeepFake (Guera & Delp, 2018), as discussed by some recent large-scale video generation works (Villegas et al., 2023). Although generated videos from CMD are relatively short and the frame quality yet distinguishable from real-world videos, one should be aware of this issue and keep considering to develop safe video generation framework in the future.
REPRODUCIBILITY STATEMENT
We provide implementation details (e.g., hyperparameter, model, and optimizer) and experiment setups (e.g., how the metrics are computed) in Section 4 and Appendix B.
ACKNOWLEDGEMENT
SY thanks Subin Kim, Jaehyun Nam, Jihoon Tack, and anonymous reviewers for their helpful feed- back on the early version of the manuscript. SY also acknowledges Seung Wook Kim for helping with text-to-video model training and thanks Yewon Kim for the valuable feedback on the design of the project page. For SY and JS: This work was supported by Institute of Information commu- nications Technology Planning & Evaluation (IIPP) grant funded by the Korea government (MSIT) (No.2019-00 10019-0000075, Artificial Intelligence Graduate School Program (KAIST); No. N0.2021---,2068, Artificial Intelligence Innovation Hub; No.2022-0-00959)
REFERENCES
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08475 08477, 2023.
Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In IEEE International Conference on Computer Vision, 2021.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representa- tions, 2018
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble expert denoisers. arXiv preprint urXiv:2211.0132 2022.
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao- Yen Tseng, Fabio Nonato, Matthias Muller, et al. LDM3D: Latent diffusion model for 3d. arXiv e-prints, 2023.
10

---

Published as a conference paper at ICLR 2024
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In JEEE Conference on Computer Vision and Pattern Recognition, 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pixx Learning to follow image editing instructions. In IEEE Conference Computer Vision and Pattern Recognition, 2023.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In IEEE Conference Computer Vision and Pattern Recognition, 2022.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances in Neural Information Processing Systems, 2017.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference Machine Learning, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghnn,, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in Neural Information Processing Systems, 2016.
Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt. StyleVideoGAN: tem- poral generative model using pretrained StyleGAN. arXiv preprint arXiv:2107.07224 2021.
Jean-Ya Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In International Conference Learning Representations, 2023.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic VQGAN and time-sensitive transformer. In European Conference Computer Vision, 2022.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In IEEE International Conference on Computer Vision, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Cade Gordon and Natalie Parde. Latent neural differential equations for video generation. In NewrIPS 2020 Workshop on Pre-registration in Machine Learning, 2021.
David Guera and Edward J. Delp. Deepfake video detection using recurrent neural networks. IEEE International Conference Advanced Video and Signal Based Surveillance, 2018.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint MrXiv:2307.047255 2023.
11

---

Published as a conference paper at ICLR 2024
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long vidcos. In Advances in Neural Information Processing Systems, 2022.
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.1322 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint urXiv:2200.22:00 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. In International Conference on Learning Representations, 2023.
Tobias Höppe, Arash Mchrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. Transactions Machine Learning Research, 2022.
Jun- Ting Hsich, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to de- compose and disentangle representations for video prediction. In Advances Neural Information Processing Systems, 2018
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In IEEE International Conference on Computer Vision, 2017.
Yuming Jiang, Shuai Yang. Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2Performer: Text-driven human video generation. arXiv preprint arXiv:2303. 13495, 2023.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals. Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, 2017.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In IEEE Conference on Computer Vision and Patlern Recognition, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 2023.
Subin Kim, Sihyun Yu, Jacho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. In Advances in Neural Information Processing Systems, 2022.
Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong, Kihyuk Sohn, and Jinwoo Shin. Col- laborative score distillation for consistent visual synthesis. In Advances in Neural Information Processing Systems, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-based model for stochastic video gener- ation. In International Conference on Learning Representations, 2020.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint (xXiv:1804.01522 2018.
12

---

Published as a conference paper at ICLR 2024
Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong. Revisiting hierarchical approach for persistent long-term video prediction. In International Conference on Learning Representations. 2021.
Xin Li, Wenging Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang. VideoGen: A reference-guide latent diffusion approach for high definition text-to-video generation. arXiv preprint (eXiv:2309000340 2023a.
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. arXiv preprint arXiv:2309.07906, 2023b.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309. /5091, 2023.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: An empirical study on video diffusion with transformers. arXiv preprint arXiv:2305.13: 2023.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation- based adversarial video prediction on large-scale data. arXiv preprint wxXiv22003.0033. 2020.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In Interna- tional Conference on Learning Representations, 2022.
Antoine Miech, Dimitri Zhukov, lean-Baptist Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howto 100: Learning text-video embedding by watching hundred million narrated video clips. In IEEE International Conference on Computer Vision, 2019.
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01322 2023.
Andres Munoz, Mohammadreza Zolfaghari, Max Argus, and Thomas Brox. Temporal shift GAN for large scale video generation. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In IEEE Conference on Computer Vision and Puttern Recognition, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers, In IEEE Interna- tional Conference on Computer Vision, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Gob, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In JEEE Conference on Computer Vision and Pattern Recognition, 2022.
13

---

Published as a conference paper at ICLR 2024
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE Conference Computer Vision and Pattern Recognition, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Infor- mation Processing Systems, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, 2022.
Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, and Pieter Abbeel. Autoregressive latent video prediction with figh-fidelity image generator. In IEEE International Conference on Image Processing, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In International Conference Learning Representations, 2023.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. In Interna- fional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0400,, 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using LSTMs. In International Conference on Machine Learning, 2015.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In International Conference Learning Representations, 2021.
Sergey Tulyakov, Ming-Yos Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEFE Conference on Computer Vision and Pattern Recogni- tion, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01770 2018
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Advances in Neural Information Processing Systems, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing mo- tion and content for natural video sequence prediction. In International Conference on Learning Representations, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, 2019.
14

---

Published as a conference paper at ICLR 2024
Ruben Villegas. Mohammad Babaeizadeh, Pieter-Jan Kindermans, Herann Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2023.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint (xXi::2308.06571, 2023a.
Wenjing Wang. Huan Yang. Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305. 10874, 2023b.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang. Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang. Cunjian Chen, Chen Change Loy. Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. LAVIE: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. /5103, 2023c.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In International Conference on Learning Representations, 2020.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. GODIVA: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In European Conference on Computer Vision, 2022.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, 2023.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.10155 2021.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203. 09481, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In IEEE Conference on Computer Vision and Pattern Recognition, 2023a.
Sihyun Yu. Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wor Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jínwoo Shin. Video probabilistic diffusion models in projected latent space. In IEEE Conference on Computer Vision and Pattern Recognition, 2023b.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, and Karsten Kreis. LION: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems, 2022.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to- video generation. arXiv preprint 2r0iv:2309. 15818, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint a2X11:2211.11018, 2022.
15

---

Published as a conference paper at ICLR 2024
A MORE DISCUSSION ON RELATED WORK
There are several recent works that have some similarities to CMD. In what follows, we discuss the relevant work and the differences with our method in detail.
Motion-content decomposition. CMD is similar to many previous video GANs that generate videos via motion-content decomposition (Villegas et al., 2017: Hsieh al., 2018; Tulyakov al., 2018: Tian et al., 2021; Munoz et al., 2021; Yu et al., 2022: Skorokhodov et al., 2022). To achieve generation with motion-conten controllability. they sample random content and motion vectors and use them to generate overall style and underlying motions. We take a similar approach using pre- trained image diffusion models by training a video autoencoder that decomposes video as content frame (similar to content vectors) and motion latent representation. Here, note that CMD does not strictly decompose motion and contents in a given video, but the scheme is similar at a high level. Moreover, CMD shares similarity to several diffusion models for video generation (Guo et al., 2023; Jiang et al., 2023; Ni et al., 2023). However, they have not achieved (a) an efficient generative modeling and (b) exploitation of pretrained image diffusion models simultancously. Specifically, (Guo et al., 2023) exploits pretrained image diffusion models, but it deals with videos as cubic ten- sors, (Jiang et al., 2023; Ni et al., 2023) do not pretrained image diffusion models, and (Ni al., 2023) uses flow for motion encoding, which is a high- nigh-dimensional cubic tensor as well.
Video prediction. Our work also has a relationship between video prediction models that forecast future video frames given at previous frames as input (Srivastava et al., 2015; Finn et al., 201 Denton & Birodkar, 2017; Babaeizadeh et al., 2018; Denton & Fergus, 2018: Lee al., 2018; Villegas et al., 2019; Kumar et al., 2020; Franceschi et al.. 2020; Luc et al., 2020; Lee et al., 2021; Seo al., 2022). Similar our method, some video prediction methods predict video frames as low-dimensional latent space rather than raw pixel space to increase the window size and achieve efficiency. The main difference between our method and these approaches is in the input condition: method provides content frame as input, whereas they provide initial frames
input.
Difference with PVDM. Our model shares a similarity to recent latent diffusion models for videos (He et al., 2022; Zhou et al., 2022; Yu et al., 2023b). In particular, our method is quite similar to PVDM (Yu et al., 2023b); this work also proposes a latent video diffusion model to target unconditional video generation based on proposing video encoding scheme to decompose them as triplane latents. In this work, each latent in triplane latents lies a different space from the video frames. Thus, it is difficult to exploit pretrained image models. Different from this work, our pri- mary focus is on conditional video generation, and we introduce the "content frames" concept to exploit pretrained image diffusion while avoiding handling giant cubic video tensors.
Difference with VideoFusion. Our work shares a similarity to VideoFusion (Luo et al., 2023). Unlike conventional approaches that add temporal layers the image diffusion models for achieving T2V generation, VideoFusion also considers the training of an additional diffusion model in addition to the pretrained image model for generating videos. However, in contrast to CMD, their primary focus is not on achieving dimension reduction for improving computation and memory efficiency.
Discussion with concurrent works. Show-1 (Zhang et al., 2023) also considers an efficient T2V generation via mixture of pixel-level image diffusion models and latent image diffusion models. LAVIE (Wang et al., 2023c) considers video generation using cascaded latent diffusion models. VideoDirectorGPT (Lin et al., 2023) proposes to combine large language models to generate text prompts for longer video generation.
et al. (2023b) considers controllable video generation con- ditioned given image and the motion direction. Text2Video-Zen (Khachatryan et al., 2023) considers sero video generation from a pretrained T2I model without any video data. However, neither of them considers the temporal coherency of videos for improving efficiency.
16

---

Published as a conference paper at ICLR 2024
B IMPLEMENTATION DETAILS
B.I DATASETS
UCF-101 (Soomro et al., 2012) is human video dataset that includes 101 different types of human actions. Each video consists of frames with 320 240 resolution with varying video lengths. The dataset contains 13,320 videos in total; 9,537 videos are in the train split, and the rest of them are in the test split. We only use train split for training and evaluation of the model, following the experimental setup used in recent video generation methods (Yu et al., 2023a; Singer et al., 2023). We resized all video frames to 64 64x64 64 resolution frames and clipped them to a video length of 16. For the zero-shot evaluation, we resize each video to 512x512 resolution.
Web 10M is dataset that consists of 10,727 ,077 text-video pairs as training split. The dataset also contains validation split that composed 5,000 text-video pairs. We use train split for training the model and use validation set for evaluation. Since some videos in the training split are not available, we exclude these videos from training. We resize all video frames to 512x 1024 resolution and clip each video into length of 16.
MSR-VTTI is a dataset consisting of 10,000 videos and corresponding captions. We use test split for zero-shot evaluation of our method, which contains 2,000 videos and corresponding text captions. We only use text captions to measure the alignment between text prompts and generated videos.
B.2 BASELINES
In what follows, we explain the main idea of baseline methods that used for the evaluation.
DIGAN (Yu et al., 2022) presents a video GAN based on adapting the concept of implicit neural representations (or neural fields) into the generator.
TATS (Ge et al., 2022) presents a video autoencoder by extending an image autoencoder in VQGAN (Esser al., 2021) and trains an autoregressive Transformer for a generation.
CogVideo (Hong et al., 2023) presents large-scale autoregressive Transformer for video gen- eration extended from a pretrained model for images.
Make-A-Vides (Singer al., 2023) presents method achieve rext-to-video generation with- out having text-video pairs but only with text-image pairs and video data.
VideoFusion (Luo et al., 2023) proposes new extension scheme of pretrained image diffusion models for video generation by training additional diffusion model that achieves frame-by- frame generation from intermediate noises from pretrained image diffusion models.
MAGVIT (Yu al., 2023b) proposes non-autoregressive Transformer for videos, based on extending son-autoregressive Transformer for images, MaskGiT (Chang et al., 2022).
VideoFactory (Wang et al., 2023b) proposes a new swapped cross-attention for better video diffusion models and introduces a large-scale rext-video dataset.
PYoCo (Ge et al., 2023) explores noise prior to extend image diffusion models for better video generation, instead of starting from i.i.d. Gaussian noises.
LVDM (He al., 2022) extends latent image diffusion models by modeling video distribution in spatiotemporally downsampled latent space.
ModelScope (Wang al., 2023a) trains latent video diffusion model from a pretrained text- to-image diffusion model by adding several temporal layers.
VideoL.DM (Blattmann al., 2023) proposes an extension scheme from text-to-image diffu- sion models to text-to-video diffusion models by adding temporal layers in pretrained diffusion models and pretrained image autoencoder.
VideoGen (Li al., 2023a) generates video from given image and text prompts by generating motion latent representation inspired by flow-hased temporal upsampling.
GOVIDA (Wu et al., 2021) trains an autoregressive Transformer with large-scale text-video pair datasets, How 100M (Miech et al., 2019).
NÜWA (Wu et al., 2022) extends the approaches in GODIVA to multi-modality, including im- ages, text, and videos.
17

---

Published as a conference paper at ICLR 2024
B.3 TRAINING DETAILS
Autoencoder. In all experiments, we use the Adam optimizer (Kingma & Ba, 2015) with learning rate of le-5, (f1,82) (0.5, 0.9) without weight decay. We use 8 NVIDIA A100 80GB GPUs for training with a batch size of 24, and it takes ~ week for the convergence with x024x512 resolution videos. For backbone networks, we use TimeSFormer (Bertasius al., 2021) following PVDM (Yu et al., 2023b), and we use single convolutional layer as pre- and post-layers at the end of the encoder and at the beginning of the decoder. We provide other hyperparameters related to the model configurations in Table 5:
Table 5: Hyperparameters related to our autoencoder. Hidden dim. (hidden dimension), depth, head dim. (head dimension), and num. head (number of heads) denote the channel, the depth, and the dimension and the number of heads used in the attention layer used in the backbone TimeSformer.
channel
Height
Width W"
Imput patch size (I1 W/W"
Hidden dim.
channel
Dataset
Depth
Head dim.
Num. head
UCR-101
32
32
(2, 2)
384
8
12
64
WebVid-10M (SDL.5)
(2.
12
64
WebVid 10M (SD2.1)
32
64
(2, 2)
384
8
12
64
Motion diffusion model. We use the Adam optimizer with learning rate of le-4, (B,, (2))
(0.9, 0.999) and without weight decay. We use 8 and 32 NVIDIA A 100 80GB GPUs to train the model on UCF-101 and WebVid (respectively) and use batch size of 256. It takes 3-4 days for the convergence with x024x512 resolution videos. Since the dimension of с of the text encoder and the hidden dimension in the motion diffusion model are different, we add and train linear projection layer that maps the text hidden feature to the vector with the same dimension as the DiT hidden dimension. Our motion diffusion model implementation heavily follows the official implementation of DiT (Peebles & Xie, 2023), including hyperparameters and training objectives used." We provide other hyperparameters related to the model configurations in Table 6.
Table 6: Hyperparameters related to our motion diffusion model.
Dataset
Config.
Key. patch size
Input patch size
Text Eneoder
Epochs
Fima
UCF-101
D/T-L/2
2 2
3000
0.999
WobVid-10M (SD1.5)
DIT-XL/2
OpenClip (VIT/H)
0.999
WebVid-IOM (SD2.1)
DIT-XL/2
8
2
OpenClip (VIT/H)
3
0.999
Content frame diffusion model. We use the Adam optimizer with a learning rate of lc-4, (81,3)= = (0.9, 0.999) and without weight decay. We use 16 and 64 NVIDIA A 1100 80GB GPUs to train the model on UCF-101 and Web (respectively) and use a batch size of 256. For UCF-101 experiments, we train the model from scratch for fair comparison with baselines. For Web 1OM experiments, we fine-tune Stable Diffusion 1.5 and 1-base (Rombach et al., 2022) for zero-shot evaluation on UCF-101 and other evaluations (respectively), following the recent text-to-video gen- eration works that use pretrained image models (Wang et al., 2023a). It takes 3-4 days for the convergence with 1024x512 resolution video frames due to high dimensionality of video frames. We provide other hyperparameters related to the model configurations in Table 7.
Table 7: Hyperparameters related to our content frame diffusion model.
Datasct
Model
Epochs
Ema
UCF-101 (non-zero-shot)
DIT-XL/2
3000
0,999
WebVid-10M (T2V)
SD 2.1 (basc)
0.999
B.4 METRICS
Sampler. For both motion diffusion models and content frame diffusion models, use the DDIM (Song et al., 2021a) sampler. We use 7) = 0.0 for both models (i.e. without additional random noises in sampling), and we use the number of steps as 100 and 50 for the motion diffusion model and the content frame diffusion model, respectively. For the content frame diffusion model, we use the classifier guidance scale w 4.0 on UCF-101 and w 7.5 on text-to-video generation.
hhttps://gi thub. com/
18

---

Published as a conference paper at ICLR 2024
CLIPSIM. Following the protocol used in most text-to-video generation work (Singer et al., 2023; Wang et al., 2023b), we calculate CLIP scores (Wu et al., 2021) between a text prompt and gener- ated video frames and report the average between them. Specifically, we each video frame into an image of resolution 224x224 and use it as an input to the CLIP image encoder. Following Vide- oLDM (Blattmann al., 2023), we use the ViT-B/32 CLIP model (Radford et al., 2021).
FVD. For Fréchet video distance (FVD; Unterthiner et al. 2018), we mainly follow the recently fixed evaluation protocol proposed by StyleGAN-V (Skorokhodov al., 2022). Specifically, this proto- col samples a single random video clip from each video and extracts the feature using a pretrained I3D network (Carreira & Zisserman, 2017). For UCF-101 (non-zero-shot), we consider represen- tative scenarios for evaluation: 2,048 real/fake samples (used in most previous methods such as DIGAN (Yu et al., 2022)) and 10,000 fake samples and clips in the training set (used in recent large-scale video generation methods such as MAGVIT (Yu et al., 2023a)). For zero-shot evaluation on UCF-101, we use the common protocol to generate 100 videos per class and calculate the fake statistics (Singer et al., 2023). For text prompts, we use the same text prompt used in PYoCo (Ge et al., 2023), as shown in Figure video clips and text prompts in the validation set to calculate real and fake statisties, respectively. For FPS, we follow the exact same setup in the concurrent VideoLDM (Blattmann et al., 2023).
For non-zero-shot evaluation on WebVid-10M, we use 5,000
'applying eye makeup', 'applying lipstick', 'archery', baby crawling', 'gymnast performing on balance beam', 'band marching), 'baseball pitcher throwing baseball', 'a basketball player shooting basketball', 'dunking basketball in a basketball match', bench press', 'biking'. 'billiards', 'blow dry hair', 'blowing candles', 'body weight squats', 'a person bowling on bowling alley', 'boxing punching bag', "boxing speed bag', 'swimmer doing breast stroke', brushing teeth', weighttifting with barbell', 'clean and jerk', 'cliff diving'. 'bowling in cricket gameplay', 'batting in cricket gameplay'. 'cutting in kitchen', 'diver diving into a swimming pool from a springboard', 'drumming', 'two fencers have fencing match indoors', 'field hockey match', 'gymnast performing the floor', 'group of people playing frisbee on the playground", 'swimmer doing front crawl', 'golfer swings and strikes the ball', 'haircuting', 'a person hammering a nail', 'an athlete performing the hammer throw', 'an athlete doing handstand push up', 'an athlete doing handstand walking', 'massagist doing head massage to man', 'an athlete doing high jump', 'horse race', 'group of people racing horse', 'person riding a horse', 'a woman doing hula hoop', 'man and woman dancing the ice', 'ice dancing', 'athlete practicing javelin throw', person juggling with balls', 'a young person doing jumping jacks', 'a person skipping with jump rope', person kayaking in rapid water', knitting', 'an athlete doing long jump', 'a person doing lunges with barbell', 'military parade', 'mixing in the kitchen', "mopping floor', 'a person practicing nunchuck', 'gymnast performing on parallel bars', 'a person tossing pizza dough', 'a musician playing the cello in room', 'a musician playing the daf', musician playing the indian dhol', 'a musician playing the flute', 'a musician playing the guitar', 'a musician playing the piano', 'a musician playing the sitar', 'a musician playing the tabla', 'a musician playing the violin', 'an athlete jumps over the bar', 'gymnast performing pommel horse exercise', 'a person doing pull ups on bar', "boxing match', 'push ups', 'group of people rafting on fast moving river', 'rock climbing indoor; 'rope climbing', 'several people rowing boat on the river', 'couple salsa dancing', 'young man shaving beard with razor', 'an athlete practicing shot put throw', 'a teenager skateboarding, 'skier skiing down', 'jet ski on the water', 'sky diving', 'soccer player juggling football', 'soccer player doing penalty kick in soccer match', 'gymnast performing on still rings', 'sumo wrestling', 'surfing', 'kids swing at the park', person playing table tennis', 'a person doing TaiChi', 'a person playing tennis', 'an athlete practicing discus throw', 'trampoline jumping', 'typing on computer keyboard', 'a gymnast performing on the uneven bars', 'people playing volleyball', walking with dog'. 'a person standing', 'doing pushups the wall', 'a person writing on the blackboard', 'a kid playing Yo- Yo'
Figure 8: Text prompts used for sero-shot evaluation on UCF-101.
19

---

Published as a conference paper at ICLR 2024
C
MORE QUALITATIVE RESULTS
"Fire"
Kerstai
kerstor
itersta
terstor
"3d fluffy Lion grinned, closeup cute and adorable, long fuzzy fur, Pixar render"
"A beautiful Christmas tree is rotating"
*A bigfoot walking in the snowstorm"
*A unicorn flying over mystical land"
"A Teddy Bear is playing the electric guitar, high definition, 4k"
"A corgi is swimming fast"
panda playing on swing set"
Figure 9: 512 1024 resolution, 16-frame text-to-video generation results from our CMD.
20

---

Published as a conference paper at ICLR 2024
"Traffic jam on 23 de Maio avenue, both directions, south of Sao Paulo"
anutter
shull erst
Shutters taa
"Sailboat sailing on sunny day mountain lake, highly detailed"
"Albert Einstein washing dishes"
"Robot dancing in times square"
"Red wine is poured into glass"
"A shark swimming in clear Carribean ocean"
"An astronaut dances in desert"
"A knight riding on horse through the countryside"
Figure 10: 512x 1024 resolution, 16-frame text-to-video generation results from our CMD.
21

---

Published as a conference paper at ICLR 2024
"An Iron Man surfing the sea"
"Blue sky clouds timelapse 4k time lapse big white clouds cumulus growing cloud formation sunny weather background"
"Pouring latte art into silver cup with golden spoon next to it. high resolution, 4k"
"Close up of grapes rotating table. High Definition"
LA.
"A big palace is flying away, anime style, best quality"
A Goldendoodle playing park by lake, high resolution, 4k"
"Beautiful flag waving"
"A Golden Retriever has a picnic on beautiful tropical beach at sunset, high resalution"
Figure 11: 512x 1024 resolution, 16-frame text-to-video generation results from CMD.
22

---

Published as a conference paper at ICLR 2024
D MORE DETAILS IN EFFICIENCY ANALYSIS
Evaluation details. For a fair comparison with baselines, we mainly use the official implementation provided in LVDM (He al., 2022)" and ModelScope (Wang al., 2023a).* For LVDM, use text 2vi 2video. yaml provided by the official implementation that uses a spatial downsample factor of 8 and a temporal downsample factor of We adjust all factors and techniques that affect memory and computation dramatically. First, all values are measured with the same machine with single NVIDIA A100 40GB/80GB GPU. Moreover, we use mixed precision operation and memory- efficient attention mechanisms in all baselines." We also use half precision (fp16) for measuring the efficiency in sampling. Finally, we disable gradient checkpointing for all baselines in measuring the training efficiency. For measuring floating point operations (FLOPs), we use the Fvcore library that measures the FLOPs of Pytorch models."
Exact values. In Table 8 and 9, we provide exact values of each bar in Figure 5 and 6.
Table 8: Exact values of component- wise efficiency analysis.
Method
Model
TFLOPs
sec/step
Memory (GB)
ModelScope
Diffusion
9.41
0.920
69.6
LVDM
Diffusion
6.28
0.456
44.2
Autoencoder
0.77
0.109
8.93
CMD
Content Frame Diffusion
0.34
0.151
21.5
Motion Diffusion
0.14
0.100
18.7
Table 9: Exact values of sampling efficiency analysis.
TFLOPs
Time (s)
Memory (GB)
Method
Step=50
Step=100
Step=50
Step=100
Batch=1
Batch=4
ModelScope
939.0
1877.9
35.1
8.50
LVDM
625.6
1251.2
24.2
48.1
8.38
18.6
CMD
46.8
92.1
3.13
6.05
5.56
8.57
Model size comparison. In Table 10, we summarize the number of model parameters of recent text- to-video generation models. Our model has similar size to ModelScope (Wang et al., 2023a) and is much smaller than popular, well well-perfoo vell-performing text-to-video generation models, e.g., VideoL.MM and Imagen- Video. In this respect, we strongly believe the generation quality of our CMD will be much stronger if one enlarges the overall model sizes by adjusting model configurations.
Table 10: Model size analysis.
Method
ModelScope
LVDM
VideoLDM
Imagen-Video Video
PyaCo
CogVideo
CMD (Ours)
# params.
1.7B
0.96B
3.1B
11.6B
2.6B
9B
1.6B
http://wwww.ccc//iiggggnnnn////e 'https: httttpps///uggnggaaaee... co spaces http://cwo...ooooooeeeeeee///ote http://htobo.cocccccocceeeeeeeee
23

---

Published as a conference paper at ICLR 2024
E SAMPLING PROCEDURE
We summarize the sampling procedure of CMD in Algorithm 1.
Algorithm content-motion latent diffusion model (CMD)
Sample random Gaussian noise XT (Oxx Ix), 2r (1 I2).
2: for = T do
3:
E₁ f==( (1 + w) (Xe, c, - wee, (xe 0, E).
4: Apply a pre-defined sampler (e.g., DDIM (Song et al., 2021a)) to X;-1 from X and ey.
5: end for
6: for t = T to do
7: E (1 + E) (Ze, No. t) WEBN (Ze, 0, Kg. t).
8: Apply pre-defined (e.g., DDIM (Song et al., 20)la)) to 2t-1 from Ze and E
9: end for
10: Decode the clip from latents: x XI:L go (x0, 20).
11: Output the generated video x X1
24

---

Published as a conference paper at ICLR 2024
F ROLE OF MOTION LATENT VECTORS
As shown in Figure 12, given a fixed content frame, our method can generate videos with different motions, by generating different motion latent vectors.
"Fire" (video 1)
"Fire" (video 2)
"Close up grapes rotating table. High Definition" (video
"Close up grapes on rotating table. High Definition" (video 2)
"Sky time-lapse video with clouds" (video 1)
"Sky time-lapse video with clouds" (video
Figure 12: 512x 1024 resolution, 16-frame text-to-video generation results from CMD, where fix content frame for each text prompt and sample different motion latent vectors.
25

---

Published as a conference paper at ICLR 2024
G QUANTITATIVE RESULTS WITH DIFFERENT GUIDANCE SCALES
In Table 11, we report FVD and CLIPSIM scores with different classifier-free guidance (cfg) scales.
Table 11: T2V generation on Web Vid-10M. + and f indicate lower and higher scores are better, respectively.cfg denotes classifier-free guidance scale.
Method
FVD
CLIPSIM t
CMD (cfg=7.0)
235.5
0.3001
CMD (cfg=9.0)
238.3
0.3020
CMD (cfg=10.0)
245.2
0.3031
CMD (cfg=11.0)
246.9
0.3034
26

---

Published as a conference paper at ICLR 2024
H QUALITATIVE RESULTS OF THE AUTOENCODER
In Figure 13 and 14, re visualize ground truth frames in the validation set in the WebVid-100 dataset (left) and reconstructions from our autoencoder (right). To visualize frames better, we only present the first frame in the paper, please refer to the supplementary material for video visualization.
shuttersteck
shutterstsck
shutffessockk
S shifttesstck
STEP1 STEP2
STEP1 STEP 2 STEP REPOU
STEP S
shritter
shutiers
butte e etstck
thuterstscr
Figure 13: 512x 1024 resolution, 16-frame video reconstruction results from CMD.
27

---

Published as a conference paper at ICLR 2024
hutter
shutterstick
shutterstack
shutterstsck
shutterstsck
Figure 14: 512x 1024 resolution, 16-frame video reconstruction results from our CMD.
28

---

Published as a conference paper at ICLR 2024
I LIMITATION AND FUTURE WORKS
While CMD shows promising results on existing video generation benchmarks, there exist several limitations and corresponding future works. In what follows, we explain such limitations in addition to the limitations that we mentioned in our main text.
Autoencoder quality. We found that our keyframe design works very well in pixel space, as also shown in promising results for the UCF-101 generation. While this concept also fairly worked well in latent space built in an image-wise manner (e.g.. Stable Diffusion latent space (Rombach al., 2022)), we found there exists considerable frame-wise quality drop if the underlying motion in the video contains extremely dynamic motion. We hypothesize this is because the latent space that we used for reconstruction does not consider the temporal coherency of videos. resulting in less temporally coherent frames as latent vectors than frames in pixel space. We believe this limitation can be mitigated by the following solutions. First, future works can consider training module in low-resolution pixel space first with training additional upsampler diffusion models (i.e., cascaded diffusion). Moreover, one consider constructing the latent space from scratch using both large image and video data. Finally, our keyframe design as a weighted sum of video frames may not be an optimal choice to represent the overall contents of the video; exploring the better forms of content vectors that are similar to the original image should be an interesting future work.
Model size. We use fairly small autoencoder and diffusion models due lack of resources used for training, compared with recent texi-to-video generation models. Exploring the quality improvement with respect to the number of model parameters also should interesting direction.
Using negative prompts. We do not apply negative prompts in text-to-video generation, which have been recently used improve the generated video quality. We believe that applying this technique to CMD will improve the video quality.
29


Reviews:
[{"review_id": "9Z1RBMoQRn", "reviewer": "Reviewer_cbVn", "summary": "The paper proposes to generate videos by decomposing the latent representation into content-related features and motion features. The proposed method consists of two blocks: decomposed autoencoder and diffusion models for motion sampling. The proposed method generates plausible videos.", "questions": "Please see my concerns in the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well-written and easy to follow.\n\n2) The proposed method achieves state-of-the-art performance in video generation.\n\n3) It makes sense to decompose the video generation into two parts, the content generation and the motion part.", "weaknesses": "1) The idea of decomposing the video generation into two factors has been proposed in previous methods [a, b, c].\n\n[a] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning\n\n[b] Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, Ziwei Liu. Text2Performer: Text-Driven Human Video Generation\n\n[c] Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, Martin Renqiang Min. Conditional Image-to-Video Generation with Latent Flow Diffusion Models.\n\n2) In the training of autoencoder, how to ensure the model learns meaningful motion representation, rather than directly copying the information containing content information from the inputs?\n\n3) The content frame is an average of the latent representations of multiple frames, and this would lead to the generated frames being blurry, especially for the first frame. From the results of Fig. 10 and Fig. 11, the first frames are kindly of blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sOT7axT8CJ", "reviewer": "Reviewer_jHjr", "summary": "This paper proposes to decompose a  video clip into a content frame and a motion vector, both of which are then generated by two latent diffusion models accordingly. The proposed method largely accelerates text-to-video generation while maintaining high generation qualtiy.", "questions": "What is the resolutoin of the generated videos in Table 1, 2 and 3?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. The proposed idea is illustrated clearly.\n\n2. The idea of training an auto-encoder to decompose the video into content & motion is novel and dose help accerlate the generation of videos.\n\n3. The experiments are adequate to illustrate the superiority of the proposed method.", "weaknesses": "1. Constrained by computational resources, the auto-encoder can only handle a limited number of frames at one time, which means that the auto-encoder can only encode motions within a limited interval. The reduced computational cost does not help us to explore the generation of longer videos.\n\n2. The content code is represented by the weighted sum of original frames, which may limit the representing ability of the content code. For example, the key components of a video may appear in different frames, (e.g. one person in frame 1 and another person in frame 2), in which case, it is difficult to maintain the information all key components in the content code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "seg2mss1uW", "reviewer": "Reviewer_cm4G", "summary": "This paper presents the content-motion latent diffusion model (CMD) for video generation. The CMD model consists of an autoencoder and two diffusion models learned in two stages. In the first stage, the autoencoder is trained to map an input video into a lower-dimensional latent representation. This latent representation consists of a single “content frame” (a weighted sum of the input frames) and a latent vector encoding motion information. In the second stage, CMD fine-tunes a pre-trained image diffusion model to learn the distribution of the content frames and additionally trains another diffusion model to generate the latent motion representation conditioned on a given content frame. CMD is compared to several baselines on popular video generation benchmarks such as UCF-101 and WebVid-10M and achieves a comparable or better performance (measured by metrics such as FVD and CLIPSIM) while maintaining lower memory and computational costs.", "questions": "- Would it be feasible to include confidence intervals for results in Tables 1-3? \n\n- Could the authors provide more intuition behind the 2D-projection-based encoding for the motion latent?\n\n- In Figure 5, wouldn’t the comparison between different models’ computational & memory consumption be represented more accurately if there is a single bar for the CMD model with three stacked parts (autoencoder, content frame DM, motion latent DM)? For example, in the Memory panel, it would show that CMD has higher memory consumption than LVDM. \n\n- Additionally, how were the baselines (ModelScope, LVDM) chosen in Figure 5? \n\n- Are the authors planning to release an implementation of their proposed framework?\n\nTypos:\n- Abstract: “can directly utilizes a pretrained” -> “can directly utilize a pretrained”.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed CMD model achieves a comparable or better performance (measured by metrics such as FVD and CLIPSIM) while maintaining lower memory and computational costs against several baselines.\n\n- CMD can utilize a pre-trained image diffusion model which can save training costs. \n\n- With minor exceptions (addressed in the questions section), the paper is written clearly. \n\n- To the best of my knowledge, the CMD model is novel and would be of interest to the ICLR community.", "weaknesses": "- Since the focus is on relatively short videos (16 frames), it would be interesting to establish whether CMD’s downstream performance is still favorable as the number of frames increases.\n\n- The presented results can be even stronger if confidence intervals are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxUXOGiRWA", "reviewer": "Reviewer_BXkg", "summary": "This paper proposes a video diffusion model that can be trained efficiently via content-motion disentanglement. An autoencoder is trained to learn common visual content and low-dimensional video motion. Then, a pretrained image diffusion model is fine-tuned to generate content frames, and the motion patents are generated with a lightweight diffusion model. This method can generate a video faster while achieving better FVD in WebVid-10M.", "questions": "Please see the above questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tCDM consumes much lower computation resources to inference.\n2.\tThe method presented in this paper is cool and has the potential to improve the video generation research community, especially the idea of decomposing a video to motion and content. \n3.\tCDM achieves good CLIPSIM metric in MSR-VTT and WebVid-10M.\n4.\tThe GPU memory usage and speed evaluations are extensive.", "weaknesses": "This is a technically solid paper. I appreciate the design of the autoencoder design in CDM and believe the video generation community can learn from it. But it suffers from the following weaknesses:\n1.\tExperiment comparison setting: According to the supplementary video, CDM’s generation has considerably slow motion or low motion amplitude. However, LVDM and ModelScope can produce diverse and large motions. Therefore, the FVD comparison setting is not fair. We need other experiments to show CDM is better. For example, LVDM is trained on WebVid-10M with frame stride 8 (i.e. sample 16 keyframes in a 128-frame video clip), please re-train CDM to match the setting of LVDM.\n2.\tEvaluation of autoencoder: Please conduct a quantitative evaluation of the performance of the CDM autoencoder. You can use a series of quantitative metrics. For example, SSIM, PSNR, etc. It would be great if you also showed how good stable-diffusion KLVAE can do on video reconstruction.\n3.\tTraining cost: To train CDM, one needs to train an autoencoder, a content diffusion, and a motion diffusion model. As a result, the training cost of CDM is very close to that of training ModelScope or LVDM. According to the supplementary material, training the autoencoder consumes 8 A100 for a week. After that, content diffusion and motion diffusion models can be trained. And they use 96 A100 for days to train on WebVid-10M. The training cost of CDM is not considerably lower than LVDM and ModelScope.\n4.\tVisual definition problem: CDM produces 512x1024 resolution videos. However, the supplementary is a 9M mp4 file, and is very hard to tell the visual definition of the samples. It would be great if the author provided evaluations on the visual definition of CDM.\n\nThis paper is generally well-written. But when I parse through the literature, I find some statements that are somewhat unproper. See below:\n1.\tIn the abstract, the author says CDM is the first that can directly utilize a pretrained image diffusion model. But, to the best of my knowledge, Video LDM, LVDM, and ModelScope are also utilizing pretrained text-to-image models. \n2.\tPlease number your equations. It will be very helpful for readers to understand your math formulas.\n3.\tThe training time for the content frame diffusion generator is not shown in the supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)