CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
Authors: Wenyi Hong
Ming Ding
Wendi Zheng
Xinghan Liu
Jie Tang
Venue: ICLR 2023 poster (2023)
Keywords: pretraining
text-to-video generation

Abstract: In this work, we present CogVideo, a 9B-parameter transformer for text-to-video generation. The CogVideo model has been trained by inheriting a pretrained text-to-image model, CogView2, which significantly reduces the training cost and alleviates the problem of scarcity and weak relevance. We also propose a multi-frame-rate training strategy for better aligning text and video clips. CogVideo achieves state-of-the-art performance in machine evaluation and outperforms publicly available models by a large margin in human evaluation. Its codes and model are also publicly available at https://github.com/THUDM/CogVideo.

Published as a conference paper at ICLR 2023
COGVIDEO: LARGE-SCALE PRETRAINING FOR TEXT- TO- VIDVIDEO GENERATION VIA TRANSFORMERS
Wenyi Hong Ming Dingi* Wendi Zheng
Xinghan Liu
Jie Tang?
TTsinghua University 'BAAI
(hwy22@mai1s, dm1 mmailmm jietange) tsinghua. edu. cn
ABSTRACT
Large-scale pretrained transformers have reached a milestone in text (GPT-3) and text-to-image (DALL-E and Cog View) generation. However, its application video generation still has several challenges: unaffordable huge computation cost and scarcity and weak relevance of the text-video datasets. In this work, present Cog Video, a 9B-parameter transformer for text-to-video generation. The Cog Video model has been trained by inheriting pretrained text-to-image model, View,, which significantly reduces the training cost and alleviates the problem of scarcity and weak relevance. We also propose multi-frame-rate training strategy for better aligning text and video clips. Cog VogVideo achieves state-of-the-e performance in machine evaluation and outperforms publicly available models by large margin in human evaluation. Its codes and model are also publicly available at https::t.. //te:///ttttt thu......... com/ WTWUDD/CCOVIICC
man skiing.
A woman is riding horse the
lion man drinking water,
A girl is dancing, anime.
Nightfall in metropolis.
Figure 1: Samples generated by Cog Video, The actual text inputs are in Chinese. Each sample is 4-second clip of 32 frames, and here we sample 8 frames uniformly for display purposes.
INTRODUCTION
Autoregressive transformers, e.g. DALL-E (Ramesh et al. 2021 and CogView (Ding et al. 2021) have revolutionized text-to-image generation. A few other works have also followed the framework to develop text-to-video transformers (Wu et al. [021b]] lb] Ge et al. 2022). e.g. VideoGPT (Yan et al.)
2021), and demonstrated its superiority over GAN-based methods (Clark al. 2019
Tulyakov et al.
"quual contribution.

---

Published as a conference paper at ICLR 2023
2018). However, the performances are still far from satisfactory. Diffusion probabilistic models, e.g. Imagen (Saharia et al.] 2022) and DALLE-2 (Ramesh et al. (0222,022 represent another line of research for pext-toimmae generation and video generation Ho et al. (2022). However, how to better incorporate the temporal information for text-to-video generation is still a challenge.
In this paper, focus on designing an autoregressive model for text-to-video generation. The critical challenge in previous work is that the generated video frames tend to gradually deviate from the text prompt. This makes vanilla autoregressive models only good synthesizing videos with regular (e.g. forward moving cars) or random patterns (e.g. speaking by random moving lips), but fail at text prompt such as "a lion drinking water". The main reason is that the former case the first frame already provides sufficient information for the subsequent changes, while in the latter the model has to precisely understand the action "drink" in order to correctly generate the desired action the lion lifts the glass to its lip, drinks and then puts down the glass.
Why could the autoregressive transformers well understand the text-image alignment, but struggle for the text-action alignment in videos? One fact is that the duration of videos varies lot. Previous models split the video into many clips with fixed number of frames for training (Wu et al. 202 Ge et al. 2022). Such treatment destroys the alignment between the text and its temporal counterparts in the video. If a "drinking" video is split into four individual clips of "holding a glass", "lifting", "drinking" and "putting down" with the same text "drinking". the model will be confused to learn the precise meaning of drinking.
The other challenge is that the perfect aligned text-video data is scarce, compared to the easy-to- collect billions of text-image pairs (Ramesh et al., 2021). VATEX is probably the largest annotated text-video dataset (Wang et al. (019). However, it has only 41,250 videos. The retrieval-basee text-video pairs, e.g. Howto (Miech et al. 2019), are weakly relevant and most captions only describe the scene without temporal information.
Present Work. Here we present large-scale pretrained text-to-video generative model, CogVideo, which is of 9.4 billion parameters and trained on 5.4 million text-video pairs. To reduce the com- putational cost, Cog has been developed to inherit the knowledge learned from a text-image pretraining model View2 (Ding et al. 2022). To ensure the alignment between text and its temporal counterparts in the video, we propose the multi-frame-ran rate training. The flexibility of the textual condition makes it possible to simply prepend a piece of text describing the frame rate to the original text prompt for modeling different frame rates. To keep the text-video alignment, we choose a proper frame rate description to include the complete action in each training sample. The frame rate token also controls the intensity of the changes throughout continuous frames in generation. We train a sequential generation model and a frame interpolation model. The former model generates key frames according to the text, and the latter recursively fills the middle frames by varying the frame rates to make the video coherent. As shown in Figure D Cog Video can generate high-resolution (880x480) videos. The human evaluation demonstrates that CogVideo outperforms most publicly available models by a large margin. Our main contributions include:
We present Video, which is the largest and open-source pretrained transformer for general text-to-video generation. CogVideo Video demonstrates state-of-the-art FVD on the UCF- 101 benchmark.
We propose the multi-frame-rn training to better align text-clip pairs, which significantly improves the generation accuracy, in particular for movements of complex semantics. This training strategy offers Cog Video the capacity of controlling the intensity of changes during the generation.
We design dual-channel attention elegantly and efficiently finetune a pretrained text-to- image generative model for text-to-video generation, avoiding the expensive full parameter pretraining from scratch.
RELATED WORK
2.1
VIDEO GENERATION
Video generation is long-standing research topic. Most previous works focus on the next-frame prediction task forecasting the future frames based the first video frame. Early works, e.g.
2

---

Published as a conference paper at ICLR 2023
CDNA (Finn et al. 2016) and PredRNN (Wang al. 2017 leverage deterministic methods to directly predict the next frame via CNNs or RNNs. However, these deterministic models are unable to capture the stochastic temporal patterns and synthesize coherent complex scenes. Recently, generative models, especially Generative Adversarial Networks (Goodfellow et al. 2014) (GANs), begin to dominate the area as they can perform unconditional or class-conditional video synthesis without the first frames. VGAN (Vondrick et al., 2016) is the first one to use GAN for video generation. It decomposes video static background and moving foreground, and then generates them with 2D and 3D convolutional networks respectively. TGAN(Saito et al. (017) proposes to separately generate the temporal latent variables and spatial information, and MoCoGAN (Tulyakov et al. 2018 similarly decomposes the latent space into context and motion subspaces. DIGAN Yu et al. 2022
applies implicit neural representations for video encoding. Recently, text-to-video generation emerges as promising direction. The framework of VQVAE (van den Oord et al.] (017) and autoregressive
transformers Vaswani et al. 2017 Brown et al. 2020) quickly become the mainstream methods (Wul et al. 202 Ge et al.) 2022). Ho al. (2022) proposes a video diffusion model along with gradient method recently for text-to-video generation. The previous methods are basically trained on specific dataset, e.g. UCF-101 (Soomro et al.) (0122), making the trained model domain-specific.
Moreover, most of these models are not publicly available.
2.2 AUTOREGRESSIVE TRANSFORMER
Recent years have witnessed the autoregressive transformer emerging as a powerful generative model. The autoregressive models become the most prevalent framework for text generation (Sutskever
et al. 2011). With its prominent capacity of fitting, transformer (Vaswani et al. 2017) gradually
becomes standard neural structure for text generation. Examples are GPT-3 Brown et al. 2020)
and 1-0M0-00BBB (Zeng et al.) 2023). In computer vision, van den Oord et al. (2017) first proposes to train a VQVAE compress the image into sequence of tokens from learned dictionary, which can be then efficiently handled by the autoregressive model. VQ-GAN (Esser et al.) 2020) learns a more semantic-aware dictionary for unconditional image generation. In the text-to-image generation. pretrained autoregressive transformers such as DALL-E (Ramesh et al., 2021) and CogView (Ding jt af. 202T) have shown superiority in open-domain image generation. Besides the pure GPT--ttyle generation, Cog View2 (Ding et al., 2022) proposes new language model CogLM for infilling in the
image generation. Recent autoregressive transformers (Rakhimov et al. 2020 Yan et al. 2021 Wu
ct al. et al.
202 122lab b) have also shown their superiority in video generation. Among them, GODIVA (Wu
202 and NÜWA (Wu et al., 2021b) focus on the meen-domain text-to-vides generation.
However, they simply generate frames frame blocks one by one in chronological order, and suffer
from poor text-video alignment (Cf. $ [
Diffusion probabilistic models, e.g. Imagen (Saharia et al. 2022) and DALLE-2 (Ramesh al. 2022), recently showed very promising results in text-to-image generation. However, this paper we mainly focus autoregressive models due to the sequential nature of temporal information.
METHOD
We first introduce multi-frame-rate training to better align text and video semantics ($ 3.Ip. To overcome the data scarcity and accelerate pretraining, we propose efficient dual-channel attention for video generation by inheriting knowledge from a pretrained text-image model ($ 3.2
3.1 MULTI-FRAME-RATE TRAINING
Before the training, we first tokenize each frame into image tokens, a similar strategy also used the framework of VQVAR (van den Oord et al. 2017).
Training. The key design here that we add a variable frame-rate token to the text and sample frames at this frame rate to compose fixed-length training sequence. The motivations are two folds:
1. Directly separating the long video into clips at fixed frame rate often leads to semantic mismatching between clips and captions. The truncated clip might only contain incomplete actions, which do not correspond to the full text.
3

---

Published as a conference paper at ICLR 2023
Input Text:
Input Frames:
Image Tokenizer
A lion is drinking water Ri 1
VOVAE
Discretize
20"20=400 image tokens per frame 5 frames
Text tokenization
Flatten
Frame Rate
Text
Framee11
Frame-2
Frame-3
Frame-4
Frame-5
Transformer (Stage 1: Sequential Generation)
Interpolste
frames
Sequence
Sequence
Frame Rate
Text
[8]
Frame-1
Frame-2
Frame-3
Frame-4
Frame-5
Transformer
(Stage 2: Recursive Interpolation)
Figure 2: Multi-frame-run generation framework in Cog Input sequence includes frame rate, text, and frame tokens. [B] (Begin-of-image) is separator token, inherited from Cog View2. In stage 1, T, frames generated sequentially on the condition of frame rate and text. Then in stage 2, generated frames are re-input as bidirectional attention regions to recursively interpolate frames. The frame rate can be adjusted during both stages. Bidirectional attention regions highlighted in blue and unidirectional regions are highlighted in green
2.
The adjacent frames usually very similar. A sudden change from the previous frame may incur a large loss. This will lead the models less inclined to explore the long-range correlation because simply copying the previous frame acts like shortcut.
Therefore, for each training sample, we align the text and the frames by sampling videos at variable frame rates, so that videos any length are evenly down-sampled T, frames. In practice, we set up series of predefined frame rates, and select the lowest frame rate for each text-video pair which we can sample at least T, frames. For each video, the frame rate information described the form of text and prepended the original text.
Although the above method improves the alignment of text and video, a side effect is that the generated video low frame rate could be incoherent due the large changes between frames. We thus train another frame interpolation model to insert transition frames to the generated samples of the sequential generation model. Frame interpolation relies heavily bidirectional information.
However, previous transformers (Wu et al.) 2021a Yan et al. 2021 Wu et al.) (021b) are mostly unidirectional. To be aware the bidirectional context, we adopt Modsl General Language Model (CogLM) Ding et al. (2022) which unites bidirectional context-aware mask prediction and autoregressive generation by dividing tokens into unidirectional and bidirectional attention regions. A token in unidirectional region can attend to the tokens in all bidirectional regions and previous unidirectional regions, while a token bidirectional regions can only attend to the tokens in all bidirectional regions. As shown in Figure
(1) all frames in stage and the 2nd, 4th frames in stage 2 are in the unidirectional region; (2) (F Frame Rate), (Text) } and all other frames belong to the bidirectional region.
In this way, bidirectional attention context in text and given frames is fully exploited without interfering with nugo-regressive frame prediction. The models of the two stages also share the same structure and the training process only with different attention masks.
The multi-frame-raine frame-rate generation is hierarchical and recursive process, illustrated in Figure Specifically, the generation pipeline consists sequential generation stage and recursive interpolation stage:
Generation.
4

---

Published as a conference paper at ICLR 2023
1. Sequentially generates T, key frames based a low frame rate and text. The input sequence is [ [(Frar (Frame Rate) J {Text) [B] (Framel)
(Frame T,}] ]. In practice, set T, 5 and the minimum sampling frame rate to frame per second (fps). The hyperparameters are determined by the memory of devices and the performance in small primary experiments.
2. Recursively interpolate frames based on the text, frame rate and known frames. In each
round of interpolation, we split the generated frames into multiple [ 1-ramee blocks overlapping at the beginning and the end, and interpolate frame between the successive frames in each block. The input sequence to CogLM is also [FFrmee Rate) ) (Text) }
[B] (Framel;
(Frame Ta). where Frame 2i(i 1,2,.., ..., 1 J) are to be
autoregressively generated. By recursively doubling (Frame Rate), ) can conduct finer interpolation to generate videos many frames.
3.2 DUAL-CHANNEL ATTENTION
Large-scale pretraining usually demands large dataset. For the open- (ext-to-video generation. ideally we hope the dataset contains sufficient text-video pairs so as to infer both spatial and temporal correlation between video and text. However, it rather expensive and time-consuming to collect high-quality text-video pairs. A compromise method is to leverage the image data to facilitate the learning of spatial semantics. Video Diffusion Model (Ho et al., 2022) and NÜWA (Wu et al. 2021b) try to add text-image pairs into text-video training, which helps achieve better results. However, adding image data will also significantly increase the training cost, especially in large-scale pretraining scenarios.
In this paper, we propose to inherit the learned knowledge from a pretrained text-to-image generation model rather than use the raw image data. Pretrained text-to-image models, c.g. CieWiew2 (Ding et al. 2022). already have a good command the text-image relations. The coverage of the data used train these models is also larger than that of videos.
How to train a video generation model on top of an im- age generation model? The proposed technique is dual- channel attention. As shown in Figure 3 we augment the original attention block (the spatial channel) with an additional cross-frame attention block (the temporal chan- nel). The temporal channel is implemented by a 3D Swin attention (Liu et al.) (021). Besides the restriction of the receptive window, the temporal channel also follows the attention mask of CogLM. More specifically, during quential generation, a token attends to the tokens the previous frames and tokens before it in this frame; and during frame interpolation, the tokens additionally attend the known frames.
Pretrained
Xout
CogView2
Addition
Layer Norm
Dual Channel Attention
Weighted Sum
Spatial Channel
Temporal Channel
We freeze all the parameters in the pretrained text-to-inne text-to-image model preserve its learned knowledge, and finetune the parameters in the temporal channel attention block to learn the cross-frame information. Specifically, the dual-channel attention block with Sandwich-LN (Ding et al. 202T) can be formulated
Layer Norm
Figure 3: The dual-channel attention.
The parameters in CogWiew2 are frozen.
x = O SpatialAAtm
+ +(1 - xx) TennorallAtm(LaayeeNommmeaa
E Fout Win + LayerNorm(z)).
The inter-channel mixture factor a is a vector E (0, 1) where d is the hidden size of the input feature Pin- To restrict the range of a within (0, 1)d we reparameterize it a E Rd is a learnable parameter. Thanks to the decomposition of attention channels dual-channel attention, the computation procedure and sapee--prrmeters of each channel can be flexibly and independently adjusted under various tasks, as long as the outputs of both channels share the same hidden size. We initialize the temporal channel the same the pretrained spatial channel that the
sigmoid(a), where
5

---

Published as a conference paper at ICLR 2023
attention output æ is still in the original CogView2 feature space at the beginning of finetuning. A detailed analysis of the attention is in Appendix
As mentioned above, adapt Swin Attention (Liu al. 2021b for the temporal channel attention to reduce the time and memory overhead. An interesting finding is that, the Swin attention provides a chance for parallel generation in faraway regions of different frames, which further accelerates the autoregressive generation. Further details and acceleration results are illustrated in Appendix
PRETRAINING
Model. CogVideo consists of two models corresponding to two stages. The backbone of the model in each stage is 48-layer Transformer with dual-channel attention, with 48 attention heads in each attention channel and hidden size of 3,072. Each of the two models has 7.7 billion parameters, while 6 billion of them are shared between both models, which finally results in total of 9.4 billion parameters in CogVideo. To avoid potential gradient explosion, we also use Sandwich LayerNorm and use PB-Relax for stabilizing training, as suggested in (Ding et al. 202T). Shifted window attention is adopted in the Stage 2 model with a window size of 10 x 10.
Dataset. We pretrain our model on a dataset of 5.4 million text-video pairs with a resolution of 160 x 160 (can be upsampled to 480 x 480 further). The data is mainly crawled from the Internet, where each video has its matching caption. About 30% of the captions are in English, which have been translated into Chinese by machine translation. About 50% of the captions are sentences, while the others are made of phrases.
Pretraining. Sequences in both stages are of the length 2.065, consisting of 64 text tokens, 5 (frames) x 400 (per frame) image tokens, and separator token. Both text and images are tokenized using video clips with minimum
icetx"] The model in stage is first pretrained for 76,000 iterations frame rate of 0.25 fps, then trained for 15,000 iterations with a minimum frame rate of fps. The model in stage 2 is pretrained for 78,500 iterations with frame rates of 2, 4, and 8 fps. Both models trained in FP16 with batch size- = 416, and optimized by Adam with max learning rate = 2 x 10-4 31 0.9, 32 0.95, weight decay- 10-2
5
EXPERIMENTS
5.1
MACHINE EVALUATION
Machine evaluation was conducted two popular benchmarks for video generation, UCF101 (Soomro et al.] 2012) and Kinetics-600 (Carreira et al.] 2018). Following Rakhimov
al. 2020); Yu al. [ (2022), we Fréchet Video Distance (FVD) (Unterthiner al. (018) and
Inception Score (IS) (Salimans et al. 2016) as evaluation metrics. IS is calculated based on the C3D model [Tran et al. 2015) which was first trained on the Sports- 1M dataset (Karpathy et al. 2014)
and then finetuned on the UCF101 datase FVD is calculated based on I3D model (Carreira & Zisserman 2017) trained on Kinetics-400, following previous works Yu al. 2022, Ge al. 2022 As the level features brought by the image tokenizer ( (VQ-VAE) may increase the distribution difference between generated samples and ground truth, which not the focus of this work, we also evaluate FVD using ground truth reconstructed by the tokenizer.
UCF-101 is human action dataset consisting of 13,320 videos annotated with 101 action classes. Due to the gaps in image style and frame rate between Cog Video's 's training set and UCF-101, we finetune Cog Video for 10,000 iterations with batch size of 192. The model is trained on the whole dataset, following the setting of Clark al. (2019). Yan al. (202I). TTian et al. (2021), Yu et al. (002)). We use class labels as input text and generate samples according to the cl class distribution during inference. For a fair comparison with previous works, we follow Ge et al. (2022) resize the original 160 x 160 Video generation to 128 x 128, and evaluate FVD and IS over 2,048 and
https: : / thub.
2 We evaluate IS and FVD with the official code TGAN-v2 and TATS respectively. I previous version of this paper, we had bug in the FVD evaluation of both UCF-101 and Kinetics-600 due using a wrong I3D checkpoint. We corrected the results in this version.

---

Published as a conference paper at ICLR 2023
Table 1: Video generation performance UCF- 101. Class labels are used as the text inputs. means trained on the training split of UCF-101 only. means that the ground truth used in FVD is reconstructed by the tokenizer.
Table Video generation performance on Kinetics-600. The metrics are based on the 16- frame generated videos which prime on the first 5 frames, following settings of Rakhimov et al. 2020). ** means that the ground truth used the reconstruction result of the tokenizer.
Method
IS (t) FVD (1)
Method
FVD(L)
VideoGPT Yan al. 2021
24.69
DVD-GAN (Clark et al.
2019
27.38
LVT (Rakhimov et al. 2020
224.73
TGANv2 Saato al. 2020)
28.87
1209
VT /Weissenbom et al 20T9
170
MoCoGAN-HID Tan
2021
32.36
838
DVD-GAN-FF Clark al. 2019
69.15
DIGAN Yu et al. 2022
29.71
655
TriVD-GAN-FP (Lac al. 2020
25.74
DIGAN Yu et al.
2022
32.70
577
Transframer (Nash el al. 2022)
25.4
TATS-bass base Ge el 2022
79.28
332
Video diffusion (Ho el a..
2022
16.2
Cog Video (Ours)
51.11
305
Cog Video (Ours)
49.76
Cog Video (Ours)***
220
VigVo (Ours)**
11.08
10
5.0
Cogvidee 15tage
Coplicee E5tage
Copaaaoo 3baae
29 A4
Doyseeeo
Cogaiboo
Ground
15.42
VideoGPT
4.35
2.5
5,61
49.53
TGAN
2.44
CogVideo
Overal Score
1.0
Phatr Testcure
Mocami Heeeirutt
Semmntic Relovers
(4) Human preference. The percentage of being chosen as the best. (b) Overall scores (1-10) for each method.
(6) Scores (1-5) on three important aspocts.
Figure 4: Human evaluation results. "CogVideo Stagee refers to generating videos sequentially by circularly reinserting the last frame into Cog Video's 's Stage 1 model.
10,000 samples respectively. As shown in Table 1 our model achieves state-of-the-art FVD, and achieves higher IS than most baselines.
Kinetics-600 contains 600 classes of human action videos, with roughly 350,000 train and 50,000 test videos in total. We use the action category as input text, and finetune Cog Video on the training set for 12,000 iterations with a batch size of 640. Following the setup of Weissenbrrn et al. (019); Rakhimov et al. (2020), we center-crop and downsample each frame to 64x64 measure FVD. Results are shown in T Our result underperforms some methods on the FVD of this dataset. However, we find that our method achieves much better FVD with reconstructed ground truth, which verifies the guess that the FVD performance is greatly influenced by the low-level features from the VQVAE tokenizer, although this influence is hard to distinguish by eyes after resized to 64 x 64 according the setting (Weissenborn et al. 2019). Unfortunately, none of those previous models or their evaluation codes on Kinetics is open- which prevents us from further analyzing the reasons.
5.2 HUMAN EVALUATION
To further evaluate CogVides we invite 90 anonymous evaluators rate for Cog Video and other open-source baselines including the GAN-based model TGANv2 (Saito et al. 2020) and the GPT- based model VideoGPT (Yan et al. 2021). 30 classes in UCF101 are randomly picked as text conditions and 4 aspects are rated. For VideoGPT, we use the official unconditional pretrained modef" For TGAN22, we use the official source code to train an unconditional generation model under the same setting as that in Saito et al. (2020). To assign unconditionally generated samples into corresponding categories, we choose TSM (Lin et al. 2019) as the action recognition model for post-classification We only keep the samples whose likelihood of a certain class is at least 80%. See Appendix for further details.
httpps: : / / thub. coomwwi 1 son 1 yan/va /vi deoGPT We use the unconditional version of VideoGPT hecause the conditional checkpoint is released.
7

---

Published as a conference paper at ICLR 2023
Finetune, dual-channel attention (b) Naive full parameter finetune
(c) Train from scratch
lion man is cating with hands.
Sled dogs running the snow,
lion dancing the seaside.
tiger playing footbull.
Figure 5: Ablation study for dual-channel attention. The samples are generated with given first frame, fps. Dual-channel attention produces more accurate shapes. The three models are trained on a "fast setting" with fewer data, iterations and batch size than Cog Video described in $ 5.3.1
Table 3:
Ablation study for dual-channel attention on Kinetics-600. FVD is evaluated on generated 11-frame samples priming on 5 frames in a s.000lsample subset of Kinetics-600's test set.
Method
Initialization
Training Setting
FVD (L)
Train from scratch
Random
full parameter training
166.13
Finetune (Naīve)
Cog View2
full parameter training
176.57
Finetune (Dual-Channel)
Cog View2
only 1 channel trained
124.92
Finetune (Dual-Channel)
CigVideo
only channel trained 108.27
Results in Figure 4 show that CogVides significantly outperforms baselines on multiple important aspects including frame texture, motion realism, and semantic relevance, and achieves the top score by the overall quality. 49.53% of evaluators choose Video as the best method, while only 15.42% and 5.6% favor VideoGPT and TGANy2, respectively.
5.3 ABLATION STUDY
In this section, conduct further studies two main technical contributions: dual-channel attention and multi teti--frmm-ratee training.
5.3.1 THE EFFECTIVENESS DUAL-CHANNEL ATTENTION
To verify the effectiveness of dual-channel attention, we conduct ablation studies on its two key components: (1) initializing with text-to-image pretrained model VieWi:w2; (2) training temporal attention channel while freezing all other parameters. The former provides an initialization point with rich text-image knowledge, while the latter enforces preserving and exploiting that knowledge. Three settings are compared against (a) finetuning with dual-channel attention (the same way as Cog Video initialized with View2 and only train temporal channel); (b) naïve full parameter finetune (initialize with Cog View2 and remove temporal module, and apply full parameter finetuning); (c) training from scratch (randomly initialize and train all parameters).
First, we evaluate qualitatively on general-domain datasets. The models are trained on a million subset of the pretraining dataset for 20,000 steps with batch size of 256, where videos related to animals are removed to demonstrate the generalization ability. Generated samples are shown in Figure [5] Dual-channel attention outperforms the other settings with more accurate contours and details, such as the limbs of the "dancing lion" and the appearance of "sled dogs", while training from scratch performs the worst. As there no animal-related video in this train set, the results indicate that both finetuning with dual-channel attention and naïve full parameter finetuning can transfer the knowledge from the text-to-image model, while the former one can better preserve the knowledge and thereby enhance training efficiency and performance.
Second, we further test the aforementioned 3 settings on Kinetics-600 for quantitative evaluation. We additionally test finetuning Cog Video Video to verify the effect of video pretraining. Each model is trained on Kinetics-600 train set for 11,000 iterations with a batch size of 160. Results were shown in Table 3 from which chalneh attention get better FVD than both naive full parameter finetuning and training from scratch, which indicates the superiority of dual-channel attention.
see that finetuning Cog Video scores the best, and finetuning with
8

---

Published as a conference paper at ICLR 2023
fps
fps
fps
Pour the less into the cup.
An African woman running the lawn.
man sweater talks happily with woman.
1 fps in Stuge
fps in Stuge
4 fps
Stnge
A lion man drinking water.
1 fps in Stage
2 fps in
Stage
fps in
Stage
man doing push-ups.
Figure 6: The frame-rate token controls the intensity of change during generation. (The top groups) Generating the following 4 frames with different frame-rate tokens. (The bottom 2 groups) 4- second clips generated with different frame-rate tokens in Stage and then interpolated to 16 frames.
5.3.2 CASE STUDIES OF MULTI-FRAME-RRN RATE TRAINING
Multi-frame-rate training strategy not only enables Cog Video generate videos of multiple frame rates, but also helps better align texts and videos of varying lengths. To demonstrate its effectiveness, we compare the samples generated with different frame rates in Stage in Figure 6
First, we use the Stage 1 model to generate following frames conditioning on the 1, 2, and 4 fps tokens. A lower frame rate corresponds to a longer time interval between neighboring frames. For example, in Figure 6 (top 3 groups) the water level rises faster and the posture of people changes more between adjacent frames with a lower frame rate. These results verify that multi-frame -raa---ratttteee training is able to control the intensity of the changes throughout continuous frames with the frame rate tokens.
In order to verify how the multi-frame frame-rate training help align videos and texts, we process samples generated as above to the same duration and frame rate. To be concrete, we first extend samples to seconds by circularly reinserting the last frame into the Stage model, then interpolate to 4 fps with the Stage 2 model. As shown in Figure (the bottom groups), samples generated with a relatively low frame rate in the first stage may have more precise modeling of longer-term movements, e.g. the periodicity of push-ups and the whole process of "lifting a glass, drinking, then putting it down". On the other hand, Cog Video can generate realistic movements with high frame rate in the first stage when only short-term dependency is needed, e.g. "talking". In other words, a higher frame rate is better at capturing the action details and modeling short-term actions, and lower frame rate is more capable of capturing global information in the time dimension and modeling long-term actions.
CONCLUSION AND LIMITATIONS
We present CogVideo to the best of our knowledge, the largest and the first open-source pretrained transformer for text-to-video generation in the general domain. CigVideo exhibits way to efficiently leverage the pretrained exi-to-image generative model for a text-to-video generation without hurting its image generation capacity. With the proposed multi-frame-ratt training framework, Cog Video is endowed with a better understanding of eex--video relations and abilities to control the intensity of changes during generation. There are still several limitations in Cog VogVo,eo, e.g. restriction on the length of the input sequence still exists due to the large scale of the model and limitation of GPU memory, and we leave them for future work.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We would like to thank Zhao Xue, Shuai Zhao, Sha Yuan for their help in data collection, Weidong Guo, Fengyu Rao, Zhaoyang Zeng, Mingkang Tang, Zhuoyi Yang for their useful discussion, Hanxiao Qu for maintaining the machines and the computational resources supported by BAAL
This research was supported by Natural Science Foundation of China for Distinguished Young Scholars 61825602.
ETHICS STATEMENT
The primary goal of CogVideo is to advance research on video generation methods. Its ability of text-to-video generation has the potential for casing the effort of short video and digital art creation. While in the meantime, we are also aware of its possible ethical impact on society. it might be used for malicious purposes such as reinforcing social stereotypes, violating privacy, generating deceptive or harmful content, etc. In the following part, we discuss these issues and present possible solutions accordingly. Being aware of these ethical impacts, we set license for Cog Video, which demands the users not to use Video (or derivatives of the model) for any deeds that may violate laws or be harmful to the society.
Problem 1: Reinforcing social stereotypes. As mentioned in DALL-E2 (Ramesh et al. 2022) and Imagen (Saharia al.) 2022), visual generation models may inherit biases from their training data and reinforce social stereotypes. For example, if there are more male engineers than female engineers the dataset, the pretrained model is inclined to generate males given input of "engineer". This problem can be alleviated to some extent by both pre-processing and post-processing, For pre-processing, we can fuzzy search keywords related to fairness (such as gender, race, and age) during data collection, and adjust their proportion. For post-procsssing, although there some researches on model post-procce pest-preossing for fairness, the methods for large pretrained model in the general domain is still an open problem. A simple solution is Word Replacement proposed in Ding et al.] (202T), based on the observation that the biases in the generated images/videss videos often come along with fuzzy user input. To be concrete,
can train additional name entity recognition model to find the words about humans, then directly add accurate descriptions (such as "white", "black", "Asian" or "male", female") before those words. The descriptions are sampled according to the real proportion in the world.
Problem 2: Violating privacy. Researchers have found that some private information contained in the dataset could be extracted from pretrained language models (Carlini et al. (021). The same problem exists in multi-modal pretrained models. As the datasets are mainly collected from the websites, private information may be included such as the user's image/video paired with their name. During data collection, we try to filter out data sources with such private information, though there inevitably remains small number of videos of public figures.
Problem 3: Generating deceptive or harmful content. Although the generated videos still have certain gap with real videos according to our human evaluation, we are conscious that pretrained models may generate realistic videos in the near future. Without adequate guardrails, such models could be used to intentionally misinform subjects and potentially empower information operations, or generate explicit content such as sexual and violent videos. During data collection, we manually filter sources with inappropriate data including pomoggaphic and violent content. We further filter out toxic texts using stop-word list and NSFW videos using models During the process we found that, in the original dataset, only 0.2% videos have NSFW value higher than 0.95 and only 0.5% captions contain stop words, while all of them are false positives according to manual inspection. During training, we choose to use Cog View2 as our initialization, whose dataset doesn't 't contain sexual or violent images. When developing API, we set restrictions to prevent users from inputting harmful text descriptions. Additional classifiers will be trained along with pretraining to discriminate the fakes generated by a specific model.
Last but not least, Video committed to promoting academic research and will never be put into commercial use. And we choose not to release datasets in order to further ensure copyright issues.
Thhe model to filter out NSFW videos: ht first frame represent video
model
We use the
10

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We have paid great exertion to ensure reproducibility.
Open-source. We create an anonymous repository http://///nnoooouuuuuuu 4open httprr//////ooi..ooaooonoomooooo
containing codes for pretraining and in- ference. As pretraining requires huge computational costs (pretraining CogVideo takes 20 days 104 A100 GPUs), we also release checkpoints the public to further ensure reproducibility.
Details of models and training procedure: introduced in $ Also, all the details of the model (e.g. structures and settings) can be found in the released code.
Dataset. A brief introduction can be found in $ 4 Here provide more details about the data and the way to reproduce Cog Video. Our data is crawled from public video websites, where each video is paired with caption (either in English or Chinese). We filter out videos longer than 60 sec, as it may cause too weak relevance between the video and captions. The data covers multiple domains including natural scenery (42%), daily activity (36%), sports (5%), animals (7%), others (10%) (food, building, city, abstract art, etc.), and are mainly real videos (rather than artificial videos such as cartoon). The content and quality crawled data recently released general domain text-video dataset,
are very similar to WebVid-10M thus one can refer to Web 10M for reproducing.
REFERENCES
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 2/), pp. 2633 2650,2650, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 1rXXiv1808.01344 2018.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: 1907.0677, 2019.
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint wwXiv:2204.14211 2022.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:20/2.0984/ 2020.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. arXiv preprint
2022.
htttps:/mmmbat whw...tthhuu........ http:////wwvvvooo webvidiii datbbieeddttaee/////omm

---

Published as a conference paper at ICLR 2023
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint urXi::2204.0344 23344, 2022.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7083-7093, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation based adversarial video prediction on large-scale data. arXiv preprint 2XXx:20033.00333. 2020.
Antoine Miech, Dimitri Zhukov, Can--Batistte Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. H0wtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2630-2640, 2019.
Charlie Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.0944 09494, 2022.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprini arXiv:2006. 10704, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 1 2022.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pp. 2830-2839, 2017.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kohayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(056258, 2020.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference Neural Information Processing Systems, pp. 2234-2244 2242, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12/.0402, 2012.
Ilya Sutskever, James Martens, and Geoffrey Hinton. Generating text with recurrent neural networks. In ICML'/I, 1 pp. 1017-1024, 2011.
12

---

Published as a conference paper at ICLR 2023
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104. 15069, 2021.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 4499-4497, 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01170 2018.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6309-63111111 6318, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 03762, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581-4591, 2019.
Yunbo Wang, Mingsheng I Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. 2019.
arXiv preprint
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021a.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417 2021b.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint a0Xiv:2104 10157, 2021.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. The International Conference on Learning Representations, 2022.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. International Conference on Learning Representations, 2023.
APPENDIX
A SHIFTED WINDOW ATTENTION IN AUTO-REGRESSIVE GENERATION
To further alleviate the large time and memory overhead in the temporal channel during training and inference, refer to Swin Attention Liu et al. (2021). The original Swin attention only applied to
13

---

Published as a conference paper at ICLR 2023
non-autoregressive scenarios, extend it to the autoregressive and temporal scenarios by applying an rugo-reeresssve attention mask the shifted windows.
An interesting finding is that, the Swin attention provides chance for parallel generation in faraway regions of different frames, which further accelerates the auto-regressive generation. The dependence of the generation of specific token relies
Auto-regressive mask. A token can only attend to previous frames or tokens before itself in the current frame.
Shifted window. Only tokens within the distance of window size in both width and height dimensions can be directly attended to.
Figure In 3D autoregressive swin attention (window size 2 as an example), the token the red box only attend to (either directly indirectly) the yellow green tokens. The gray tokens in the i-th frame and the token in the red box can generated in parallel.
we can start generating parts of the tokens in the following frames before finishing the generation of all the previous frames they work in parallel. Suppose X,Y is the height and width of each frame, and Az, Ay are the height and width the shifted window. For two tokens at (f1, 21, y1) and (12, 22, v2), t f2, the latter cannot attend to the former either directly indirectly if
As shown Figure 7
(a) Y2)
(41 y2) (l2 L 1)(A,Y + Ay),
(3)
which means that the i-th token in the t-th frame can be generated with (i A, A,y)tt token the 11th frame parallel. In this way, we can generate La,Y tokens in parallel at most, thus greatly enhancing parallelism and accelerating inference com- pared to auto-regressive with standard attention which can only generate one token at time.
parsde generation
To verify the acceleration effect provided by parallel generation, we generate 6 frames x32x32 tokens each frame) with varying shifted window sizes, and measure the time cost w/wo parallel generation. The effect of Swin attention equals to full attention when setting window size to 32. As shown Figure g (1) Ap- plying swin attention to auto-regressive generation accelerates inference. (2) Using parallel generation can further speed up in- ference without affecting generated videos, and achieves around
Figure Acceleration results of parallel generation in auto- regressive generation with Swin attention.
2x acceleration when window size < 8.
B ATTENTION ANALYSIS
To explore the attention mechanism of dual-channel attention, we visualize (1) the attention distribu- tion in the temporal channel and (2) the mixture factor controlling the ratio between the spatial and temporal channel in equation
Figure visualizes the distribution among frames and texts in sequential generation (Stage 1) with heat maps, where only 24 of 48 attention heads in 6 layers are shown for display purposes. The attention patterns be broadly classified into the following categories:
14

---

Published as a conference paper at ICLR 2023
Most of the attention is the text. E.g. the attention heads in violet
Most of the attention is on a certain frame. E.g. the attention heads in pink focus mainly
on the previous frame: the attention heads in blue focus mainly on the first frame besides the text; the attention heads in yellow focus mostly on the frame itself.
Attention is spread over several frames. E.g. the attention heads in green
Layer
Layer 12
Layer 20
Layer 28
Layer 36
Layer 44
Figure 9: The attention distribution among frames and texts in sequential generation (Stage 1). Only 24 of 48 attention heads in 6 layers selected for display purposes. Each attention head is visualized with heat map of size 5x6, where lighter color represents a larger value. The 5x5 block on the left indicates the sum of attention scores (after softmax) between each pair of frames, and the rightmost column indicates the sum of the attention score of each frame to text. That is to say, the grid in row i column (j < 5) represents ExEF.,EF, attm and the grid row column represents ExeR.yeT attn y- where F,, T denotes the set of tokens in the i-th frame and text respectively, and attn,,p denotes the attention score of token x to y.
Some attention heads exhibit a single pattern, while others may exhibit mixture of them. Attention heads in the layer tend to show similar patterns. In lower layers (e.g. layer 12) the heads tend to allocate attention according to position, while in higher layers more attention is allocated to text (e.g. layer 44) or spread over multiple frames. One possible explanation is that there are more high-level features in higher layers such video semantics, by which more frames and texts can interact with each other to make high-level feature analyses.
It is worth noting that many heads in the temporal channel do not allocate much attention to the frame itself, especially in higher layers, while attending to itself important for inference. This shows that the Cog Video performs certain degree of decoupling in the analysis of temporal and spatial features. While the spatial channel is in charge feature analysis within the frame, the temporal channel can allocate more resources explore relationships among different frames. We further illustrate this perspective with Figure I0, which shows that features calculated by Cog View2 in the spatial channel heavily relied
15

---

Published as a conference paper at ICLR 2023
alpha
0.3
0.2
0.
0.0
Layer
Figure 10: The mixture factor controlling the ratio between the spatial and temporal channel in equation dual-channel attention. Only a in half of the layers are shown for display reasons. As a vector of dimension 3072, we show the mean and variance among all of its dimensions in this figure.
C DETAILS ABOUT HUMAN EVALUATION
In this section, we introduce more details about human evaluation for measuring generation quality. The conduction of our human evaluation generally follows previous works including Ramesh et al. (2021): Ding et al. (2021)
We randomly extract 30 classes from UCF101 for video generation, using corresponding video samples in the dataset ground truth items in the evaluation. Based on captions of selected classes, we generate video samples from models including TGANv2, VideoGPT, and our model, Cog Video. To further illustrate the effectiveness of hierarchical multi-frame-rate generation, we also include a Isstgee version of the CogVich Video model which only uses the Stage model and extends samples by circularly reinserting the last frame into the model. For TGANv2, we the official source code to train an unconditional generation model under the same setting as that in Saito et al. (2020). For VideoGPT, unconditionally generated samples into corresponding categories, we choose TSMLin et al. (2019) as the action recognition model for post-classification. We only keep the samples whose likelihood of certain class is at least 80%. A randomly selected subset of samples displayed in Figure
the official unconditional pretrained model generate samples. To assign
For each sample of the video mentioned above, ask evaluators give scores between and 5 indicates the best while 1 indicates the worst) from three aspects including frame texture, motion realism, and semantic relevance. Then the evaluators are required give general score of quality for each sample between and 10, where higher score indicates better quality. After video samples from each caption are evaluated, the evaluators are asked select the best one from them. We show snapshots of the evaluation website in Figure 12
Throughout the process of human evaluation, we invited nearly 100 anonymous evaluators, while 90 of them completed the whole evaluation and were counted in the final results. None of the questions in the evaluation have any time limit. offer each evaluator 75 RMB as a reward for the evaluation. Results of the human evaluation, including the average score and standard deviation for each group, have already been introduced Figure i in the main body. As ground truth samples take an absolute predominance in the best selection question, we have removed the part of ground truth samples in the selection pie plot for clearer model comparison.
D GENERATED VIDEO SAMPLES
Thanks to the recursive interpolation model in Stage 2, CogVideo is able generate relatively high-frame-ratee rate videos, as shown in Figure 13 We provide further examples generated by CogVider in Figure 14] The generated videos in f format be found in supplementary material, with the filename "oog Video patlsssmmmplppp.. The length and the frame rate of provided videos are seconds and 8 fps, respectively.
16

---

Published as a conference paper at ICLR 2023
Sking, 3
Biking, :1
CogVideo
CogVideo (1 Stage)
GMM
VideoGPT
TGAN22
Groundtruth
Figure 11: subset of human evaluation samples. The captions are randomly selected from UCF-101.
The original samples are clips of 16 frames, which are downsampled to 4 frames uniformly for display purposes.
CogVedeo Tact
#NNDORRONTCRRI AIC (1/09TOP, ISA TROWOY TRIRRYY-CRAPLIIIIE CONPL1NIEE
T0X -.T.AA/ECCSAEEEPSSAGGG
6N RETANTIA SIENOPINNCEAAEEER
Figure 12: Snapshots of the evaluation website.
17

---

Published as a conference paper at ICLR 2023
A man is running in the sea. -1 A
Figure 13: A 4-second video sample generated by Cog Video, which firstly sequentially generated at fps and then recursively interpolated for 3 iterations.
E SUPPLEMENTAL HUMAN EVALUATION AND QUALITATIVE COMPARISON
In order to demonstrate Cog Vodeo's performance more thoroughly, we additionally conduct human evaluation compare with Video Diffusion Model (o al. 2022) and NUWA (Wu et al.) 2021b). Considering both of them didn't 't release codes or checkpoints (which is the reason for not including them in our original human evaluation), we evaluate on the 28 text prompts shown on the webpage of VDM and 6 prompts on the webpage of NUWA. We invite 21 anonymous evaluators to rate on aspects with score 1-5 (5 indicates the best): overall quality, frame texture and content, and indicating Cog Video gets better scores all
motion realism. The results are shown in Figure 15 three metrics.
5.0
Cogvideo Video Diffusion Model
CogVVidoo
NUWA
3.91
3.76
3.5
3.3
3.37
3.0
3.0
3.03
2.93
2.5
2.5
2.0
2.0
1.5
Overal Score
Texture Content
Mosion Realsm
1.0
Overall scone
Textuure & Content
Mation Rea lkmm
(a) CogVideo Video Diffusson Model
(b) CogVideo NUWA
Figure 15: Supplemental human evaluation. (left) CogVideo VDM; (right) Cig vs NUWA.
We further provide the qualitative comparison with NUWA(Figure 16] and Video Diffusion Model(Figure [7). From the samples we can see that, Cog Video can generate more realistic and the sea" (NU"A,, "a man is folding
detailed objects, e.g. "ducks in PVDd" "running Cog Video can also generate videos with better motion realism and semantic alignment. For example, in "pouring coffee into coffee cup" (VDD), the liquid level keeps raising in Cog Video's sample, while the liquid level sometimes drops in VDM's sample. 2) in "sunset sea" (VDD,, Cog Video shows the whole process of sunset.
a piece of yellow
18

---

Published as a conference paper at ICLR 2023
F OUT OF DISTRIBUTION SAMPLES
While our training data only consists of real videos (rather than animations) and a small amount of abstract video (e.g. animation of abstract texture), Cog Video is capable of generating out-of- distribution (OOD) videos.
Videos not existing in the real world, such as animals acting like human beings.
Stylized videos such as watercolor painting and Chinese traditional drawing, while our dataset does not include any stylized videos.
Samples are shown in Figure two-folds:
The out-of-distribution generation capability of CogVideo is
Frame-Ievel: generate reasonable images not exisiting in the dataset. Cog Video losslessly inherits the OOD frame-level generation capability from Cog View2 since it preserves all Cog View2' parameters, showing the superiority of dual channel attention.
Video-level: given OOD frames, Cog Video able to generate reasonable actions. For example, when giving an image of cat with red hat playing the guitar, CogVideo can transfer the human hands to cat's paws, and make pluck the guitar strings.
We have admit that, though, the success rate of extreme out-of-distribution generation is not very high, due to several reasons: 1) Generalization is too hard for extreme out-of-distribun e.g. birds playing guitar. It's difficult to relate birds' wings to human arms. 2) The generalization capability is bounded by the image generation model, and sometimes even the first frame is in poor quality.
cases,
G LIMITATIONS
Although Video demonstrates state-of-the-an performance in text-to-video generation, it still has certain limitations and sometimes produces failure cases. The major limitations are summarized below. We further attach each them with possible solutions. is worth noting that, because CogV Video is based on autoregressive stochastic sampling, these text prompts mostly yield good cases with multiple times of sampling.
1. The quality heavily relies on the first frame, thus Cog Video inherits limitations from autoregressive text-to-image generative model, including:
A) Unreasonable shape, which may exist throughout the video.
B) Text-content mismatching, especially when the text is complex. E.g. missing compo- nent, component mismatch, incorrect attributes binding.
C) If there are flaws in the first frame, sometimes Cog Video is not robust enough to self-recover,
amplifies the flaws.
Increasing data and training for both the based image model (CogView2) and CogVideo may alleviate these problems. Leveraging pretrained text model can boost text understanding.
2. Slight temporal inconsistency (unnatural changes). Possible solutions include joint modeling multiple frames in super-resolution and increasing training time.
3. Slight blurry induced by VQ-VAE lossy compression. Our 480 X 480 pixel results are decoded from 60 60 tokens with VQ-VAE, thus may contain slight blur. One possible solution is to further train super-resolution or deblur model on the pixel level.
19

---

Published as a conference paper at ICLR 2023
REAET, W 12 A nunning surtlower
drinding
SR woman
crying.
A. lo are quarroling
Twn heppy
making video colls
takking and laughing.
W niknng backpackk.
1158 2020, boy is surting in the eww.
riding
horse the soa
A young man ls drrnking coffee.
-1 EE man coling pizza.
phono.
danding
-, A ourninng heart.
Figure 14: Further samples generated by Cog Video. The actual text inputs in Chinese. Each
sample is a 4-second clip of 32 frames, and here we sample frames uniformly for display purposes.
20

---

Published as a conference paper at ICLR 2023
Comparison with NUWA
Play golf at swimming pool.
NUWA
CogVideo
Running on the sea.
NUWA
CogVideo
The white sailboat sailed on the sea.
NUWA
CogVideo
A man is folding piece of yellow paper.
NUWA
CogVideo
Figure 16: Qualitative comparison with NUWA LUWA/Wuee al. 2021b). Samples of NUWA obtained
from the paper's 's appendix. Samples of CogVideo generated by Stage model, with frame rate of fps.
21

---

Published as a conference paper at ICLR 2023
Comparison with Video Diffusion Model
Pouring coffee into coffee cup.
Video Diffusion Model
CogVideo
Sunset at sea.
Video Diffusion Model
CogVideo
Ducks pond.
Video Diffusion Model
CogVideo
Construction site activity.
Video Diffusion
Model
CogVideo
Figure 17:
Qualitative comparison with Video Diffusion Model al. 2022). Samples of
Video Diffusion Model are obtained from their official website https: / /tt::/vvi www.edidifffsssee github. ic Samples of Video generated by Stage model, with frame rate of fps.
22

---

Published as a conference paper at ICLR 2023
A cat in red jacket and blue jeans is dancing,
A cat in red jacket and blue sunglasses is drinking tea from a bowl.
A cat in red hat is playing the guitar.
A cat is drawing on drawing board.
People walking in the rain with unbrellas, Chinese traditional drawing.
People walking in the rain with unbrellas, watercolor painting.
Figure 18: Out of distribution samples generated by CogVidea
23

---

Published as a conference paper at ICLR 2023
Shape (prompt: drawing on paper with pen)
B. Text-content matching (prompt: cat drinking bottle of milk with straw)
(prompt: a dog is typing on the keyboard)
Self-recovery (prompt: an oil painting of couple in formal evening wear going home get caught a heavy downpour with umbrellas)
Figure 19; Some failure cases generated by Cog Video.
24


Reviews:
[{"review_id": "hDc074yVHC", "reviewer": "Reviewer_Tzyn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an extension to CogView2, a text to image model, to include a temporal dimension that allows for the generation of short video clips. The authors. The authors extend the architecture by including a temporal attention block in parallel to the spatial attention block pretrained on text->image data. Quantitative results are overall quite good and the human evaluation is very favourable. However, the paper misses some important comparisons and ablations. ", "summary_of_the_review": "The paper is a well written and has solid results. It is definitely a step in the right direction but leaves some open questions and seems to miss some important comparisons. I am nevertheless leaning towards accept.\n\nUPDATE: I read the rebuttal and am mostly happy with it. I think this paper deserves to be presented at ICLR. \n", "strength_and_weaknesses": "## Strengths\n* Solid qualitative results\n* Human evaluation\n* Good approach aiming at reusing an existing image-to-text model and thereby reutilizing spent compute.\n* Simple extension to existing architecture\n* Good presentation \n\n## Weaknesses\n* Qualitatively the videos show strong artifacts between frames, i.e., unnatural changes between frames. The dynamics are somewhat natural but the consistency in color and form is not very good. This is likely due to the strong prior of the model to draw individual frames (images) because of pretraining. With longer training this might become more natural. Maybe it could even be enforced more strongly. The authors do not address this. \n* The model is still clearly limited in its ability to generate natural videos but the authors are not exploring failure cases at all. A chapter or paragraph on limitations would be great.\n* No ablation on impact of model and (pretraining) data size.\n* Lacking comparisons:\n  * No details about how the data was crawled.\n  * No comparisons in terms of data (quality/scale) with similar works like Ho et al\n  * No comparison to Ho et al on UCF101\n  * No human evaluation against Ho et al. which seems comparable in quality\n* Seems worse on some benchmarks to more recent work even though these do not have extra pretraining data on text-video.\n* Hard to pin-point where improvements come from. When comparing to prior work it is hard to figure out where the main gains come from. Is it the new pretraining dataset, is it the pretrained image-text model, is it the modeling innovations in this paper.\n\n## Questions & Comments\n* The finding of FVD on kinetics is interesting! It would be interesting to see the FVD of the reconstruction on the original as well. Could it be that VQ VAE used in this work is not working well?\n* From scratch training better than naive fine-tuning? That’s odd. \n* Multi-frame rate training evaluation would be better with some quantification. Maybe one could look at perplexities when conditioning on a frame rate by using another ground-truth frame rate and look at the perplexity to see whether the models conditioning is really working.\n* Is alpha really important? Why not just taking the sum instead? I strongly suspect it doesn’t matter.\n* Time and spatial subsampling is not completely new but was also used for instance in Weissenborn et al 2019. The adaptivity by conditioning is novel though.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X_tdlNG-kR", "reviewer": "Reviewer_M4hz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an image-to-text generation model based on the autoregressive transformer architecture. The videos are flattened into a sequence of spatial-temporal patches, tokenized by a VQVAE, and concatenated with the text input. The authors propose several engineering improvements that lead to more effective training. The resulting CogVideo model outperforms previous methods on multiple benchmarks.", "summary_of_the_review": "The biggest value of this paper is that it is one of the first text-to-video generation models with reasonable performance. The proposed engineering improvements could be useful. However, I have some concerns regarding the motivation of the proposed method and its advantage over alternative model designs. Overall, I'm slightly leaning towards acceptance, and may change my score depending on the authors' response.", "strength_and_weaknesses": "Strengths:\n- Text-to-video generation is an emerging and exciting field. This paper is one of the first work to demonstrate the possibility of such a system.\n- CogVideo achieves good performance on several benchmarks, outperforming previous methods by a large margin.\n- The paper provides sufficient engineering details. The code and model is also open-sourced.\n\nWeaknesses:\n\n- Why must the videos be sampled into a fixed number of frames during training? This fixed-length restriction seems to be the fundamental motivation of the proposed multi-frame-rate training, but the reason of this restriction is not clear. An autoregressive transformer should be able to process variable-length input. I assume that the GPU memory constraint is the main reason? If so, there could exist better solutions as discussed below.\n\n- The proposed method flattens a video as a sequence of spatial-temporal image patches. While this seems to be a straightforward extension from a text-to-image autoregressive model, it incurs a large amount of computation cost. One alternative solution is to encode each frame (instead of each patch) into a latent vector, and uses another model to generate the frame from the latent vector. Could the authors discuss on the pros and cons on these two design choices?\n\n- Is there any positional encoding to indicate the spatial-temporal position of each patch? \n\n- One potential concern for the current model architecture is that the video generation process may rely too much on the video inputs and less on the text, as text only occupies a small part of the input sequence. Have the authors considered using cross-attention on text instead?\n\n- How important is text information for the second interpolation stage? Is it possible to train an interpolation model using only video data?\n\n- The proposed multi-frame-rate training method does not fundamentally address the misalignment between text and video. The misalignment occurs due to the weakly-annotated dataset. CogVideo is still trained on web datasets. Have the authors performed any data cleaning to ensure better quality than Howto100M? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QQ1LdAfCqw", "reviewer": "Reviewer_5tA9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a large-scale video pretraining network, i.e., CogVideo, for text-to-video generation. The results are plausible. The technical contributions of the paper are 1) Multi-frame-rate training, and 2) dual-channel attention.", "summary_of_the_review": "The performance of the model is not so impressive. The proposed methods are not so interesting.", "strength_and_weaknesses": "Strengths:\n1) The paper presents a large-scale video pertaining network.\n2) The paper is well-written and easy to follow.\n\nWeaknesses:\n1) The results are not of high quality. From the anonymous web demo, the generated videos are blurry and not temporally consistent.\n2) The novelty is limited. The whole architecture is just a combination of two transformers, one for sequential generation and the other for recursive interpolation. Both of them are existing techniques. They are directly adapted from VQVAE-based image generation methods.\n3) The authors employ a temporal channel attention block to handle the temporal coherence. However, there is no ablation study on this. The authors only did ablation studies on fully fine-tuning them, partially fine-tuning them, or training from scratch. One possible ablation study is removing this module.\n4) The comparisons on the UCF-101 and Kinetics-600 datasets are not fairly as the dataset used in CogVideo is significantly larger than others.\n5) The improvements are mainly contributed to the 5.4 million text-video pairs. However, the authors seem not to have a plan to release the dataset. I understand the main focus of this paper is to provide a large-scale pertaining video generation network. However, the performance of this model is far from existing text-to-image models. Therefore, with this model but without data, other researchers still cannot follow this direction.\n6) From Fig. 5, it seems that the results of (a) and (b) are both reasonable.\n7) There is no qualitative comparisons with existing methods in the main paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "94CxIYcuPDg", "reviewer": "Reviewer_kSBX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an autoregressive model for text-to-video generation. The model is based on a Transformer backbone, pre-trained on a text-to-image task, and has three key components: 1) multi-frame-rate training (i.e. during training, the model observes videos sampled at different frame rates so that it can capture both short and long-term context); 2) a frame interpolation model to insert transition frames; and 3) the use of dual-channel attention (i.e. spatial and temporal attention is done in two separate but parallel layers, and the outputs are interpolated with a learnable parameter). Both the machine evaluation (using the popular FVD and Inception scores) and human evaluation show that the proposed approach produces better quality videos than the considered baselines (7 baselines, several of which published in the current year, 2022, were considered).", "summary_of_the_review": "I believe that the paper could have good chances to influence the community, except for the fact that several industrial labs have recently published text-to-video image generation models that seem to generate higher quality videos (although this is of course debatable, since there isn't an apples-to-apples comparison available). Regardless of this, I believe that the work is of high quality, the experimentation and evaluation is sound, and it should be accepted to the conference.", "strength_and_weaknesses": "**Strengths**\n- The paper presents a thorough evaluation of their model, based on both automatic scoring (using FVD and Inception scores), as well as a human evaluation from 90 anonymous evaluators (that were economically. compensated for their work).\n- Some of the key (according to the authors) components were carefully ablated, i.e. the effectiveness of dual-channel attention.\n- The appendinx contains very useful details about the implementatin and the evaluation of the model. I was able to find the answer to many questions that I had in these, especially regarding human evaluation. Kudos to the authors.\n\n**Weaknesses**\n- Several ways of \"factorizing\" the Temporal and Spatial attention have been presented in the past, but the paper does not compare against these approaches. For example, see the factorized attention in the \"ViViT: A Video Vision Transformer\" paper or in \"VidTr: Video Transformer Without Convolutions\".\n- The effectiveness of the multi-frame-rate training is not ablated in the paper, which is a central contribution of the paper, according to the authors, there's merely a visual inspection of the type of videos generated conditioned on different frame rates (see figure 6).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "620ftPR1mu", "reviewer": "Reviewer_9Xh6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "CogVideo proposes a combination of models that can generate videos from a given description. The system uses two model that generate frames in a hierarchical way: the first model generates the \"key-frames\" while the other model interpolates between the generated key-frames. The authors also proposed \"a dual-attention mechanism\" which allows for mix video-image training. The model is trained on a non-public dataset of ~5-4M text-video pairs. The authors tested the quality of generated videos on Kinetics-600 and UCF-101 and also provided a human study on videos generated on random classes of UCF.", "summary_of_the_review": "Overall, while the provided results look good, more analysis and comparison is required to show case the strengths and limitations of the model. Given more analysis the paper should have more impact on the field.", "strength_and_weaknesses": "===== Strengths\n+ The paper is well motivated. The authors enumerate the main issues that make text to video a hard problem and try to address them one by one. \n+ The evaluations are done using both human metrics and machine metrics (FVD).\n+ The authors open sources the code behind the paper which is always a plus.\n+ The authors provided a large number of videos in their demo website which is also appreciated.\n\n===== Weaknesses\n- Weak baselines: The main baselines for the paper (in human evaluation) is TGAN (published in 2017) and VideoGPT (2021). The first one is out-dated at this point and the latter generates videos at a low resolution (64x64 and 128x128) which can affect the human judgement. It's not clear if the authors reduced the resolution of their model for a more fair comparison. I also encourage the authors to provide a similar comparison with newer and higher resolution models such as  NUWA and Video Diffusion.\n- Other problems: following the previous weakness and given the fact that there are not that many text conditional video generation models out there, I suggest authors to compare their model with in other video problems (such as frame conditional video prediction) where there are more baselines to compare to.\n- Out of distribution results: While the videos generated by the model look impressive, it is not clear how the model performs for out of distribution queries. The paper only contains two \"new compositions\" that probably don't exist in the training set: \"A lion man is drinking water\" and \"A tiger is playing football\" while the rest of the examples probably exist in the training set. The demo website contains more such new compositions however the generated videos are not generate suggesting that the model is overfitted to the training data and is just playing videos from memory! Unfortunately, the training data is also private which makes such analysis even harder but I suggest authors to perform an analysis on this matter by generating examples which are more detailed and most likely non-existent in the training set (e.g. \"a cat in red jacket and blue sunglasses is drinking tea from a bowl\").\n- Reproducibility: While the authors provided the source code, the paper lacks details of the model and the used data is private which severely affects the reproducibility. The authors also did not provide any numbers that indicate the required computation to train a 9B model. This is interesting given the fact that the paper mentions computational cost as one of the main issues in text to video.\n- Writing. The quality of the writing can be improved. Particularly, I found the Section 3.2. to be poorly written, assuming that the reader is familiar with details of the previous work CogView2 and CogLM.\n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster