CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
Authors: Wenyi Hong
Ming Ding
Wendi Zheng
Xinghan Liu
Jie Tang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: pretraining

Abstract: Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E) generation. Its application on video generation is still faced difficulties: The huge computation makes training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movements. In this work, we present 9-billion-parameter CogVideo, which is trained by inheriting the knowledge from the pretrained large-scale text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, the CogVideo outperforms the previous public available models at a large margin in both machine and human evaluation. 


Cog Video: Large-scale Pretraining for Text-to-Video Generation via Transformers
Anonymous Author(s)
Affiliation
Address
email
Abstract
Large-scale pretrained transformers have created milestones text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video gen- eration is still facing many challenges: The potential huge computation makes it unafforable for full training; The scarcity and weak relevance of text-video datasets hinder the model to understand complex movement semanties. this work, we present 9B- 1B-parameter transformer Cog Video, trained by inheriting a pretrained text-to-image model, Cog We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, Cog Video outper- forms all publicly available models large margin in both machine and human evaluations.
11
A man sking.
woman running beach the late afternoon.
couple are having dinner.
Nightfall metropolis.
ls is drinking water,
A girl is dancing. Anime.
Figure 1: Samples generated by Cog Video. The actual text inputs are in Chinese. Each sample is 4-second clip of 32 frames, and here we sample frames uniformly for display purpose.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Introduction
12
37 Present Work. Here we present a large-scale pretrained text-to-video generative model, Cog Video, 38 which is of 9.4 billion parameters and trained 5.4 million text-video pairs. We build VidVideo Autoregressive transformers, e.g. DALL-E [9] and Cog View [5]. have revolutionized text-to-image generation recently. It is natural to investigate the potential of autoregressive transformers on text- to-video generation. Previous works followed this basic framework [36] 91. c.g. VideoGPT (37), verifying its superiority over GAN-based methods [4] 27
14
15
18
but are still far from satisfaction.
One common challenge is that the generated video frames tend to gradually deviate from the text prompt, making the generated characters hard to perform the desired actions. Vanilla autoregressive models might be good at synthesizing videos with regular (e.g. straightly moving cars) random patterns (c.g. speaking by randomly moving lips), but fail on text prompt such as "a lion is drinking water". The main difference between the two cases is that, in the former case the first frame already provides sufficient information for the subsequent changes, while in the latter the model has to precisely understand the action "drink" in order correctly generate the desired action the lion lifts the glass to its lip, drinks and then puts down the glass.
18
20
21
22
24
Why do the autoregressive transformers well understand the text-image relations, but struggle to understand the text-action relations in videos? We hypothesize that the datasets and the way to utilize them are the main reasons.
26
27
28 First, it is possible to collect billions of high-quality text-image pairs from Internet [19], but the text-video data are more scarce. The largest annotated text-video dataset, VATEX [2). has only
29
30
1,0250 videos. The retrieval-based tixt- pairs, e.g. Howto100M [17], are weakly relevant and most of them only describe the scene without the temporal information.
31
32 Second, the duration of videos varies lot. Previous models split the video into many clips with fixed number of frames for training, which destroys the alignment between the text and its temporal counterparts in the video. If "drinking" video is split into four individual clips of "holding a glass", "lifting", "drinking" and "putting down" with the same text "drinking". the model will be confused to learn the accurate meaning of drinking.
33
34
35
36
39 based on a pretrained text-to-image model, Cog View2 6 in order to inherit the knowledge learned from the text-image pretraining. To ensure the alignment between text and its temporal counterparts in the video, we propose the multi-frame-raat hierarchical training. The flexibility of the textual condition makes it possible to simply prepend piece of text describing the frame rate to the original text prompt for modeling different frame rates. To keep the text-video alignment, we choose a proper frame rate description to include the complete action in each training sample. The frame rate token also controls the intensity of the changes throughout continuous frames in generation. Specifically, we train a sequential generation model and frame interpolation model. The former model generates key frames according to the text, and the latter recursively fill the middle frames by varying the frame rates to make the video coherent. As shown in Figure T Cog Video can generate high-resolution 4880x480) videos. Human evaluation demonstrates that Cog Video outperforms all publicly available models at large margin. Our main contributions can be concluded as follows:
41
42
43
44
45
46
47
48
49
50
51
We present Video, which is the largest and the first open-source pretrained transformer for text-to-video generation the general domain.
52
53
Video VidVideo elegantly and efficiently finetunes text-to-video generative model from pre- trained text-to-image generative model, avoiding the expensive full pretraining from scratch.
54
We propose the multi-frame-rate hierarchical training better align text-clip pairs, which significantly improves the generation accuracy, in particular for movements of complex semantics. This training strategy endows Cog Video with the capacity of controlling the intensity of changes during the generation.
56
58
2
Related Work
59
2.1
Video Generation
61
Video generation is long-standing research topic. Most previous works focus on the next-frame prediction task- forecasting the future frames based the first video frame. Early works, e.g.

---

63
leverage deterministic methods to directly predict the next frame CNNs or RNNs. However, these deterministic models are unable to capture the stochastic temporal patterns and synthesize coherent complex scenes. Generative models, especially Generative
CDNA 8 and PredRNN B3
64
65
66
Adversarial Networks [TO] (GANs), begin to dominate the area as they can perform unconditional or class-conditions video synthesis without the first frames. VGAN [3T] is the first one to use GAN for video generation. It decomposes video to a static background and a moving foreground, and then generates them with 2D and 3D convolutional networks respectively. TGAN[20] proposes to separately generate the temporal latent variables and spatial information, and MoCoGAN 127]
67
68
70
similarly decomposes the latent space into context and motion subspaces. DIGAN [8] applies
72
implicit neural representations for video encoding. Recently, text-to-video generation emerges as a promising direction. The framework of VQVAE [29] and autoregressive transformers 30 [ quickly
74
Ho et [1]] proposes video diffusion model along with gradient method recently for text-to-video generation. The previous methods are basically trained on a specific dataset, e.g. UCF-101 [23], making the trained model domain-specific. Moreover, most
becomes the mainstream method [35 36 9.
76
of these models
publicly available.
78
2.2
Autoregressive Transformer
79
Recent years have witnessed the autoregressive transformer emerging as a powerful generative model. The autoregressive models become the most prevalent framework for text generation 24] With
81 its prominent capacity of fitting, transformer (30) gradually becomes the standard neural structure for text generation. One milestone is GPT-3 [1. In computer vision, van den Oord al. [29]
85 first proposes to train a VQVAE to compress the image into a sequence of tokens from a learned dictionary, which can be efficiently handled by autoregressive models. VQ-GAN 7 learns more semantic-aware dictionary for unconditional image generation. In the text-to-image generation, pre- trained autoregressive transformers such as DALL-E [9] and Cog [5] have shown superiority in open- dominn image generation. Besides the pure GPT--ttle generation, CogView2 6 proposes a new language model CogLM for infilling the image generation.
84
85
86
87
86
89
Recent autoregressive transformers generation. Among them, GODIVA 35 generation. However, they simply generate frames or frame blocks one by one in a chronological order, and may suffer from poor text-video alignment (Cf. $ [
18
have also shown their superiority video and NUWA 36 focus the open-domain text-to-video
37
35
36
91
92
3 Method
93
In this section, first introduce multi-frume-rate hierarchical training to better align text and video semantics in & 33.1 and then illustrate an efficient method dual-channel attention to inherit the knowledge in pretrained text-image models for video generation in 3.2 To overcome the
85
87 large memory and time overhead caused by the large model and long sequence, we refer to Swin Attention [14] and extend it to autoregressive video generation in 3.3
98
89
3.1 Multi-frame-rate Hierarchical Training
100
Here we present the multi-frame-rate hierarchical training and generation. We follow the framework of VQVAE [29] and first tokenize each frame into image tokens. Each training sample consists of 5 frame of tokens, but our training method differs in the construction of training sequences and generation process.
102
103
104
Training. The key design is to add a frame-rate token to the text and sample frames at this frame-rate to compose fixed-length training sequence. The motivations two folds:
108
Directly separating the long video into clips at fixed frame-rate often leads semantic mis- matching. We still use the full text but the truncated clip might only contain incomplete action.
107
The adjacent frames are usually very similar. A giant change over the previous frame will probably incur a large loss. This will lead the models less inclined to explore the long-range correlation because to simply copy the previous frame acts like shortcut.
109
3

---

Input Text:
Input Frames:
Image Tokenizer
lion drinking water.
VOVAE
Discretize
20-20-400 Image takens per frame
5 frames
Text tokenization
Flatten
Frame Rate
Text
[B]
Frame-1
Frame-2
Frame-3
Frame-4
Frame-5
Transformer (Stage 1: Sequential Generation)
Interpoiate-
frames
Sequence
Sequence
Frame Rate
Taxt
[B]
Frame-1
Frame-2
Frame-3
Frame-4
Frame-5
Transformer
(Stage 2: Recursive Interpolation)
Figure 2: Multi-frame-ratl hierarchical generation framework in CogVideo. Input sequence includes frame rate, text, frame tokens. [B] Pregin-of-image separator token, inherited from Cog View2. In stage 1, T, frames are generated sequentially on condition of frame rate and text. Then in stage generated frames are re-input as bidirectional attention regions to recursively interpolate frames. Frame rate can be adjusted during both stages. Bidirectional attention regions are highlighted in blue and unidirectional regions are highlighted in green
111 Therefore, each training sample we want the text and the frames match as possible. We predefined series of frame-rates, and select the lowest frame-rate for each text-video pair, as long as we can sample least 5 frames this frame-rate in the video.
112
113
114
Although the above method increase the alignment of text and video, the generation at a low frame- rate could be incoherent. We train another frame interpolation model insert transition frames the generated samples of the sequential generation model. Thanks to the generality of CogL. (6, the two models can share the same structure and training process only with different attention masks.
116
117
118
Generation The multi-frame-rate hicrarchical generation a recursive process, illustrated in Fig-
ure 2 Specifically, the generation pipeline consists of sequential generation stage and a recursive interpolation stage:
120
Sequentially generate T, key frames based low frame rate and text. The input sequence is [[Frame Rate) (Text} [B] (Framel}
122
(Frame T.}]. In practice, we always set T. 5 and the minimum sampling frame rate to 1 fps.
123
Recursively interpolate frames based the text, frame rate and known frames. In each round of interpolation, we split generated frames into multiple 1-fame blocks overlapping at the beginning and the end, and interpolate frame between the successive frames each block. The input sequence is [[Frame Rate) (Text) [B] (Frame1}
125
126
(Frame T,Y. where Frame 2i( = 1, .... [Ix) are to be autoregressively generated. By recursively halfing {Frame Rate}, we can conduct finer and finer interpolation to generate videos of many frames.
128
129
130
The effect of CogLM. Tasks such as frame interpolation rely heavily on bidirectional information. However, most previous works use GPT [35] 37 36). which unidirectional. To be aware of the bidirectional context, we adopt Cross-Moda General Language Model (CogLM) proposed in [6] which unites bidirectional context-aware mask prediction and autoregressive generation by dividing tokens into unidirectional and bidirectional attention regions. While bidirectional regions can attend all bidirectional regions, unidirectional regions can attend to all bidirectional regions and previous unidirectional regions. As shown in 2 (1) all frames in stage I and the 2nd, 4th frames in stage
131
132
135
135

---

137 2 are in the unidirectional region: (2) (Frame Rate), (Text) and all other frames belong to the
138
bidirectional region. In this way, bidirectional attention context is fully exploited in text and given frames without interfering auto-regressive frame prediction.
139
140
3.2
Dual-channel Attention
x
141
Large-scale pretraining usually demands large dataset. For open-
142
domain text-to-video generation, ideally we need the dataset to cover sufficient text-video pairs to infer both spatial and tem-
143
144
poral correlation between video and text. However, to collect high quality text-video pairs is often difficult, expensive and time- consuming.
145
148
Layer Narm
A natural idea is make use of the image data facilitate the learning of spatial semantics. Video Diffusion Model [1] and NUWA (36] try add text-image pairs into text-video training, which achieves better results on multiple metrics. However, as for training video-only generation model, adding image data will significantly increase training cost, especially in large-scale pretraining scenarios.
Drwelchenee Atlertion
R
148
t
Attention: base Fipwtell Charnee)
Attention- plus
150
151
Layer Norm
152
153
X1
154 In this paper, we propose to leverage pretrained image generation models instead of image data. Pretrained text-to-image models, e.g. Cog View2 [6. already have a good command of the text- image relations. The coverage of the dataset to train these model is also larger than that of videos.
Figure 3: Dual-channel atten- tion. We initialize Attention- plus the same as Attention-base so that the model behaves ex- actly the same as Cog View2 when it is initialized.
156
158
The proposed technique is dual-chunnel attention, where we only add a new spatial-temporal attention channel to the pretrained Cog View2 [6] at each transformer layer. All the parameters in the Cog View2 are frozen in the training, and only the parameters in the newly added attention layer(See the Attention- pllus in Figure 3 are trainable.
160
162
163
Here we also emphasize that directly finetuning Cog View2 for text-to-video generation cannot well inherit the knowledge, because the temporal attention follows different attention pattern and quickly ruins the pretrained weights during the initial phase of training with large gradients.
164
165
166
Specifically, Transformer layer with dual-channel attention can be computed
4 LayerNorm(zz))
T Attention-base(ff)) + (1 - ox), -Attentionn-luu(())..
(2)
T(+1 = FFN(Layeroorm(z) + zi)):
(3)
167
where denotes input features of layer I; Attention-base and Attention-plus denote two attention channels: FFN and LayerNorm represent Feed-Forward Networks and LayerNorm respectively; a is a vector with length of hidden-size and normalized to 1). The whole structure is the same as
188
170
Cog View2 when ignoring Attention-plus.
171
Both channels are computed as normal multi-head attention with a certain receptive field formulated as follows. For token (t, x. y) in frame block of size (T,, X, Y) (where (t, x. y) corresponds to coordination along time, height and width dimension), receptive field RF is a 3D block with extent
172
173
174
4 la by N
RF (t. (k),, |x
< L. 51 Iy, It - k- < 4. (,) & Mask }.
(4)
175
where Mask(t,..)) represents CogLM attention mask for token (t,z,y) For Attention-base, we restrict receptive field to current frame, i.e, 4, = XX,,, Y,4 1, to fully use Cog spatial modeling ability (therefore referred to as spatial channel). For Attention-plus. which is the only new parameters in Cog Video, we set receptive field 3D local block throughout the whole time dimension, i.e. Lg- Ag, Ly = Ag, l T, (therefore referred to as temporal channel). An, Ay satisfying A, S X, Ay < Y. With A, and Ay. Cog is able to flexibly trade off between quadratic attention cost and size of receptive field. In practice, we use shifted window attention [15] as approximation of 3D block attention and extend it to CogL.M scenario, as
178
177
178
179
180
are hyper-paramenn
182
illustrated in subsection 3,3
5

---

184 It is worth noting that two channels are fused and share the same FFN in each layer, because FFN
185 module of heavy parameters containing much vision knowledge. Due similarity between images and videos, bringing its knowledge to temporal channel will facilitate video modeling. Finally, sharing FFN can reduce parameters, thus speed up training and reduce memory overhead.
187
188
3.3
Shifted Window Attention in Auto-regressive Generation
To overcome large time and memory overhead in temporal channel during training and inference, we refer to Swin Attention proposed [14] and extend it to auto-regressive scenario by applying rego-segressvve attention mask in shifted windows.
190
192
Different from non-autoregressive scenario which original Swin Transformer explores, we propose that Swin Attention can fur- ther accelerate auto-regressive inference because of restricted receptive field. As shown in Figure receptive field is re- stricted by
194
196
197
Auto-regressive mask. token can only attend to pre- vious frames tokens before itself in current frame.
1=i+1
tmi+2
189
Shifted window. Only tokens within distance of win- dow size in both width and height dimension can be directly attended to.
Figure 4: Receptive field (in yellow or green) for the token in red box. Shifted window size is 2 x 2 in this example.
Suppose X,Y is the height and width of each frame, and Ag.Ay the height and width of shifted window. For two tokens at (41, T1, M1) and (t2, 22, v2), 4 t2. 204 the latter cannot attend to the former either directly or indirectly if
202
203
(x1 x² + (y1 y2) 2 (t2 t1 + 1)(AYY + Ay)
(5)
is satisfied. That is to say, the i-th token in frame t can be generated with the (i A + Ay)-th token in frame t 1 in parallel. In this way, we can generate XY XY, tokens in parallel at most, thus greatl enhance parallelism and accelerate inference compared to auto-regressive with standard attention which can only generate one token at a time.
A,,
207
208
4
Training
Based on methods above, the training details of Cog Video are listed as follows:
Model. The backbone of Cog Video in both stages is a Transformer with dual-channel attention. The Transformer has 48 layers, with the hidden size of 3072 in each attention channel, 48 attention heads and 9.4 billion parameters total. Among them, billion parameters are fixed to Cog View2'sss parameters, which includes Position-wiz wise Feed-Forwar Networks (FFN), spatial channel of dual-
212
213
214
215
channel Attention, first frame's 's positional embeddings and all image and text vocabulary embeddings. The specific implementation of Transformer structure is almost identical to Cog View (5] such as using Sandwich LayerNorm and PB-Relax stablize training. Shifted CogL. attention window adoppted in recursive interpolation model with window size 10 10.
216
217
218
219
Dataset. We pretrain our model dataset of 5.4 million captioned videos with a spatial resolution of 160x166 For sequential generation model (Stage-1), we adjust frame rate in each sample to accomodate the whole video, while the minimum frame rate is to I fps. For recursive interpolation model(Stage-22 2), we split videos into clips of different length to accomodate prediction on multiple frame rates including 2,4.8 fps.
220
222
223
224
Pretraining. The sequence lengths in both stages are 2065, consisting of 64 text tokens, 5 (frames)
400 (per frame) image tokens, and seperator token. Both text and images are tokenized with i The parameters are updated by Adam with max learning rate = 2 10 ,1 = 0.9, Bz = 0.95, weight decay- x 1 See Appendix for pretraining details.
The
226
htttps///gttaub..c////U////

---

Table 1: (Left) Video generation performance on UCF-101. Class labels are used as text inputs. * denotes the model trained on the training split of UCF-101 only. (Right) Video generation performance on Kinetics-600, Metrics are measured on generated videos of 16 frames priming on first 5 frames, following settings in [8] denotes groundtruth used in FVD testing is blurred with our image tokenizer icetk.
Method
IS (t)
FVD (4)
VidcoGPT 37
24.69
Method
FVD
DVD-GAN 4
27.38
Latent Video Tranformer [8]
224.73
TGAN22 21 *
28.87
1209
Video Transformer
170
Mo
32.36
838
DVD-GAN-FP[4
69.15
DIGAN 38
29.71
655
TrVDD-AAN-FPI6
25.74
DIGAN 38
32.70
577
TATS-base
79.28
332
CogVideo (Ours)
109.23
CogVideo )*
59.55
CogVideo Video (Ours)
50.46
626
Cog Video (Ours)**
545
Experiments
228
229
5.1
Machine Evaluation
230
Machine evaluation is conducted on two popular benchmarks for video generation, i.e., UCF101 23
231
and Kinetics-600 3 Following Rakhimov al. [18]. Yu et [38], we use Fréchet Video Dis- (ance(FVD) [28) and Inception score(IS) [22] as metrics in the evaluation. FVD is calculated based
232
233
on I3D model trained on Kinetics-400, and IS is based on C3D model [26] which was first trained
with Sports- I dataset [12] and then fine-tuned on the UCF101 dataset. Our evaluation code is the
235
same as the official TGAN-v2 implementation
UCF-101 is a human action dataset consisted of 13,320 videos annotated with 101 action classes. 236
237
Due to the image style and frame rate gap between Cog Video's training set and UCF-101, we use class labels as the input text and fine-tune Cog Video on the whole dataset for 10,000 iterations with
238
239
batch size 192. During inference, we sample class labels according the class distribution. FVD
240
and IS are evaluated over 2048 and 10,000 samples respectively, following Yu et al. 38 Results are shown in Table (Left).
241
242
Kinetics-600 dataset contains 600 classes human action videos, with roughly 350k train and 50k test videos in total. We the action category as input text, and fine-tune Cog Video the training set for 12,000 iterations with batch size 640. Following the setup of Weissenborn et al.
244
Rakhimov al. [18], center-crop and down-sample each frame 64x64, and measure with
246
FVD. Results are shown in Table (Right).
247
5.2
Human Evaluation
248
To further evaluate VidVideo, we invite 90 anonymous evaluators to rate for CigVideo and other open- source baselines including GAN-based model TGANv2 (1] and GPT-based model VideoGPT [37].
249
30 classes in UCF101 are randomly picked as text conditions. and several aspects are rated (See Appendix for details). For VideoGPT, we use the official unconditonal pretrained mode generate samples. For TGANv2, we use the official source code to train unconditional generation model under the same setting as that Saito et al. 21 To assign unconditionally generated samples into corresponding categories, we choose TSM 13 as the action recognition model and only samples with confidence >80%. Results Figure 3 show that Cog Video significantly outperforms baselines multiple important aspects including frame texture, motion realism and semantice relevance, and achieves the top score by overall quality. can be seen that 49.53% evaluators choose Cog Vidco as the best method, and only 15.42% and 5.6% favor VideoGPT and TGANv2. respectively.
251
252
253
255
257

---

T0AS
Daghhino tilape Ground Pruss
Laytene paaaee
Wend'
(s) Humrnn preference. The chosen best.
(0) Overall scores (1-10) for each method.
1c1 Scores (1-5) three important
Figure 5: Human evaluation results. "CogVideo IStage" refers to the method in ablation study, which generates videos sequentially with Cog Video's 's Stage-1 Model only by recursively reinserting last generated frames into input and generate future frames.
Table 2: Ablation study 5.000-sample subset of Kinetcis-6000 testset. FVD is evaluated
generated 1 11-frame samples priming on 5 frames and ground-truth blurred by our image tokenizer. The setting column indicates the difference between each method and Cog Video. Models of each setting are trained on Kinetics-600 trainset for 10,000 iterations with batch size of 320.
Method
Setting
FVD (L)
CogVideo
None
108.27
1-stage Generation N Noverlap 1)
hierarchical
137.13
1-stage ConnerationNoovertii = 2)
hicrarchical
120.82
Initialzed to CogView2
Pretrain
124.92
Randomly Initialzed
Pretrain
Cog View
166. 13
259
5.3
Ablation Study
To verify the effectiveness of hierarchical multi-frame-rate generation and incorporating Cog View2, we conduct ablation study quantitatively and qualitatively on Kinetics-600 and UCF-101 datasets.
261
262
generation. In comparison with Cog we fine-tune 1-stage video generation model on Kinetics-600 from the sequential generation model Video, which generates long videos by recursively reinserting last Novertap frames into the input to sample next Nowerlap frames. Larger Neverlap means more previous frames can be utilized during the inference, but will increase time overhead.
Hierarchical multi-frame--
263
265
257
Dual-channel attention with View2's weights. We additionally train (1) A randomly initialized model; (2) A model incorporating CogView22 weights but leaving temporal channel randomly initialized and unfixed (equivalent CogVider without pretraining on videos) Kinetics-600.
258
289
270
5.3.1
Quantitative Evaluation
6.23
Cogiliee
271
All aforementioned models have been trained for 11,000 iterations with batch size of 160. Quantitative results are
5.75
5.50
273
shown Table We can see that the hierarchical method clearly superior to 1-stage generation with different N., and model initialized with Cog View2's weights has lower FVD than randomly initialized one.
5.20
4.50
275
4.2
5.75
277
Figure 6 plots the training loss curve of (1) finetuning Cog Video: (2) training model from random initialization;
3.50
279
(3) training model initialized Cog View2 and partially
Figure Training loss ablation study.
fixed. We see that Cog V endows model with

---

Randomiy Inttiaiied
Input Text: Lunge
Given frames:
Finetuned CogVideo, hierarchical generation
CagVidea, 1-Stage
Noverlop
Inttialzed with CogView?
Finetuned CogVideo, 1-Stage Moveriap
Figure 7: Video samples in ablation study, which are generated priming on class label and first frames in Kinetics-600. All samples down sampled by extracting one in every three frames for display purpose. (a) Use fine-tuned Cog Video to hierarchically generate samples. (b) Train a model on Kinetics-600 which is initialized and partially fixed to Cog View2, and hierarchically generate samples. (c) Train a model on Kinetics-600 which is randomly initialized, and hierarchically generate samples. (d)(e) Use fine-tuned Cog Video generate frames in I stage with different N Noverlop
good initialization point from which the loss function can converge faster to lower value. Also, fixing part of the parameters to View2 reduce optimization cost, which gains more than 2x acceleration when using optimization CPU-offload mode in deepspeed.
282
284
5.3.2
Qualitative Evaluation
285
Qualitative comparison shown Figure While model trained from random initialization tends to produce irrational deformation, model incorporating Cog View2 is able to model objects better. And samples generated hierarchically performs better on content consistency and motion rationalization.
287
288
We also conduct human evaluation between I-stage and hierarchical video generation model under the same setting 5.2 As shown in 5] hierarchical model, i.e. Cog Video, outperforms I-stage model on semantic relevance, motion realism as well as texture quality. This is probably because I-stage model tends to constantly generate small movements which make the whole video unrealistie, and if one generated frame collapses, the subsequent frames often suffer from severe degradation.
289
290
292
Conclusion
294
present Cog Video, to the best of our knowledge, largest and the first open-source pretrained transformer for text-to-video generation for the general domain. Cog Video is also the first attempt to efficiently leverage pretrained text-to-image generative model to text-to-video generation model without hurting its image generation capacity. With the proposed nulti-frame-rate hierarchical training framework, Cog Video is endowed with better understanding of text-video relation and ability to control the intensity of changes during generation. We extend swin attention to CogLM, which achieves acceleration in both training and inference. There are still some limitations in Cog e.g. restriction length of the input sequence still exists due the large scale of model and limitation of GPU memory, and we leave them for future work.
296
296
300
302
Broader Impact. This paper aims to advance the open-domain text-to-video generation, which will ease the effort short video and digital art creation. The efficient training method transfers knowledge from text-to-image models to text-to-video models, which helps avoid training from scratch, and thus reduce the energy consumption and carbon emission. A negative impact is the risk of misinformation. To alleviate it, believe the benefits outweigh the downsides.
304
306
train additional classifier to discriminate the fakes. We
308

---

References
309
310
[1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, el. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
312
313
[2] J. Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages
314
315
2017.
[3] J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 01340, 2018.
317
318
[4] A. Clark, J. Donahue, and K. Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: 1907.099000000 06571,,66671,11 2011.
319
320
[5] M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
322
323
[6] M. Ding, W. Zheng, W. Hong, and J. Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204. 14217, 2022.
324
325
[7] Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint (rXXi:20122098441 2020.
(8] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
328
329
[9] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638 2022.
330
[10] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint 1:X6vv4406.200 2014.
333
334
[11] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. arXiv preprint arXiv:2204.00 033458, 2022.
335
336
[12] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1732, 2014.
338
339
[13] J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083-7093, 2019.
340
[14] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022 2021.
343
345
[15] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
346
[16] P. Luc, A. Clark, S. Dieleman, D. d. L. Casas, Y. Doron, A. Cassirer, and K. Simonyan. Transformation- based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.0400 04035, 2020.
348
350
[17] A. Miech, D. Zhukov, J.B.. Alayrac M. Tapaswi, I. Laptev, and J. Sivic. 100to100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
351
352
353
[18] R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and Burnaev. Latent video transformer. arXiv preprint arXiv:2006.10704 2020.
10

---

355
[19] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint 1eXx::2102.1099,, 2021.
356
357
[20] M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830-2839, 2017.
358
359
[21] M. Saito, S. Saito, M. Koyama, and S. Kobayashi. Train sparsely, generate densely: Memory- efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(10):55662600 2020.
381
365
[22] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2234-2242, 2016.
365
388
[23] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,, 2012.
367
368
[24] I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. In ICML' page 1017-1024, 2011.
389
370
[25] Y. Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. N. Metaxas, and S. Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104. 15669,, 2021.
372
373
[26] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489-4497, 2015.
374
378
[27] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018
377
378
379
[28] T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint 1rXiv:1812.01177., 2018.
381
[29] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309-6318, 2017.
383
385
[30] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and 2017.
386
I. Polosukhin. Attention is all you need. arXiv preprint
[31] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
388
[32] X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang. Vatex: A large-scale, high- quality multilingual dataset for vidco-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581-4591 2019.
390
382
[33] Y. Wang, M. Long, J. Wang, Z. Gao, and P. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
383
394
385
[34] D. Weissenborn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models, arXiv preprint arXiv: (etiv::I9000000 02634, 2019.
398
387
[35] C. Wu, I.. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2 /4806, 2021.
398
399
[36] C. Wu, Liang. L. Ji, F. Yang, Y. Fang, D. Jiang, and Duan. N uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111. 2021.
11

---

401
[37] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXi::2144.0015, 2021.
402
403
[38] Yu, J. Tack, Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics- aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10577, 2022.
Checklist
405
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
408
contributions and scope? [Yes]
409
(b) Did you describe the limitations of your work? [Yes]
410
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
412
413
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
415
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
417
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We will release code later.
418
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
421
422
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
423
424
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
426
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
427
(a) If your work uses existing assets, did you cite the creators? [Yes] See footnotes.
428
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [No]
430
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [No]
431
432
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
433
[No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See supplemental material.
436
437
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
438
439
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
440
12


Reviews:
[{"review_id": "tRFC1KuJoxY", "reviewer": "Reviewer_fg6U", "summary": "This work addresses the large-scale pre-training in the open-domain Text-to-Video Generation. The major challenge to this task is the lack of large text-video paired datasets available and the weak relevance between them. In this paper, the authors propose a 9B-parameter transformer model CogVideo which is trained by inheriting the learned spatial semantics from a pre-trained text-to-image model (CogView2 in this paper).\n\nThey claim that the key difference between the text-to-video generation and the text-to-image generation is that the former needs huge of paired data to infer both spatial and temporal correlation between two modalities while the latter only requires the learning of spatial correlation. Therefore, they propose a dual-attention channel that adds an additional attention layer based on original structure of CogView2 to address the learning of temporal correlation (as illustrated in Figure 3). During training, it only optimizes the parameters of newly added temporal attention layer while keep all the parameters of CogView2 frozen.\n\nTo address the weak alignment of text and variable-length video (the example in Line 32-36 gives a good illustration of this issue), they propose multi-frame-rate hierarchical training. Concretely, it involves a two-stage generation process. In stage 1, they propose to add a frame-rate token to the text to generate the image frames at low frame rate. Then in stage 2, they introduce another frame interpolation transformer to generate the immediate transition frames between the generated frames of the first transformer model in stage 1.\n\nThe proposed hierarchical generation framework looks promising on long videos given a text input. The idea of interpolation transformer model that can utilize bidirectional frame context to finish the interpolation of current frame also make sense. However, there are some critical weaknesses and questions listed below.\n", "questions": "**Questions for the authors:**\n1) Why are there no detailed analysis about main experimental results in Table 1 in the main text of the paper?\n2) It seems there are only an ablation study on Kinetics-600. Why not including the ablation study on UCF-101 datasets in Section 5.3? It would be more convincing to cover the comparison results on UCF-101.\n3) In Figure 7, it looks there is no obvious performance gap between (a) and (e). Thus, what’s the advantage of hierarchical generation from the perspective of visualization results, not just in human evaluation results. Could you provide more visualized case analysis in appendix later to justify the superior performance of your proposed method?\n4) In Section 3.2, Is the restricted receptive field formulation in Eq. 4 like the 3D nearby attention mechanism in NUWA [36]? If yes, what’s the difference between two?\n5) How many seconds do the longest videos that your method generate can last? In my understanding, this is the key advantage of the hierarchical generation framework.\n6) How to understand the recursive process of hierarchical generation? Do you mean that immediate interpolated (generated) frames at each iteration will be used to generate new immediate frames between previous ones? Why not directly generating all immediate frames at once?\n\n**Minor issues:**\n1) Could you please add a citation or hyperlink for DeepSpeed in the paper later? Since it is a great work in the community, you should mention it at least.\n2) Grammar error in Line 287, “samples generated hierarchically performs…” => “samples … perform …”.\n\n", "limitations": "The major limitation is that how long the generated videos by the proposed CogVideo can last. The submitted demos show that the generated videos can only last several seconds. It would be more exciting that CogVideo can generate longer videos for given text input.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1) The proposed multi-frame-rate hierarchical training has the potential on the text-to-video generation, especially the idea of frame interpolation transformer model that can utilize the bidirectional frame context to generate immediate frame. It indeed looks a promising solution to handle long video generation given a text input.\n2) The proposed CogVideo inherits the parameters of a pre-trained text-to-image models and avoid the expensive pre-training from scratch. \n3) In addition, the authors claim that they will open-source the proposed large-scale text-to-video generation model. This is another contribution to the community.\n\n**Weaknesses:**\n1) In the main experiment results in Table 1, it seems that CogVideo obviously underperforms TATS-base [9] on UCF-101 dataset and TriVD-GAN-FP [16] on Kinetics-600. This conflicts with the claim in abstract. Is that due to the issue of metric itself or other reasons? Could you explain more about results in Table 1 since I cannot find related information in the main text?\n2) It is not clear how much the temporal attention channel contributes to the performance gains of CogVideo. In the ablation study, it does not include the comparison of with/without temporal attention layer, i.e., compare the performance gap of directly fine-tuning CogView2’s weights and keeping CogView2’s weights frozen while tuning the temporal attention layer.\n3) The qualitative evaluation only includes one case analysis, it would be more convincing to provide more visualized comparison between different variants. Since the visualization samples are much more intuitive than evaluation metrics to directly judge the real performance of the model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "g9VIxpXLuoB", "reviewer": "Reviewer_WUEH", "summary": "This paper is the first work to propose an open-source pretrained transformer to solve the text-to-video task. The authors propose a two-stage framework CogVideo by finetuning a text-to-image model and avoid pretraining from scratch to reduce the training cost. The idea of multi-frame-rate ensures the flexibility and accuracy of the generated video. Experiments and visualized samples show the effectiveness of the method on video generating.", "questions": "1)Is the fps a learnable parameter or a hyper-parameter? And how to use the fps token concretely in the transformer?\n\n2)Compared with other training method, does CogVideo accelerate the training process? What environment and how much time does the training process need?\n\n3)Compared with the 1-Stage finetuned CogVideo model, how does the full model deal with the problem that one generated frame in the sequential generation stage collapses and influences the later frames? I do not have any idea that the two stage pipeline can solve this problem.", "limitations": "No.\nAnd I do not see any serious concerns.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "（1）Strengths：\n\nThe paper proposes the pretrained two-stage Sequential Generation and Recursive Interpolation pipeline and better reconstruct the alignment relation in a video. And the two-stage method does help the model converge better and faster, which is proved by the line chart of training loss shown in the ablation studies. I think the CogVideo pipeline has its value in real-world application scenarios. Also, the paper is well writing.\n\n（2）Weaknesses:\n\n-First, the paper is not the first attempting to investigate hierarchical transformer structure in text-to-video challenge. For example, [1] also exploits autoregressive and interpolation transformers and gain better performance on UCF-101. However, the authors do not explain the difference or innovation compared with [1].\n-The authors have claimed that the CogVideo pipeline can solve the text-to-video generation in the general domain, but the experiment results in tab.1 are all attained on human action video datasets( UCF-101 and Kinetics-600), which maybe somewhat weak to prove the above conclusion. I guess maybe general domain represents the motions in the text have some temporal relation. If the concept of general domain can be defined in the paper, it will be much better.\n\n[1] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638, 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DWtYKM1P0i", "reviewer": "Reviewer_xPD7", "summary": "In this paper, the authors propose a large-scale pre-training model for text-to-video generation, called CogVideo, which is trained by inheriting a pre-trained text-to-image model CogView2. A multi-frame-rate hierarchical training strategy is designed to better align text and video clips. The proposed method is validated on standard benchmarks such as UCF-101 and Kinetics-600.", "questions": "1.The authors should explain the novelty and contributions of the proposed CogVideo.\n2.The performance of CogVideo is not as competitive as claimed. The authors should give more discussions and analyses.\n3.More details about the dataset with 5.4 million captioned videos are expected.", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of the work.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.This work explores an important topic - text-conditional video generation, and presents an effective method for it.\n2.A multi-frame-rate hierarchical training strategy is proposed to improve the generation quality and control the intensity of changes during generation.\n3.The authors verified the proposed model through both quantitative experiments and qualitative analyses.\n\n\nWeaknesses:\n1.The novelty and technical contributions of this work are quite limited. It simply assembles several existing algorithms, such as CogView2 [6] and Swin Transformer [14].\n2.The performance of the proposed CogVideo is not as strong as the authors claimed. As shown in Tab. 1, the metrics of CogVideo (IS and FVD) fall behind previous methods. It is not acceptable to only compare with publicly available models.\n3.The authors trained the CogVideo on a large dataset of 5.4 million captioned videos, but did not give any detailed information regarding this dataset. What are the sources of the videos and captions? How are they collected? Will the dataset be released? \n4.The authors claimed that they greatly enhance parallelism and accelerate inference, but no results about inference speed of the CogVideo model are provided.\n5.In Sec. 5.1, the authors \"fine-tune CogVideo on the whole dataset (UCF-101) for 10,000 iterations\". Is it appropriate in this field? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Bgkf3KH2Ger", "reviewer": "Reviewer_bKXQ", "summary": "This paper proposed a large-scale pretrained text-to-video model and multi-frame-rate hierarchical training strategy to better align text and video clips. ", "questions": "\n1. Is there a gap between the generated video and the ground truth, so ground truth is not placed in human evaluation for comparison?\n\n2. I wonder whether the mixture factor /alpha will finally converge to 0 or 1, making the dual-channel attention block useless.\n", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper are as follows:\n\n1. This paper is well written and easy to follow.\n2. The proposed model is the largest and the first open-source pre-trained transformer for text-to-video generation in the general domain.\n3. The experimental results show that the proposed methods can outperform a number of existing baselines.\n\nThe weaknesses of this paper are as follows:\n\n1. The main limitation of the work is the huge consumption of GPU memory, but the related information (type and number of GPU) is not provided. I hope this information can be given.\n\n\nI am not an expert in this field and do not tap into DALL-E and CogView, I will adjust the final score with other reviewers' comments and the response from the author.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns", "Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Reject