Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work addresses the large-scale pre-training in the open-domain Text-to-Video Generation. The major challenge to this task is the lack of large text-video paired datasets available and the weak relevance between them. In this paper, the authors propose a 9B-parameter transformer model CogVideo which is trained by inheriting the learned spatial semantics from a pre-trained text-to-image model (CogView2 in this paper).

They claim that the key difference between the text-to-video generation and the text-to-image generation is that the former needs huge of paired data to infer both spatial and temporal correlation between two modalities while the latter only requires the learning of spatial correlation. Therefore, they propose a dual-attention channel that adds an additional attention layer based on original structure of CogView2 to address the learning of temporal correlation (as illustrated in Figure 3). During training, it only optimizes the parameters of newly added temporal attention layer while keep all the parameters of CogView2 frozen.

To address the weak alignment of text and variable-length video (the example in Line 32-36 gives a good illustration of this issue), they propose multi-frame-rate hierarchical training. Concretely, it involves a two-stage generation process. In stage 1, they propose to add a frame-rate token to the text to generate the image frames at low frame rate. Then in stage 2, they introduce another frame interpolation transformer to generate the immediate transition frames between the generated frames of the first transformer model in stage 1.

The proposed hierarchical generation framework looks promising on long videos given a text input. The idea of interpolation transformer model that can utilize bidirectional frame context to finish the interpolation of current frame also make sense. However, there are some critical weaknesses and questions listed below.

strengths_and_weaknesses: **Strengths:**
1) The proposed multi-frame-rate hierarchical training has the potential on the text-to-video generation, especially the idea of frame interpolation transformer model that can utilize the bidirectional frame context to generate immediate frame. It indeed looks a promising solution to handle long video generation given a text input.
2) The proposed CogVideo inherits the parameters of a pre-trained text-to-image models and avoid the expensive pre-training from scratch. 
3) In addition, the authors claim that they will open-source the proposed large-scale text-to-video generation model. This is another contribution to the community.

**Weaknesses:**
1) In the main experiment results in Table 1, it seems that CogVideo obviously underperforms TATS-base [9] on UCF-101 dataset and TriVD-GAN-FP [16] on Kinetics-600. This conflicts with the claim in abstract. Is that due to the issue of metric itself or other reasons? Could you explain more about results in Table 1 since I cannot find related information in the main text?
2) It is not clear how much the temporal attention channel contributes to the performance gains of CogVideo. In the ablation study, it does not include the comparison of with/without temporal attention layer, i.e., compare the performance gap of directly fine-tuning CogView2’s weights and keeping CogView2’s weights frozen while tuning the temporal attention layer.
3) The qualitative evaluation only includes one case analysis, it would be more convincing to provide more visualized comparison between different variants. Since the visualization samples are much more intuitive than evaluation metrics to directly judge the real performance of the model.

questions: **Questions for the authors:**
1) Why are there no detailed analysis about main experimental results in Table 1 in the main text of the paper?
2) It seems there are only an ablation study on Kinetics-600. Why not including the ablation study on UCF-101 datasets in Section 5.3? It would be more convincing to cover the comparison results on UCF-101.
3) In Figure 7, it looks there is no obvious performance gap between (a) and (e). Thus, what’s the advantage of hierarchical generation from the perspective of visualization results, not just in human evaluation results. Could you provide more visualized case analysis in appendix later to justify the superior performance of your proposed method?
4) In Section 3.2, Is the restricted receptive field formulation in Eq. 4 like the 3D nearby attention mechanism in NUWA [36]? If yes, what’s the difference between two?
5) How many seconds do the longest videos that your method generate can last? In my understanding, this is the key advantage of the hierarchical generation framework.
6) How to understand the recursive process of hierarchical generation? Do you mean that immediate interpolated (generated) frames at each iteration will be used to generate new immediate frames between previous ones? Why not directly generating all immediate frames at once?

**Minor issues:**
1) Could you please add a citation or hyperlink for DeepSpeed in the paper later? Since it is a great work in the community, you should mention it at least.
2) Grammar error in Line 287, “samples generated hierarchically performs…” => “samples … perform …”.


limitations: The major limitation is that how long the generated videos by the proposed CogVideo can last. The submitted demos show that the generated videos can only last several seconds. It would be more exciting that CogVideo can generate longer videos for given text input.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is the first work to propose an open-source pretrained transformer to solve the text-to-video task. The authors propose a two-stage framework CogVideo by finetuning a text-to-image model and avoid pretraining from scratch to reduce the training cost. The idea of multi-frame-rate ensures the flexibility and accuracy of the generated video. Experiments and visualized samples show the effectiveness of the method on video generating.
strengths_and_weaknesses: （1）Strengths：

The paper proposes the pretrained two-stage Sequential Generation and Recursive Interpolation pipeline and better reconstruct the alignment relation in a video. And the two-stage method does help the model converge better and faster, which is proved by the line chart of training loss shown in the ablation studies. I think the CogVideo pipeline has its value in real-world application scenarios. Also, the paper is well writing.

（2）Weaknesses:

-First, the paper is not the first attempting to investigate hierarchical transformer structure in text-to-video challenge. For example, [1] also exploits autoregressive and interpolation transformers and gain better performance on UCF-101. However, the authors do not explain the difference or innovation compared with [1].
-The authors have claimed that the CogVideo pipeline can solve the text-to-video generation in the general domain, but the experiment results in tab.1 are all attained on human action video datasets( UCF-101 and Kinetics-600), which maybe somewhat weak to prove the above conclusion. I guess maybe general domain represents the motions in the text have some temporal relation. If the concept of general domain can be defined in the paper, it will be much better.

[1] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638, 2022
questions: 1)Is the fps a learnable parameter or a hyper-parameter? And how to use the fps token concretely in the transformer?

2)Compared with other training method, does CogVideo accelerate the training process? What environment and how much time does the training process need?

3)Compared with the 1-Stage finetuned CogVideo model, how does the full model deal with the problem that one generated frame in the sequential generation stage collapses and influences the later frames? I do not have any idea that the two stage pipeline can solve this problem.
limitations: No.
And I do not see any serious concerns.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors propose a large-scale pre-training model for text-to-video generation, called CogVideo, which is trained by inheriting a pre-trained text-to-image model CogView2. A multi-frame-rate hierarchical training strategy is designed to better align text and video clips. The proposed method is validated on standard benchmarks such as UCF-101 and Kinetics-600.
strengths_and_weaknesses: Strengths:
1.This work explores an important topic - text-conditional video generation, and presents an effective method for it.
2.A multi-frame-rate hierarchical training strategy is proposed to improve the generation quality and control the intensity of changes during generation.
3.The authors verified the proposed model through both quantitative experiments and qualitative analyses.


Weaknesses:
1.The novelty and technical contributions of this work are quite limited. It simply assembles several existing algorithms, such as CogView2 [6] and Swin Transformer [14].
2.The performance of the proposed CogVideo is not as strong as the authors claimed. As shown in Tab. 1, the metrics of CogVideo (IS and FVD) fall behind previous methods. It is not acceptable to only compare with publicly available models.
3.The authors trained the CogVideo on a large dataset of 5.4 million captioned videos, but did not give any detailed information regarding this dataset. What are the sources of the videos and captions? How are they collected? Will the dataset be released? 
4.The authors claimed that they greatly enhance parallelism and accelerate inference, but no results about inference speed of the CogVideo model are provided.
5.In Sec. 5.1, the authors "fine-tune CogVideo on the whole dataset (UCF-101) for 10,000 iterations". Is it appropriate in this field? 
questions: 1.The authors should explain the novelty and contributions of the proposed CogVideo.
2.The performance of CogVideo is not as competitive as claimed. The authors should give more discussions and analyses.
3.More details about the dataset with 5.4 million captioned videos are expected.
limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a large-scale pretrained text-to-video model and multi-frame-rate hierarchical training strategy to better align text and video clips. 
strengths_and_weaknesses: The strengths of this paper are as follows:

1. This paper is well written and easy to follow.
2. The proposed model is the largest and the first open-source pre-trained transformer for text-to-video generation in the general domain.
3. The experimental results show that the proposed methods can outperform a number of existing baselines.

The weaknesses of this paper are as follows:

1. The main limitation of the work is the huge consumption of GPU memory, but the related information (type and number of GPU) is not provided. I hope this information can be given.


I am not an expert in this field and do not tap into DALL-E and CogView, I will adjust the final score with other reviewers' comments and the response from the author.
questions: 
1. Is there a gap between the generated video and the ground truth, so ground truth is not placed in human evaluation for comparison?

2. I wonder whether the mixture factor /alpha will finally converge to 0 or 1, making the dual-channel attention block useless.

limitations: None
review_title: null

Meta-review: metareview: The authors address an important problem of text-to-video generation in this work. Although there is a set of solid contributions that are specific to video generation, author-reviewer discussion as well as the following reviewer-area chair discussion did not address all the concerns raised by the reviewers. Furthermore, the lack of and also superficial address of the ethical concerns raised by the reviewers as well as ethics reviewers strongly suggests that the authors need to substantially revise the manuscript along both technical and ethical aspects. The paper is thus recommended to be rejected.
confidence: Less certain
Decision: Reject