Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This article presents, Phenaki, an approach to text-to-video modelling and synthesis based on an autoregresserve transfomer-based encoder-decoder model. 

# Stage 1. C-ViViT:
Frames from a given video are first linearly projected into patch-embeddings then processed by a spatial transformer; the resulting output of the spatial transformer is further processed across time with a causal (temporal) transformer. 

# Stage 2. MaskGIT
The outputs of the causal transformer are project to a discrete latent space via a learned codebook. A bidirectional transformer is trained via a self-supervised reconstruction mechanism (masking) to reconstruct sequences of the tokens output by the causal transformer. This is done by taking the discretized outputs of stage 1 and randomly masking (according to some procedure) parts/subsets of these causal tokens. The bidirectional transformer is trained to undo the masking operation and reconstruct the original unmasked token sequence. The bidirectional transformer also conditions this reconstruction process on text information (ie. prompt) provided by consuming embeddings provided by a pretrained language model.

# Stage 3: Text driven Video Generation
Once all these pieces come together, generation occurs via providing empty/blank tokens and a text prompt to the bidirectional transformer. The bidirectional transformer outputs a set of tokens that align with those in the codebook, these tokens are projected back to the image space by an image decoder that is the inverse of the ViViT procedure and architecture. Auto-regressive generation can be achieved by shifting the tokens generated by the bidirectional transformer forward by one time-step, appending an empty/blank token for the new time step and providing that to the bidirectional transformer along with the text prompt.

strength_and_weaknesses: ## Strengths:
1. The chosen approach allows for unprecedented flexibility and performance in conditional video generation tasks.

2. It is a simple extension of existing architectures from other domains (video classification, image generation)

3. A good awareness of the ethical considerations with regard to this kind of research and its potential misuses.


## Weaknesses: 
1.  **Technical Novelty**: The technical novelity in this article is quite limited. Phenaki is essentially a merger of MaskGIT and ViViT (and T5), the ensuing capabilities and advantages of this model come from the capabilities of the base models being used together. The technical novelity in the paper is limited to intelligent architectural choices. Of note is the addition of text conditioning to the bidirectional vision transformer and the change towards a causal transformer for better temporal modelling of video, allowing for unconstrained video generation.

2. **Code (and Demo) is not available**: The model itself is not dangerous but its uses could be. I would agree with keeping the trained models closed source since they could immediately be misused by the lay individual. But keeping the code closed source does not benefit the field as a whole and prevents researchers, reviewers and other academics from carrying out accurate experiments, bechmarks and comparisons. There are already several attempts at reproducing the model already. It would be interesting to see what the ethical reviewers have to say with regard to this argument. For the sake of accurate benchmarking and comparisons, I would urge the authors to open source their code since the stated reasons for keeping it closed source (i.e. "understanding of data, prompt and output filtering") are not impacted by this.

3. **Clarity**: Figure 2 is incomplete and the explanation of encoder-decoder model unclear.  It can be argued that the decoding process is one of the most important procedures in this work yet it is barely covered. It is unclear whether decoding and generation primarily happens in the latent space? are the outputs from previous timesteps decoded and then re-encoded and fed into the bidirectional transformer? can the entire decoding process for several timesteps purely occur in the latent space? 

4. **Missing Sections** : The computational requirements and cost of training such models is of increasing interest to the wider community. There are references throughout the paper to computational cost and efficient but this topic is never fully explored. It would be good to get some idea of computational requirements, from the equipment used for the experiments to (tpu/gpu hrs/days, flips), to wall-clock inference speed and time (per frame, time-step, etc). This is not in the article or its associated supplementary material.

5. **T5X**: There is mention of discussing each of the model components in the paper in the last line of the first paragraph in section 2. But T5X is not elaborated on. What is T5X? Is this the original T5 model repackaged? Is it the T5-XXL (4.6B) model?

6. **LAION**: Mentions of the dangers of the LAION-400M dataset are provided without reference. The work that raised this issues should be directly referenced within that paragraph (ie. [Multimodal datasets](https://arxiv.org/abs/2110.01963)).

7. **UCF-101** : The standard UCF-101 benchmark is missing and should be included. It is a tractable and accessible benchmark that allows for better comparison to prior art within the video generation field. I encourage the authors to have a look at Table 1.d in the [TATS](https://arxiv.org/abs/2204.03638) paper. The model presented in this work should be evaluated on base level video generation ability and class-conditional video generation capability. Whether the text prompt is a sentence or a word/action should not present a challenge but such benchmarks are useful for evaluating base level capabilities of such models. In the case of using pre-trained models, it should be highlighted that the model is pre-trained on a range of other datasets rather than the standard practice of training on only the benchmark dataset. It would also be useful to know of its base-level long horizon [video prediction](https://wilson1yan.github.io/teco/) capabilities and dynamics.



### Missing References:
The field of video generation has existed for some time now and was long established within the GAN world. References to this prior art are missing. These include but are not limited to: 

- [Long Videos of Dynamic Scenes](https://arxiv.org/abs/2206.03429)
- [RV-GAN](https://openaccess.thecvf.com/content/CVPR2022W/WiCV/html/Gupta_RV-GAN_Recurrent_GAN_for_Unconditional_Video_Generation_CVPRW_2022_paper.html) - CVPR-W 2022
- [TATS](https://songweige.github.io/projects/tats/index.html) - ECCV 2022
- [StyleGAN-V](https://openaccess.thecvf.com/content/CVPR2022/html/Skorokhodov_StyleGAN-V_A_Continuous_Video_Generator_With_the_Price_Image_Quality_CVPR_2022_paper.html) - CVPR 2022  
- [LDVD-GAN](https://www.sciencedirect.com/science/article/pii/S0893608020303397) - Neural Networks 2020
- [DIGAN](https://arxiv.org/abs/2202.10571)

I would recommend the authors have a look at Section 2, paragraph 2, of the [DiGAN](https://arxiv.org/abs/2202.10571) paper covering prior video generation work.

**Concurrent work** such as [Imagen](https://arxiv.org/abs/2210.02303) and [Make-a-video](https://arxiv.org/abs/2209.14792) should also be refenced in the revised article.

clarity,_quality,_novelty_and_reproducibility: - **Clarity**: The explanation of certain parts of the model are lacking and confusing. T5X is not elaborated on and I found myself referencing the original ViViT paper for illustrations and explanations since Figure 2 was unclear. What are the subscripts for and which dimension they operated over, etc.  It is unclear if the left C-ViViT illustration on figure 2 focuses on one spatial patch position over time? Additionally, the video generation process is unclear, specifically how decoding to the image space occurs for the different tasks demonstrated.
- **Originality**: Although the technical novelty of this work is low, the ingenius approach to combination of these tools for the purposes of video generation is novel and unique. It brings to video generation several capacities that have existed apriori on their own, but never in one model. Those being unconstrained long-horizon video generation, long form text conditional generation/prompting, fast generation, online video steering, video infilling/painting, and optional pre-training on image datasets. This combination of attributes do not exist in prior work making this article very significant.
- **Quality**: The engineering quality of this work is high, but the written article requires more work.
summary_of_the_review: Overall this is a great piece of work and a significant contribution to the field. If the authors remedy the issues i raised as weaknesses then I will increase my score. But as is, I support the publication of this work at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new video generation method from given text prompts, called Phenaki. The proposed method address four tasks such as text conditional video generation, text-image conditional video generation, time-variable text-conditioned video generation, video prediction, and image conditional video generation. For achieving this, the authors advanced ViViT to C-ViViT and designed Transformer-based decoder. They compare their method with multiple state-of-the art methods including NUWA, NUWA-Infinity, and CogVideo. The results look promising. 

strength_and_weaknesses: ### Strength
- Video generation from text prompts is a challenging and innovative research topic. In particular, time-variable video generation from text condition is a very meaningful achievement.
- The proposed method including C-ViViT and Transformer-based decoding seems a little novel. C-ViViT looks beyond simple extension of ViViT.
- The authors provide the results of five meaningful tasks, which are very promising and impressive.
- The paper is easy to read and clear.

### Weakness
Overall, I like this paper. But there are some room for improvements.
- [Major] Despite its impressive results, more ablation studies or analyses on key hyperparameters are required. For example, the number of video tokens seem have influence on the quality and compuational cost (both training and inference). The trade-off analysis is helpful. The results on maksing ratio and predicted token ratio can help to understand the model properties. 
- [Major] In addition to quality metrics, the results on inference time can enhance the contributions. 
- [Minor] How is the performance on few shot or fine-tuning of Phenaki in Table 1?
- [Minor] How long can the model consistently or stably generate videos for prompt sequences when generating time-variable generation?
- [Minor] Why did the author use 50%/50% setup for Table 1 instead of 80/20 setup even if 80/20 is better than 50/50?
- [Minor] TATS [Ge et al. 2022] needs to be added in related work. The comparison is not required because it is published in ECCV 2022. Also, because the authors present image conditional video generation, some video generation works need to be added in the related work such as StyleGAN-V [Skorokhodov et al. 2022] and DIGAN [Yu et al. 2022].

### References
- Ge et al. [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://arxiv.org/abs/2204.03638). ECCV 2022.
- Skorokhodov et al. [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://arxiv.org/abs/2112.14683). CVPR 2022.
- Yu et al. [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://arxiv.org/abs/2202.10571). ICLR 2022.

clarity,_quality,_novelty_and_reproducibility: ### Clarity
- Overall, this paper is easy to read and understand. But the organization might be improved. I recommend using a subsection of evaluation metrics for experiments. Also what is the difference between FID video in Table 1 and FVD? CLIP means CLIP score?
- The reference style is different from the default ICLR style. Please check it. 

### Novelty
C-ViViT is beyond a simple extension of ViViT and the authors proposes some interesting ideas. 

### Reproducibility
Even if the authors explain some details of implementation and experimental configurations, it is not easy to reproduce because they did not present the detailed information on video data. Also, they did not submit the source code and write reproducibility section. So, reproducibility is restricted.
 

summary_of_the_review: Overall, I like this paper. Although there are some room for improvement (please see the weakness), I decided to give accept as score considering the meaningful contributions of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces masked visual token modeling (MVTM) for video generation by extending ViT-VQGAN and MaskGIT into the video domain. The modified ViT-VQGAN tokenizer for video has two blocks with spatial- and temporal attention. To mitigate the nature of fixed code resolution in image (or video) tokenizer, it modifies full attention in the temporal blocks into causal attention. MaskGIT-like masked token modeling learns the distribution of videos represented as the sequence of tokens. The novelty of this work lies in the joint leaning on image and video datasets. Examples in the paper demonstrate that those combination generalize well on 
strength_and_weaknesses: ### Strength
**appropriate modifications of previous works for video generation**

This paper modifies ViT-VQGAN and MaskGIT for video generation. These modifications seem to be plausible to process the video data. In the aspect of masked visual token modeling for video, this approach is fancy and new although implemented architectures are borrow from the previous works

**joint learning of image and video datasets**

This work demonstrate how to use both image and video datasets for video generation. I think that this idea is crucial for future research on video sythesis. I hope that there will be more analysis on this idea.

### Weaknesses

**validity of the statement (novelty over the ViViT architecture)**

I think that the descriptions in the last 4 sentences in **Novelty over the ViViT architecture** are wrong. The main statement here insists that causal attention is needed to support arbitrary temporal length. As I understand, C-ViViT architecture with all-to-all attention (I understand bidirectional attention) still supports encoding and decoding for arbitrary temporal length. (If it is not the case the bidirectional transformer can not model the masked token sequences.) If my conjecture is right, this work should discuss the difference between all-to-all and causal attention. For example, comparing the reconstruction performance differences between two attentions.

In this aspect, long video generation is not related to this video tokenizer, it is a matter of auto-regressive generation using bidirectional transformer inspired from MaskGIT. These confusing descriptions appear several times on the paper. Do I understand wrong about the proposed method?

**analysis on joint learning of image and video datasets**

It is not a weakness but suggestion. If possible, analyze the vocabularies for image and video dataset. For example, we can use this analysis to generate video with the text with out-of-vocaburary in video dataset and in-vocaburary in image dataset and check the quality of this video.
clarity,_quality,_novelty_and_reproducibility: ### Clarity and Reproducibility
There is no problem with clarity and reproducibility because the proposed architecture is a straightforward extension of the existing architecture. If the checkpoints for the reported performance on experiments are released, the reproducibility of the works will be better. 

### Quality 
The quality of the paper can be improved better by explaining details of the imported architecture and training, sampling technique in the paper. How about adding preliminary parts for explaining those?

### Novelty
Novelty of this paper is clear. Although it borrows architectures from the several previous works, the combination itself is non-trivial and has a impact on video generation task. 
summary_of_the_review: This paper introduces a straightforward extension of masked visual token modeling for the video domain. By designing proper settings for video, the proposed method can generate high-quality, diverse-length videos from the various text description. The current version of the paper makes me confused about some details of the works, However, I believe that the overall work is quite fascinating for future research on video synthesis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies text-to-video generation. It proposes an architecture based on the VQGAN framework. For the encoder-decoder part, a novel ViT-like model is introduced to encode both videos and images into discrete tokens, which allows joint training on the text-image dataset. For the sequence modeling part, a MaskGiT-like model is trained to predict video tokens. The authors show the effectiveness of the proposed framework as well as joint training on image / text conditional generation with qualitative and quantitative evaluation. 
strength_and_weaknesses: [Strengths]

1. The idea to extend ViViT to C-ViViT for encoding videos and images into discrete tokens is reasonable and novel.
2. Adapting the MaskGiT-like framework to video VQGAN models is novel.
3. The ablation study of the image-video joint training is valuable to the community.
4. The quality of the generated short and long videos based on texts is much better than the previous arts.

[Weaknesses]

The paper misses an important baseline [1] which also shows results in generating videos based on variable prompts. [1] shares a similar goal of generating long videos with a similar video VQGAN architecture. Therefore, I think it is important to discuss and compare this method, especially on the following points:

1. [1] also leverages a VQGAN framework for video generation by extending the auto-encoder part with 3D convolutions. It would be necessary to compare with it in Table 3 on video reconstruction. Currently, only 2D convolution and per-frame ViT models are compared. It is not clear that C-ViViT works better than 3D convolution.
2. [1] also show video results conditioned on variable length of text prompts, which means the argument that this paper is the first to study this problem is not true. It is more reasonable to say this is the first to study generating videos from variable lengths of **open-domain** text prompts, since I believe [1] only shows results on a limited domain. 

Including the above discussion should make the paper more complete and a stronger submission. In addition to the concern above, some of the experimental results in the paper are not convincing:

1. The authors claim that the video dataset does not contain any stylized videos like pencil drawings which makes such videos impossible to be learned solely from the video dataset. However, in the shown qualitative results, some difficult prompts like "astronaut riding a horse" can still be generated on Mars. Can the authors show examples of the "pencil drawing styles" with a model trained on video datasets only?

2. One advantage of using MaskGIT claimed in the paper is the speed of sampling longer sequences. However, there are no experiments showing that, and no ablation on the hypermeters $\gamma$, $\beta$, and sampling steps which are claimed to be important in the paper. 

[1] Long video generation with time-agnostic vqgan and time-sensitive transformer. ECCV 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The quality is affected by the missing baselines and necessary experiments. The architecture and video results are novel.  
summary_of_the_review: This paper shows improved results on open-domain text-to-video generation using VQGAN-like framework. It shows impressive results when conditioning on a variable length of open domain texts. However, the paper misses important baselines and contains some unconvincing yet tempting conclusions. If the authors can resolve these concerns and add the corresponding experiments and discussions, I would like to change my current rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces Phenaki, a new encoder-decoder model for open-domain text-to-video generation based on the proposed C-ViViT and MaskGIT training. 

Initially, this paper received mixed scores. After author rebuttal, the scores have changed to 6668. All the reviewers are generally happy about the paper, and recommend acceptance. Specifically, all the reviewers agree that (1) the method is novel and unique, and (2) results are impressive with unprecedented flexibility and performance in conditional video generation tasks. During rebuttal, the authors have further provided additional comparison with TATS by Ge et al. 2022. On the other hand, the writing quality and paper clarity can be further improved, more ablation on the hyper-parameters of MaskGIT can be added, and reproducibility is restricted.

All reviewers agree to accept the paper, and the AC also thinks that this paper makes solid contributions, and would like to recommend acceptance of the paper. 
justification_for_why_not_higher_score: The writing quality and paper clarity can be further improved. More ablation study on the hyper-parameters of MaskGIT and some other aspects of the model can be added. Since code and pre-trained data cannot be released for now, the reproducibility can be restricted.
justification_for_why_not_lower_score: All the reviewers recommend accept, the AC also thinks that the paper makes valuable contributions, there is no reason to reject the paper.
Decision: Accept: poster