Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on text-to-video generation, but without having text-video pairs. To achieve this, the proposed method relies on a pre-trained text-to-image generation model, and then further extend it to generate a video, semantically aligned with the given text.
strength_and_weaknesses: Strength:
1. The paper is well written and easy to follow.
2. The proposed method shows better quantitatively results, compared to the baselines, and also have a good human evaluation.
3. The training of proposed method does not need text-video pairs, which can avoid the difficulty to have a text-video dataset.

Weaknesses:
1. Basically, the proposed method relies on many previous works: (1) the T2I model used in the proposed method is DALL-2, (2) the proposed Pseudo-3D convolutional layers is based on separable convolutions (Chollet, 2017), actually a similar idea to separate spatial and channel information to reduce computational cost have been widely adopted in different areas, and (3) as mentioned by authors, factorized space-time attention layers have also been used in VDM (Ho et al., 2022) and CogVideo (Hong et al., 2022). 
2. Compared to CogVideo, the better performance achieved by the proposed method might benefit from a powerful pretrained T2I model (i.e., DALLE-2). If authors use CogView as the T2I model, same as CogVideo, could the proposed method still have a better performance?
3. I am confused about the quality of synthetic video results, as authors only show a small number of frames for each video results, and the frames in the same video only show limited differences between each other. Also, current text-to-image generation methods, such as DALLE-2, can produce loads of images with some differences, and combine these results together, we can also observe some meaningful story from them. 
4. Authors claim their method accelerates training of the T2V model, but there are no experiments to support this claim.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to follow. The proposed method is reasonable. Considering the simplicity of the method, the reproducibility should be high.
summary_of_the_review: Although the proposed can produce a good results without using text-video pairs, it relies on many previous works and has limited differences between them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces Make-A-Video, a text-video generation approach trained based on a text-image model. The core idea of Make-A-Video is to take use of learned text-vision correlation from well-trained text-image model to accelerate the learning of text-video generation. It also claims that no paired text-video data is required for video generation. The quantitative results on MSR-VTT and UCF-101 are state-of-the-art and better than CogVideo and VDM, which are two main baselines to compare. More qualitative examples are provided to elaborate the motion consistency and richer content in the generated videos from Make-A-video.
strength_and_weaknesses: Strength:
1. Novel idea on training text-image and image-video separately to bypass paired text-video data;
2. Well-motivated to use text-image model for text-video generation;
3. State-of-the-art results;

Weaknesses:
1. The main idea is novel to combine text-image generation and image-video reconstruction; however, text-image pipeline is exactly same as previous work, and it's hard to say the components for image-video generation are novel as they are commonly used.
2. Results and analysis are not sufficient. The results are state-of-the-art but it's not clear where the improvements come from. At least some quantitative results on ablation internally should be provided to make the model less blackbox. For example, how much improvement does FRAME INTERPOLATION NETWORK bring in terms of the evaluation on MSR-VTT and UCF-101, or more qualitative examples to show the difference in the generated videos w/ and w/o the module. Otherwise it's really hard to understand how and why the model works.
3. Some claims are unclear and not well-supported: In Previous Work, "Second, we fine-tune the T2I model for video generation, gaining the advantage of adapting the model weights effectively, compared to freezing the weights as in CogVideo". This is not completely true: first, only some components of T2I model are tuned for image-video generation, starting from Dt; second, some more results should be provided to prove that in Make-A-video, fine-tuning the weights in T2I is better than freezing them, other than comparing with CogVideo.

clarity,_quality,_novelty_and_reproducibility:  1. The main contribution of this paper seems from image-video generation, where the input images are sampled from the videos, as in 3.4; no contribution and novel components in text-image part. Please clarify this.
 2. In 3.2, "In qualitative inspection we found this to significantly outperform per-frame super resolution", more qualitative examples should be provided for this claim.
 3. It seems SRh is completely same as in (Ramesh et al., 2022). Make-A-video only operates on multiple frames separately to increase the resolution. If so, the contribution as in the claim "high resolution" is not really from this work but from the text-image model in (Ramesh et al., 2022). Please clarify.
4. In training details, "The decoder, prior, and two super-resolution components are first trained on images alone (no aligned text)." is not clear: is this a step for image_embedding-to-high_resolution training after text-image training? The main question is how the training steps are designed in order. Is it correct (1) text-image training; (2) image_embedding-to-high_resolution training; (3) image-to-video training with proposed layers? Please add more details to clarify as this should be super important to the final results.
5. Some training details are missing: the objective to train video generation; also for text-image generation. It's true that many works use them commonly but it's worth mentioning e.g., which objective function is used for the training of the components.
6. How inference works is missing. During inference, given a text, how to generate multiple frames and generate the video. Details should be provided.


summary_of_the_review: Overall, the paper introduces an idea to break the dependency on text-video pairs for text-video generation and achieves state-of-the-art performance. However, some training and inference details are missing; more analysis on the components of the model internally should be provided to elaborate why the model works, as it seems all quantitative and qualitative results are compared with other approaches but internal analysis is missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Make-A-Video, an approach for generating short video clips conditioned on a given open domain text. The model does not need any text-video pairs and instead relies on learning dynamics from unlabeled videos and applying them to existing text to image systems. The approach also applies multiple levels of super resolution, both temporal and spatial, to increase the resolution of generated videos. 
strength_and_weaknesses: ===== Strengths
+ The main idea behind the paper is well motivated and an interesting one. There are no massive dataset of labeled video datasets (yet!) while it is relatively easier to collect large amount of raw videos. 
+ The paper is easy to read and understand. The notations are clear to follow which makes it easier to follow what it going on. 
+ There are many qualitative examples on the appended website: https://gen-videos.github.io/ This is always a big plus for video papers.
+ While the quantitative benchmarks for video generation are quite weak, authors did their best to compare with previous work in a fair setting.

===== Weaknesses
- Missing details. Unfortunately, there are *many* missing details in the paper which substantially reduces its impact on the field. The biggest one imho is the loss. While the system consists of many smaller parts, trained separately, it is not clearly what the loss for any of these components are. In fact, the term "loss" or "objective" which the authors heavily relied on to optimize, does not appear in the paper even once! This is really interesting for a deep learning paper. While I understand that creating such systems (unfortunately) is limited to big industrial labs, reproducibility still should be a big concern. 

- Quantitative results are exaggerated. It is no secret that the benchmarks for video generation are still quite weak. The metrics (e.g. FVD) are inconclusive and unrepresentative of the differences. While I appreciate the authors effort to provide a large set of videos in their website, as well as a short human eval, it seems the quantitative results in Table 1 and 2 are exaggerated and in absence of evaluation details it is hard to check that clearly.  Particularly, FVD is sensitive to resolution but this paper compares different resolution without mentioning how.  but how? Higher resolution usually results in better FVD (because of I3D resolution)  and therefore comparing Make-A-Video at 256x256 to e.g. CogVideo at 160x160 or VMD at 64x64 is not a fair comparison. Also for CogVideo, it is not clear where the videos came from, because the official published version is only in Chinese and as of writing this review, the official github page clearly mentions that "Currently only Chinese input is supported." To make things even worse, the provided CogVideo videos in your website,  are significantly worse compared to CogVideo's official page. Again, missing details here severely affects the quality of the paper.

- Lack of analysis. The main idea of learning dynamics from a large corpus of videos and then applying them to generated images is interesting and novel to the best of my knowledge. However, like any other system with multiple components, it has clear down sides compare to an e2e system. First of all, the dataset of videos that the authors used (WebVid) has labels. The paper clearly mentions that it did not use these labels but this raises the question of why? The paper does not provide any analysis on how using more videos improves the system and if 10M labeled videos was enough to learn dynamics then the motivation of the paper is questionable. Second, the paper does not provide any analysis on the weaknesses of the proposed method. I would assume a system that learns the dynamics separately will have weak correlation between dynamics and the terms to describe them (e.g. walking jumping dancing etc), is this the case? if not why not? Where does the alignment come from? Or as another example, how long the videos can be before becoming incoherent? The website includes a few examples of "long" videos however it's not clear how far the model can be pushed and what its limitations are. In absence of any such analysis and the fact the code/model is behind tall walls, it is hard to answer such questions.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is clear and easy to read, however substantial details are missing
- Quality: The paper achieved some high quality results which can push the field forward.
- Novelty: The main idea (learning and applying dynamics on top of T2I systems) is interesting and novel.
- Reproducibility: The paper gets poor score on reproduciblility. There are many missing details and the computational requirements are omitted.
summary_of_the_review: Overall, while the results of the paper is impressive, it lacks substantial details which makes it close to impossible to reproduce, no analysis to judge the limitations of the approach and exaggerated quantitative numbers. Therefore, the paper should be improved, scientifically, to be acceptable at a major scientific conference, imho. Particularly, more details, more analysis and fairer comparison.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a text2video method, which builds on image2video model with spatial-temporal modules.
The method is straight-forward and the results are appealing.
strength_and_weaknesses: **Strength**

1) This paper is well-written and easy to follow.

2) The method is simple but effective.

3) The results are compelling.

4) This work uses solely open-source datasets, making it easier to reproduce.

**Weakness**

1) A relatively comprehensive benchmarks. The automatic quantitative evaluation is only done in MSR-VTT and UCF-101. The class and coverage is somewhat limited. Since the results on general situation is more for fun rather than the practical application, some domain-specific evaluation is wanted, to evaluate the practical application ability of the proposed models. For example, text2face, text2human or text2scene, which could has broad application scenarios. Thus, a more comprehensive benchmarks, especially for domain-specific evaluations, is wanted.

2) A absolutely fair comparison to SOTA. For the results on Table 1 and 2, the resolution, number of parameters are different for these compared models. It makes the comparison with SOTA of IS and FID seem to be weak, since other factors are neglected. A absolutely fair comparison benchmark is also wanted.

3) The precise control of action in sequence. It will be better if authors can show the results for text2face/text2human, which is more challenging than general situations, since the more subtle action should be generated. Could the model control the action precisely, for example, with the text: a young man first smiles, then nod his head and finally turns left to speak to others. Will the subtle action and the sequential relation be faithfully maintained?

clarity,_quality,_novelty_and_reproducibility: This work uses solely open-source datasets, making it easier to reproduce. Will the code and model be released?
summary_of_the_review: This work presents a new method for text to video generation, which is a really important problem. The method is effective and the results are compelling. In general, I like this paper and recommend for acceptance so far.

The main concerns are listed in Weakness above. Although some of the concerns are not specific for this paper but for this research domain, which lacks standard benchmark, It will be great to see the opinions and discussions for this problem in this paper.

Many previous methods are more likely to show off the results rather than make it as a research problem. This paper tried to use open-source datasets, which has started to consider the reproducibility for the community. I really appreciate this awareness.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a text-to-video generation framework. It builds upon advances in text-to-image synthesis and brings the generated images alive by learning motion from a large corpora of unlabeled video data. The proposed architecture with a spatiotemporally factorized diffusion model is evaluated extensively using automatic metrics and human evaluation. 
A strong rebuttal has cleared out several critical issues raised by the reviewers; hence two reviewers have raised their scores. Also the rebuttal has addressed the concerns raised in the review with a below borderline score. AC recommends acceptance of the paper, and urges the authors to polish their final version with any remaining questions. Congratulations to the authors!
justification_for_why_not_higher_score: It's a good poster.
justification_for_why_not_lower_score: Two out of four reviewers have raised their recommendation after the rebuttal to 6 and 8. The concerns of one reviewer that put 5 have been addressed in the rebuttal. AC agrees with the majority of the reviewers and recommends acceptance.
Decision: Accept: poster