Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work tackles the tasks of video generation, prediction and interpolation using a score-based diffusion modeling approach. The main innovation is in using diffusion models to block-wise autoregressively generate/predict/interpolate video both forward and backward in time efficiently (i.e. using <4GPUs in <12days).
strengths_and_weaknesses: Strengths:
1. This paper is well written, clear and provides a straight-forward extension of the diffusion modelling approach to video.
2. The proposed approach is more computational efficient than concurrent works in the literature (i.e. using <4GPUs in <12days).
3. Technical contributions include a more efficient blockwise generation mechanism and the ability to interpolate between frames. Concurrent works use similar architectures and are capable of video generation and prediction. This work combines all of these within one architecture.

Weaknesess:
1. This work misses comparisons to prior video GAN based work. Taking the UCF-101 experiments as an example, although this work motivates the case for focusing on FVD as a metric, prior work utilises FID and IS as metrics. Not evaluating on these metrics makes it difficult to compare this work against prior art. For historical comparison, I would strongly encourage the authors to extend or append their results to a table similar to the one found in other concurrent/prior works (e.g. Table 1 from the [Video Diffusion Models](https://arxiv.org/abs/2204.03458) or [DIGAN](https://openreview.net/forum?id=Czsdv-S4-w9) papers)
2. This work is limited to 64x64 generation. Many GAN based approaches are more performant and efficient at this resolution. The 64x64 generation should be made explicitely clear in any results/discussions, especially when comparing results to other works. [TGAN-F/LDVD-GAN](https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397) discussed how results from these metrics are dependant on resolution and the pre-processing pipeline.
3. Although an interesting direction for diffusion models, the novelty of this work in terms of new capabilities is limited when compared to prior art or GAN-based approaches (e.g.  [Deep Video Generation, Prediction and Completion of Human Action Sequences - CVPR 2018](https://arxiv.org/abs/1711.08682)).
questions: BAIR, KTH, UCF-101 are some of the standard video generation benchmarks. Although the results tables appear to be quite exhaustive for BAIR/KTH, UCF101 is lacking in comparison... which matters since it is the standard benchmark for unconditional video generation. Could you improve the UCF-101 benchmark by making the experimental procedure clearer (see Appendix A.2 in the [DIGAN](https://openreview.net/pdf?id=Czsdv-S4-w9) paper), also do provide FID and IS results on this dataset, appropriately presented to allow for easy comparison to prior work such as progressiveVGAN,LDVD-GAN/TGAN-F,TGAN-ODE, etc.


The introductory paragraph touching on vehicle safety is an interesting way to introduce the problem, but it isn't touched upon beyond the first paragraph. In the end, how does your work impact this?

limitations: More effort and thought has to be put into the broader impact statement. Two generic sentences on broader impact is unacceptable for this kind of work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces the modified U-Net architecture for taking video frame features and the conditional diffusion process for frame-level image generation. Modified U-Net architecture is equipped with conditioning components with non-recurrent architecture fusing frame-level features. The space-time fusing process is conducted in two-step; channel-wise concatenation of frame-level features and space-time adaptive normalization (SPATIN) inspired by SPADE. This non-recurrent and space-time conditioning U-Net architecture is evaluated on the several video generation, prediction, and interpolation datasets. Although the scale of the datasets is restricted, the versatility of the algorithm is validated. 
strengths_and_weaknesses: ### Strengths
- The author validates the versatility of the proposed neural diffusion architecture for video generation on several video generation tasks--- unconditional generation, prediction, and interpolation.

### Weaknesses
- The scale of the datasets used in the experiments is small-scale compared to the other generation methods these days.
The disadvantage of non-recurrent and sliding window sampling is not explained. The word ‘non-recurrent’ only appears in the abstract.

questions: ### The effect of $k$ on the proposed algorithm
The experiments only show the maximum number of frames during training is 4, 5 for video prediction/generation, and 10 for interpolation. Compared to the other model with $k=10, 15$ for training, I guess that the proposed model seems to show better performance on lower $k$ as explained in lines 257~260. How about the smaller or larger number of frames affect the performance of the proposed algorithm?

### Diversity issues in video generation/prediction
What is the diversity of the generated videos? Most video generation works only report fidelity metrics, such as FVD, FID, and IS. However, the fidelity metrics do not show the diversity of the generated videos. The diversity metric introduced in [1] would be good.

[1] Diverse Video Generation using a Gaussian Process Trigger, ICLR 21

### The scale of the dataset
The experiments only contain small-scale datasets. How about applying the proposed algorithm to a larger scale dataset, such as Kinetics, Something-something v2? Is there any bottleneck of the proposed algorithm except the amount of computation? An algorithm with a non-recurrent and sliding window sampling may be weak in this case compared to recurrent counterparts. 

### Time coverage of snippets.
Several works typically count the number of frames, not the time span measured in seconds. It would be good to the time span of the video snippets for generation, prediction and interpolation.

limitations: I’ve mentioned it in the question part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed a DDPM based model for video prediction, called MCVD, based on randomly masking frames for conditioning. The random masking allows the model to be conditioned on past or future frames, enabling the model to perform a variety of tasks such as infilling, prediction and generation. Authors compared the performance of the proposed method in conditional (on BAIR and Cityescapes), inpainting (on SMMNIST, KTH and BAIR) and unconditional setup (on BAIR and UCF-101), demonstrating its capability on generating high quality videos.


strengths_and_weaknesses: Strengths And Weaknesses:
===== Strengths
- The paper is well written and is easy to follow. 
- The authors compared the proposed method to many other models in conditional and unconditional setups.
- The proposed method is novel (however there are multiple similar parallel works).
 
===== Weaknesses:
- Limited datasets. Although, authors tested the proposed method on multiple datasets, I find the choice of datasets to be narrow. In both prediction and unconditional generation, the datasets are limited to smaller size dataset (BAIR and UCF) and there is no experiment on larger datasets such as RoboNet or Kinetics. This makes me wonder if the model is heavily overfitting on the training data and there is not much generalization. FitVid paper (Babaeizadeh et al) has an interesting discussion on how overfitted methods can generate realistic high quality videos but are not capable of generating anything new. 

- Analysis. There is limited analysis and ablation studies. While the paper shows great and promising results on video tasks, it is not clear how the design choices affect its results. e.g. why the masking should be random for inpainting? In absence of any ablations, I can argue that a fixed set of conditioned frames is a much better defined problem and may result in even higher quality predictions. Table 3 has some for masking strategies but the paper can use more targeted experiments along the same lines. Another angle is computational issues which is mostly ignored in the paper. One major issue in video prediction literature is the huge number of parameters required for getting good results. How is the situation in a score based model? Does a bigger model substantially improves the result or not? Are these models already overfitting? Such analysis can substantially improve the quality of the paper.

- Limited qualitative results. Given the nature of videos, they are hard to compare on printed paper. I highly recommend authors to host predicted videos in an anonymous website and provide comparative videos with other baselines.
questions: Please check weaknesses. 

limitations: Yes
review_title: null

Meta-review: metareview: The paper proposes the use of diffusion model for masked video modeling and shows promising results in video generation and completion. All of the reviewers agree that the paper is a good fit for publication at NeurIPS. I appreciate that the authors engaged with the reviewers and improved the paper!
confidence: Certain
Decision: Accept