Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper extends the image-based diffusion model methods into the video domain. It builds upon the 3D U-Net architecture with attentions along the the decoupled space and time dimensions. The proposed video diffusion model also enables text-conditioned video generation. They outperform state-of-the-art methods in different benchmarks for video prediction and unconditional video generation. They also show the joint image-video training and classifier-free guidance for video diffusion models. The proposed diffusion model can also perform on the long sequence via auto-regressive inference.

strengths_and_weaknesses: Strengths
1. The proposed 3D U-Net diffusion model achieves state-of-the-art performances on unconditional video modeling on UCF101, video prediction on BAIR Robot Pushing and Kinetics-600, and text-based video generation tasks.
2. The proposed joint image and video training is effective.
3. The paper starts with the background section which gives a good guidance to readers.
----

Weaknesses
1. How does the proposed method work on 128x128 resolution results, compared to those of 64x64?
2. In figure 1, where is the num-frames dimension denoted?
3. Is the space-time factorization necessary? What would be the performance vs efficiency of using 3D convolution based U-Net compared to the proposed factorized one?
4. How much performance drop will happen without the temporal attention, ie, per-frame method?

questions: * Is the space-time factorization necessary? What would be the performance vs efficiency of using 3D convolution based U-Net compared to the proposed factorized one?
* How much performance drop will happen without the temporal attention, ie, per-frame method?
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that diffusion models also perform well for video modeling. By choosing a factorized attention module, it can directly extend a 2D U-Net to 3D space-time, allowing joint training with images possible. Results on unconditional text-to-video generation and video prediction show the strong performance of the proposed model. 

 
strengths_and_weaknesses: 1. A straightforward extension of [39] to videos with a factorized encoder, thus allowing joint training with images possible. It shows tricks like classifier-free guidance is also effective in video generation. 

2. Ablate different sampling methods in Table 6 shows that the proposed reconstruction-guided conditional sampling method performs the best. 

3. Results on UCF101, K600, and BAIR-RP in Table 1, 2, 3 are outstanding. 

questions: 1. The major technical contribution seems to come from [39] and the novelty on videos such as factorized attention and co-train are extensively studied by previous works. Nevertheless, combining these components together are worth noting. 

2. The writing is more or less like a draft version, it makes people with less prior knowledge very hard to follow. 

3. In formula (8), How much extra training time does the reconstruction cost?
limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper adapts diffusion models for image generation to generate temporally coherent videos. The adaptation were done by (1) using a 3D U-Net diffusion model architecture to generate a fixed number of video frames; (2) using reconstruction guidance to generate temporally coherent longer videos. This paper claims that the proposed method generates SOTA results on benchmarks for unconditioned video generation and video prediction. The paper also claims that joint training using both images and videos improves the generated video quality. 
strengths_and_weaknesses: Strength: 
- The approach is a natural retention from diffusion models for image generation.
- Results are great both in terms of visual quality and numerical results.

Weaknesses:
- The claim of joint training using images and videos is better supported with more experiments:
1. As mentioned in the paper, images are better sampled outside of the video dataset.
2. In the unconditional video generation task, do joint training help? 
- Video frame interpolation related literatures are better discussed in the sections of the video extension. 
questions: Given the training hardware used, how long does training take on these tasks? 
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed new diffusion model for video generation. The proposed model use 3D-Unet as a backbone for the diffusion model. Furthermore, the paper introduce novel conditional sampling technique for spatial and temporal video generation. The proposed methods present on unconditional video modeling, video prediction, and text conditioned video generation. Moreover, the proposed method show large improvement comparing to the baselines. 
strengths_and_weaknesses: Strengths:
- The paper introduce a novel diffusion model for video generation 
- The paper proposed two novel ideas to drive video X^b from sampled video X^a
- The improvement in the results is relatively high.
- The paper is well written and easy to follow.



Weaknesses:
- Results - looking on the generated video, the resolution is low. Can you trained the model for higher resolution? We all know that diffusion model is capable to generate high fidelity images, so it seems straight forward to use the super resolution approach like in CDM [cascade diffusion model] and get better video.
- The paper should compare to other previous results. for example, StyleGAN-V [CVPR22], DIGAN [ICLR22].
questions: It will be helpful to provide the reason why you choose to use the 3D-Unet. Are there any other architecture that can be also appropriate for this task?

Regarding the dataset that chosen - Can you think about dataset that has more diverse content and interactions? for example, not only video that has repetitive movements. 
limitations: The author provide an section regarding potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper proposes a diffusion model for video capable of generating long and high-resolution videos. Diffusion models have generated some more excitement around generative models as well, so the paper is well-timed. The reviewers had a few concerns regarding additional experiments and clarifications, and it appears that the authors have satisfied those concerns. Overall, the reviews are positive, and there was a decent amount of interaction between the reviewers and authors, though much of the discussion was straightforward and didn't seem to require a good deal of discussion.

I therefore recommend accept for the paper based on there being a clear consensus and the discussions and revision satisfying most outstanding concerns.

Overall I have no recommendations of the reviewers based on this paper. The paper might have been very straightforward to read, and given the consensus I think that it is the case that this was a relatively easy review for everyone (neural reviewer score for everyone, though maybe due to paper being easy).
confidence: Certain
Decision: Accept