Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a DDPM based model for video frame prediction. It uses a randomized masking the randomly condition some frames on others allowing for long term predictions. Authors tested different sampling strategies on a newly introduced dataset to demonstrate its capability in generating long videos. The paper compares its result to CWVAE and Video Diffusion Models. 
strengths_and_weaknesses: ===== Strengths:
- The randomized masking algorithm is simple and effective particularly given the capability of limiting it to a fixed number of frames to put an upper bound on computation.
- The paper is well written, easy to follow and read.
- The new dataset is effective in demonstrating the capability of the proposed method for predicting long videos in a dynamic setting.

==== Weaknesses:
- Analysis. There are multiple points that the paper can be improved by adding more analysis. For example, while the proposed masking strategy makes sense, there is no experiment to show how effective it is vs simpler strategies such as just masking every 10 frames. I understand that the proposed method allows for different sampling strategy but it would be great to show how a simple autoregressive + infilling sampling with a simple masking strategy perform on a similar problem. 

- Comparisons. The paper only compares its results to other models in Table 1 which is not enough for a fair comparison. I suggest authors to include more comparisons. CWVAE has more dataset numbers, metrics and baseline numbers to compare to. FitVid also has a section on long video predictions (with overfitting) on Human3.6M. I also recommend comparing the model with other video prediction models on shorter videos, given the richer number of models in that domain which can help demonstrating the capability of DDPMs for video prediction.

- Overfitting. Another possible improvement is to condition the model on an out of distribution data e.g. an unseen street. Given the size of the dataset, my main concern is that the model memorized everything and is not capable of generalization which is the main goal. FitVid did similar analysis and found overfitting as the main reason for their capability of generating super long videos.

- Qualitative results. Given the nature of videos, I highly encourage authors to include videos on an anonymous website for better demonstration and comparison.  

questions: Please refer to weaknesses for my questions. Here is a short list:
- Is the model overfitting or can it generalize to unseed cases?
- How does the model perform vs a simpler DDPM method. 
- How does the model compare vs other models is other datasets (CWVAE Table 1 from arxiv has a good list)
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a framework for generating long (many minutes, to hours) videos using diffusion models, extending to a series of progressively more complex environments such as Deepmind Lab, Minecraft, and CARLA (a self-driving simulator). As a key contribution, the authors introduce an efficient diffusion model that is trained to generate future frames based on a sparse selection of past frames. The authors investigate a series of selection methods, including various local or strided schedules, as well as a novel learned schedule. 

strengths_and_weaknesses: Strengths
- The paper is well written and clear
- The authors present a novel diffusion model for videos that is simple and easy to scale.
- Experiments show high quality, visually consistent results on domains of varying difficulty, from more synthetic environments like DMLab to more realistic environments like CARLA

Weaknesses
- Novelty is a bit limited on the diffusion side, since it is a fairly direct application of image diffusion models to video

- I’m not too convinced at the ability for the model to accurately learn long-range dependencies, primarily since:

(1) FVD on GQN-Mazes is fairly sensitive to noise and will probably not capture long-range information such as maze layout. 

(2) MineRL doesn’t really require long-range information since, as mentioned in the paper, it mainly consists of agents moving forward, 
so only a few past frames are really needed for accurate video generation 

(3) CARLA is a pretty small dataset, and it seems like a sufficiently large model can just memorize the town layout

questions: (1) Regarding my concern about learning long-range dependencies, it might provide a little more insight into the model if the authors were able to evaluate their GQN-Mazes model on the accuracy metric introduced in CW-VAE (https://proceedings.neurips.cc/paper/2021/file/f490d0af974fedf90cb0f1edce8e3dd5-Paper.pdf ). Note that only the Neurips version of the paper mentions it (arxiv was not updated). The relevant accuracy code can be found here: https://gist.github.com/vaibhavsaxena11/97b2d0a195c08ab2ed75cebb7d763799 .

(2) As mentioned before, it’s a little unclear to me what the purpose of the CARLA dataset is. Since it contains only videos of a small town, it should be fairly easy for a model to memorize the town’s layout. If we're aiming to test a model's ability to memorize a scene, I don't think that's particularly difficult or interesting.

(3) I think it would be useful to include some baselines on autoregressive models, such as TATS [1], or HARP [2], which can generate arbitrarily long videos using a sliding window with limited context
	



[1] Ge, Songwei, et al. "Long video generation with time-agnostic vqgan and time-sensitive transformer." arXiv preprint arXiv:2204.03638 (2022).

[2] Seo, Younggyo, et al. "Autoregressive Latent Video Prediction with High-Fidelity Image Generator." (2021).


limitations: Overall, the authors adequately addressed limitation and potential negative societal impacts of their work.

- As mentioned in the paper, diffusion models are slow to sample from, especially in the case of long-horizon video generation in which you may need to generate thousands or tens of thousands of frames.
- Sampling schemes, even when optimized, are dataset specific and not video specific, so the model is not able to dynamically retrieve context-dependent frames for generation
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a video diffusion model that enables you to sample video by flexible conditioning(depending on your application scenarios). Technical side, the author inject the conditioning information by concatenating it(conditional images) with original images, and feed them into UNet to predict the score/noise of DDPM. The author designs a sampling strategy to enable the pretrained network can be  competent of flexible sampling.  

For experiments, it mainly compares with Video Diffusion Model(VDM), CWVAE, in video completion task of three datasets(one of them is proposed by themselves). They also did several ablation studies to provide more insight about sampling schemes when testing, and sampling schemes when training. 


strengths_and_weaknesses: Pro:

- The paper focuses on the video generation problem by diffusion method, which is a suitable, promising solution  by injecting the inductive-bias of grid-structured data like image, video, audio,etc.
- Clear illustration of the proposed idea about sampling strategy, training network, etc.
- The intuition of the method is technically sound in aiming to solve the complex sampling problem in diffusion models.

Con:

- Too few baselines to compare with, the author only compare with VDM a).in different datasets with them (they use UCF, which is way more complex than your dataset choice), b), they didn't release the code, lot's novelty impl of VDM is not included yet, like gradient-method, replacement method(check details from VDM), and also too few details are revealed to make the comparison with VDM not direct enough.
- Why In table 1, the gap of  loss between VDM and FDM in dataset CARLA Town01 is  so large? Typically, loss is not a good metric here to compare with your method and baselines in Table 1,2. As the model, the hyperparameter, the data preprocessing, and many details can deviate your conclusion from loss comparison. 
- The idea about sampling training task is too simple, there is a lot of human bias  design inside, so I am not convinced by the proposed method especially when the baselines are too few.




questions: - More details about the comparison?
- More explanation about the "Con" parts of above session?
limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes one of the first video diffusion models (concurrently with a couple other papers) and presents an architecture that admits many different approaches to sampling a long video. Video generation as important but unsolved problem, and this paper makes an important step towards long video generation. I do agree with reviewers that more baselines can be beneficial, and the authors have agreed to include at least one more baseline. Regardless, I believe the experiments in the paper add to the growing literature on diffusion models and will be interesting to the NeurIPS audience.
confidence: Less certain
Decision: Accept