Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper proposes a future frame prediction framework where the video generation can transition between different actions using a Gaussian process trigger. The framework consists of three components: an encoder which encodes the frame to a latent code, an LSTM which predicts the next latent code given the current one, and a Gaussian process which samples a new latent code. The framework can decide whether to switch to the next action by adopting the new latent code, depending on the number of frames passed or the variance of Gaussian.

Strengths:
The paper is easy to follow overall. The usage of Gaussian process to trigger the transition to the next action is reasonable and intuitive. Quantitative evaluations show that the method outperforms existing works for both reconstruction and output diversity for various datasets.

Weaknesses and comments:
There are quite a few typos in the writing, especially toward the latter part of the paper. I’d encourage the authors to do a thorough check to ensure the paper is typo-free.
It seems switching actions at some fixed number of frames beats using the Gaussian variance for FVD, which is quite surprising. Can the authors provide some insights? Is it due to some inherent nature of FVD, or there’s still some room for improvement for the choosing criteria?
How important is the heuristic of changing states when using GP? Currently it is triggered when the variance is larger than two standard deviations. How will it affect the performance if a different threshold is used?
There’s a mistake in Table 1. The diversity score for DVG@15,35 is the best for KTH frames [10,25] (48.30), but DVG GP is bolded (47.71). This might also be an interesting point to discuss about why fixed number of frames performs better than GP.

review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### SUMMARY

The authors propose to use a Gaussian Process (GP) to model the uncertainty of future frames in a video prediction setup. In particular, they employ a GP to model the uncertainty of the next step latent in a latent variable model. This allows them to use the GP variance to decide when to change an "action sequence", corresponding to a deterministic dynamics function implemented using an LSTM. 

### STRENGTHS AND WEAKNESSES

[+] Empirical results

[+] Well-motivated model

[+] Clear presentation

[-] Experimental section could be improved (missing baselines in some tables, results seem to differ from those in the literature)

### DETAILED COMMENTS

The paper proposes a novel approach for video prediction. Following the standard latent variable model setup used by many VAE-based video prediction models, the authors propose to use a GP to model the uncertainty in the latent space while also learning a deterministic dynamics model (LSTM) on this latent space. Then the GP is used to decide when a future frame has high uncertainty, and in those cases multiple latents can be sampled from the GP. In general the paper is clear and well-written.

The experimental section could be improved. In particular, more details about how the comparison to some baselines was made would be appreciated. For example,  the results for the VRNN model in Figure 4 and 5 do not follow the results in the literature, where it outperforms SVG and SAVP, and its unclear whether its due to an architectural change, suboptimal hyperparameters, or a different reimplementation. Further this model is missing from some other comparisons such as Table 1. For SAVP the results for Figure 4 seem much worse than those reported in the original paper. On the other hand, the authors did some ablation experiments and included different metrics to analyze the performance of their method.

### SCORE
I vote for accepting the paper. The model formulation is clear, well-motivated and novel. The results are positive and overall it seems like a valid alternative to current approaches that will be of interest to the video prediction community. I would encourage the authors to provide a more thorough experimental section.

### POST-REBUTTAL UPDATE
After reading the other reviews and the authors' rebuttal, I stand by my rating of 6.
review_title: Review for Diverse Video Generation using a Gaussian Process Trigger 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this work, the authors propose to apply Gaussian Processes to generate future video frames with high diversity. Specifically, they use variance of GP prediction as a trigger to control when we should switch to a new action sequence.

Strength 

1 The paper is written well, and the organization is OK

2 The idea of using GP for video generation sounds interesting

Weakness

1 The way of using GP is kind of straightforward and naive. In the GP community, dynamical modeling has been widely investigated, from the start of Gaussian Process Dynamical Model in NIPs 2005. 

2 I do not quite get the modules of LSTM Frame Generation and GP Frame Generation in Eq (4). Where are these modules in Fig.3 ? The D in the Stage 3? Using GP to generate Images? Does it make sense? GP is more suitable to work in the latent space, is it? 

3 The datasets are not quite representative, due to the simple and experimental scenarios. Moreover, the proposed method is like a fundamental work. But is it useful for high-level research topics, e.g.,  large-scale action recognition, video caption, etc?
review_title: GP for Video Generation

Meta-review: All three reviewers agree on accepting the paper and think that the proposed approach will be of interest for those working in vdieo prediction.  The authors are asked to include the extra discussion with R3 as part of the paper and include the proposed changes by R2 to provide more thorough experimentation.  The paper is recommended as a poster presentation.
Decision: Accept (Poster)