Title: Stochastic Variational Video Prediction
Authors: Mohammad Babaeizadeh
Chelsea Finn
Dumitru Erhan
Roy H. Campbell
Sergey Levine
Venue: ICLR 2018
Keywords: video prediction
stochastic prediction
variational inference
unsupervised learning

Abstract: Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images requires the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world video. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication.

TL;DR: Stochastic variational video prediction in real-world settings.

Review discussion:
Quality: above threshold
Clarity: above threshold, but experiment details are missing.
Originality: slightly above threshold.
Significance: above threshold
Pros:
This paper proposes a stochastic variational video prediction model. It can be used for prediction in optionally available external action cases. The inference network is a convolution net and the generative network is using a previously structure with minor modification. The result shows its ability to sample future frames and outperforms with methods in qualitative and quantitive metrics.
Cons:
1. It is a nice idea and it seems to perform well in practice, but are there careful experiments justifying the 3-stage training scheme?
For example, compared with other schemes like alternating between 3 stages, dynamically soft weighting terms.
2. It is briefly mentioned in the context, but has there any attempt towards incorporating previous frames context for z, instead of sampling from prior? This piece seems much important in the scenarios which this paper covers.
3. No details about training (training data size, batches, optimization) are provided in the relevant section, which greatly reduces the reproducibility and understanding of the proposed method.
For example, it is not clear whether the model can generative samples that are not previously seen in the training set. It is strongly suggested training details be provided.
4. Minor, If I understand correctly, in equation in the last paragraph above 3.1,  z instead of z_t
Thank you for your great comments. comments and constructive criticism. We updated the paper to address all of your comments. Please let us know if you have any more suggestions or comments. Thanks!
- “No details about training (training data size, batches, optimization) are provided in the relevant section”
Thank you for your great comment. To further investigate the effect of our proposed training method, we conducted more experiments by alternating between different steps of suggested method. The updated Figure 4c reflects the results of this experiments. As it can be seen in this graph, the suggested steps help with both stability and convergence of the model.
We also provided details of the training method in Appendix A to address your comment regarding using soft-terms as well as reproducibility. We will also release the code after acceptance.
- “It is briefly mentioned in the context, but has there any attempt towards incorporating previous frames context for z, instead of sampling from prior? This piece seems much important in the scenarios which this paper covers.“
This is one of the future work directions mentioned in the conclusion section. We’ve expanded this discussion in the conclusion a bit to address this better.
- “Minor, If I understand correctly, in equation in the last paragraph above 3.1, z instead of z_t”
Thank you for the detailed comment. We fixed the typo.
1) Summary
This paper proposed a new method for predicting multiple future frames in videos. A new formulation is proposed where the frames’ inherent noise is modeled separate from the uncertainty of the future. This separation allows for directly modeling the stochasticity in the sequence through a random variable z ~ p(z)  where the posterior  q(z | past and future frames) is approximated by a neural network, and as a result, sampling of a random future is possible through sampling from the prior p(z) during testing. The random variable z can be modeled in a time-variant and time-invariant way.
Additionally, this paper proposes a training procedure to prevent their method from ignoring the stochastic phenomena modeled by z. In the experimental section, the authors highlight the advantages of their method in 1) a synthetic dataset of shapes meant to clearly show the stochasticity in the prediction, 2) two robotic arm datasets for video prediction given and not given actions, and 3) A challenging human action dataset in which they perform future prediction only given previous frames.
2) Pros:
+ Novel/Sound future frame prediction formulation and training for modeling the stochasticity of future prediction.
+ Experiments on the synthetic shapes and robotic arm datasets highlight the proposed method’s power of multiple future frame prediction possible.
+ Good analysis on the number of samples improving the chance of outputting the correct future, the modeling power of the posterior for reconstructing the future, and a wide variety of qualitative examples.
+ Work is significant for the problem of modeling the stochastic nature of future frame prediction in videos.
3) Cons:
Approximate posterior in non-synthetic datasets:
The variable z seems to not be modeling the future very well. In the robot arm qualitative experiments, the robot motion is well modeled, however, the background is not. Given that for the approximate posterior computation the entire sequence is given (e.g. reconstruction is performed), I would expect the background motion to also be modeled well. This issue is more evident in the Human 3.6M experiments, as it seems to output blurriness regardless of the true future being observed. This problem may mean the method is failing to model a large variety of objects and clearly works for the robotic arm because a very similar large shape (e.g. robot arm) is seen in the training data. Do you have any comments on this?
Finn et al 2016 PNSR performance on Human 3.6M:
Is the same exact data, pre-processing, training, and architecture being utilized? In her paper, the PSNR for the first timestep on Human 3.6M is about 41 (maybe 42?)  while in this paper it is 38.
Additional evaluation on Human 3.6M:
PSNR is not a good evaluation metric for frame prediction as it is biased towards blurriness, and also SSIM does not give us an objective evaluation in the sense of semantic quality of predicted frames. It would be good if the authors present additional quantitative evaluation to show that the predicted frames contain useful semantic information [1, 2, 3, 4].
For example, evaluating the predicted frames for the Human 3.6M dataset to see if the human is still detectable in the image or if the expected action is being predicted could be useful to verify that the predicted frames contain the expected meaningful information compared to the baselines.
Additional comments:
Are all 15 actions being used for the Human 3.6M experiments? If so, the fact of the time-invariant model performs better than the time-variant one may not be the consistent action being performed (last sentence of 5.2). The motion performed by the actors in each action highly overlaps (talking on the phone action may go from sitting to walking a little to sitting again, and so on).
Unless actions such as walking and discussion were only used, it is unlikely the time-invariant z is performing better because of consistent action. Do you have any comments on this?
4) Conclusion
This paper proposes an interesting novel approach for predicting multiple futures in videos, however, the results are not fully convincing in all datasets. If the authors can provide additional quantitative evaluation besides PSNR and SSIM (e.g. evaluation on semantic quality), and also address the comments above, the current score will improve.
References:
[1] Emily Denton and Vighnesh Birodkar. Unsupervised Learning of Disentangled Representations from Video. In NIPS, 2017.
[2] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICML, 2017.
[3] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv preprint arXiv:1710.10196, 2017.
[4] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NIPS, 2017.
Revised review:
Given the authors' thorough answers to my concerns, I have decided to change my score. I would like to thank the authors for a very nice paper that will definitely help the community towards developing better video prediction algorithms that can now predict multiple futures.
Thank you for your insightful comments and suggestions. We have addressed most of your concerns. Please see our responses below and let us know if you have any further comments on the paper. Thanks!
- "Additional evaluation on Human 3.6M: PSNR is not a good evaluation metric for frame prediction"
Thank you for this suggestion. We’ve updated the paper (please look at Figure 7 and 6th paragraph of 5.3) to address your comment. In order to investigate the quality difference between SV2P predicted frames and “Finn et al (2016)”, we performed a new experiment in which we used the open sourced version of the object detector from “Huang et al. (2016)”:
https://github.com/tensorflow/models/blob/master/research/object_detection/models/ssd_mobilenet_v1_feature_extractor.py
to detect the humans inside the predicted frames. We used the confidence of this detection as an additional metric to evaluate the difference between different methods. The results of this comparison which shows higher quality for SV2P can be found in (newly added) Figure 7.
- "Are all 15 actions being used for the Human 3.6M experiments?"
We’ve updated the 2nd bullet point in 5.1 to clear this up in the paper. Yes, we are using all the actions. In regard to changing actions: since the videos are relatively short (about 20 frames), there aren't any videos where the actor changes the behavior in the middle.
That said, the identity of the behavior is not the only source of stochasticity, since even within a single action (e.g., walking), the actor might choose to walk at different speeds and in different directions.
- "I would expect the background motion to also be modeled well.”
We've added a discussion of this in Section 5.3 (paragraph 4). Note that the approximate posterior over z is still trained with the ELBO, which means that it must compress the information in future events. Perfect reconstruction of high-quality images from posterior distributions over latent states is an open problem, and the results in our experiments compare favorably to those typically observed even in single-image VAEs (e.g. see Xue et al. (2016))
- "Finn et al 2016 PNSR performance on Human 3.6M: In her paper, the PSNR for the first timestep on Human 3.6M is about 41 (maybe 42?) while in this paper it is 38"
For “Finn et al. (2016)”, we used the open-source version of the code here:
https://github.com/tensorflow/models/tree/master/research/video_prediction
which is a reimplementation of the models used in the Finn et al. ‘16 paper. We are not exactly sure where the discrepancy is coming from.
However, we would like to point out that whatever issue resulted in slightly slower PSNR for the deterministic model would have affected our model as well, since we used the same code for the base model.
Hence, the comparison is still valid.
The submission presents a method or video prediction from single (or multiple) frames, which is capable of producing stochastic predictions by means of training a variational encoder-decoder model. Stochastic video prediction is a (still) somewhat under-researched direction, due to its inherent difficulty.
The method can take on several variants: time-invariant [latent variable] vs. time-variant, or action-conditioned vs unconditioned. The generative part of the method is mostly borrowed from Finn et al. (2016). Figure 1 clearly motivates the problem. The method itself is fairly clearly described in Section 3;
in particular, it is clear why conditioning on all frames during training is helpful. As a small remark, however, it remains unclear what the action vector a_t is comprised of, also in the experiments.
The experimental results are good-looking, especially when looking at the provided web site images.
The main goal of the quantitative comparison results (Section 5.2) is to determine whether the true future is among the generated futures.
While this is important, a question that remains un-discussed is whether all generated stochastic samples are from realistic futures. The employed metrics (best PSNR/SSIM among multiple samples) can only capture the former, and are also pixel-based, not perceptual.
The quantitative comparisons are mostly convincing, but Figure 6 needs some further clarification. It is mentioned in the text that "time-varying latent sampling is more stable beyond the time horizon used during training".
While true for Figure 6b), this statement is contradicted by both Figure 6a) and 6c), and Figure 6d) seems to be missing the time-invariant version completely (or it overlaps exactly, which would also need explanation).
As such, I'm not completely clear on whether the time variant or invariant version is the stronger performer.
The qualitative comparisons (Section 5.3) are difficult to assess in the printed material, or even on-screen. The animated images on the web site provide a much better impression of the true capabilities, and I find them convincing.
The experiments only compare to Reed et al. (2017)/Kalchbrenner et al. (2017), with Finn at al. (2016) as a non-stochastic baseline, but no comparisons to, e.g., Vondrick et al. (2016) are given. Stochastic prediction with generative adversarial networks is a bit dismissed in Section 2 with a mention of the mode-collapse problem.
Overall the submission makes a significant enough contribution by demonstrating a (mostly) working stochastic prediction method on real data.
Thank you for insightful comments and constructive criticism. We updated the paper to address all of your comments. Please let us know if you have any more suggestions or comments. Thanks!
- "As a small remark, however, it remains unclear what the action vector a_t is comprised of, also in the experiments."
Thank you for the good point. We’ve updated the paper to clarify what the actions are for each dataset. Please check 5.1 for more clarification.
- "a question that remains un-discussed is whether all generated stochastic samples are from realistic futures"
We’ve updated the paper to clarify this issue. Please look at the 2nd paragraph of Section 4 for updates.
In short, as we observed empirically from the predicted videos, the output videos are within the realistic possibilities.
However, in some cases, the predicted frames are not realistic and are averaging more than one future (e.g.
first random sample in Figure 1-C).
- "The employed metrics (best PSNR/SSIM among multiple samples) can only capture the former, and are also pixel-based, not perceptual."
Thank you for the great comment. We’ve updated the paper (please look at Figure 7 and 6th paragraph of 5.3) to address your comment. In order to investigate the quality difference between SV2P predicted frames and “Finn et al (2016)”, we performed a new experiment in which we used the open sourced version of the object detector from “Huang et al. (2016)”:
https://github.com/tensorflow/models/blob/master/research/object_detection/models/ssd_mobilenet_v1_feature_extractor.py
to detect the humans inside the predicted frames. We used the confidence of this detection as an additional metric to evaluate the difference between different methods. The results of this comparison which shows higher quality for SV2P can be found in (newly added) Figure 7.
- "time-varying latent sampling is more stable beyond the time horizon used during training".
While true for Figure 6b), this statement is contradicted by both Figure 6a) and 6c)."
Thank you for the great question. We’ve updated the paper (last two paragraphs of 5.2) to include your observation. Please note that our original claim was that the time-variant latent seems to be more “stable” beyond the time horizon used during training (which is highly evident in Figure 6b). And we are NOT claiming that time-variant latent generates “higher quality” results.
However,  we agree that this stability is not always the case as it is more evident in late frames of Figure 6a.

Meta-review:
role: PC
data: comment: Not quite enough for an oral but a very solid poster.

Decision: Accept (Poster)