Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
REFERENCES ARE LISTED AT THE END OF THE REVIEW


Summary:
This paper proposes a method for video prediction that, given a starting and ending image, is able to generate the frame trajectory in between. They propose two variations of their method: A sequential and a tree based methods. The tree-based method enables efficient frame sampling in a hierarchical way. In experiments, they outperform the used baselines in the task of video prediction. Additionally, they used the learned pixel dynamics model and an inverse dynamics model to plan actions for an agent to navigate from a starting frame to an ending frame.


Pros:
+ Novel latent method for goal conditioned prediction (sequential and hierarchical)
+ Really cool experiments on navigation using the predicted frames
+ Outperforms used baselines

Weaknesses / comments:
- Missing baseline:
The Human 3.6M experiments are missing the baseline from Wichers et al., 2018. I would be good to compare against them for better assessment of the predicted videos.

- Bottleneck discovery experiments (Figure 8):
The visualizations shown in Figure 8 are very interesting, however, I would like to see if the model is able to generate multiple trajectories from the same frame. It looks like the starting frames (left) are not the same.


Conclusion:
This paper proposes a novel latent variable method for goal oriented video prediction which is then used to enable an agent to go from point A to point B. I feel this paper brings nice insights useful for the model based reinforcement learning literature where the end goal can be guided by an image rather than predefined rewards. It would be good if the authors can include the suggested video prediction baseline from Wichers et al., 2018 in their quantitative comparisons.


References:
Nevan Wichers, Ruben Villegas, Dumitru Erhan, Honglak Lee. Hierarchical Long-term Video Prediction without Supervision. In ICML, 2018

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary: The following work proposes a model for long-range video interpolation -- specifically targetting cases where the intermediate content trajectories may be highly non-linear. This is referred to as goal-conditioned in the paper. They present an autoregressive sequential model, as well as a hierarchical model -- each based on a probabilistic framework. Finally, they demonstrate an application in imitation learning by introducing an additional model that maps pairs of observations (frames) to a distribution over actions that predicts how likely each action will map the first observation to the second. Their imitation learning method is able to successfully solve mazes, given just the start and goal observations.

Strengths:
-The extension to visual planning/imitation learning was very interesting
-Explores differences between sequential and hierarchical prediction models

Weaknesses/questions/suggestions:
-In addition to SSIM and PSNR, one might also want to consider the FVD and LPIPS, both which should correlate better with human perception.
-How does the inverse model in section $ p(a | o,o')$ account for the case in which multiple actions may eventually result in o -> o', given than o' is sufficiently far from o? Does the random controller need to be implemented in a specific way to handle this?
-I think a fairly important unstated limitation is that latent-variable based methods tend not to generalize well outside of their trained domain. In table 1, I assume DVF was taken off-the-shelf, but all other methods were trained specifically on H3.6M?


LPIPS: https://github.com/richzhang/PerceptualSimilarity
FVD: https://github.com/google-research/google-research/tree/master/frechet_video_distance


Overall, I think the results seem pretty promising -- most notably the imitation learning results. I hope that the authors can address some of my concerns stated above.


** Post Rebuttal:
The authors have adequately addressed my concerns regarding clarity and metrics. The current draft also better motivates the task of long-range interpolation vs short range interpolation. I maintain my original rating.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper reformulates video prediction problem by conditioning the prediction on the start and end (goal) frame. This essentially changes the problem from extrapolation to interpolation which results in higher quality predictions. 

The motivation behind the paper is not clear. First of all, the previous work in video predicted is typically formulated as "conditioned frame prediction" where the prediction of the next frame is conditioned on "a set of context frames" and there is no reason why this set cannot contain the goal frame. Their implementation, however, is motivated by their application and therefore these models are usually only conditioned on the start frames. Unfortunately, besides the reverse planning in imitation learning, the authors did not provide a suite of applications where such a model can be useful. Hence, I think the authors should answer these two questions to clear up the motivation:
1. Why conditioning on the goal frame is interesting? It specifically helps to provide more concrete details than getting from Oakland to San Fransico.
2. Where the current conditional models suffer by conditioning on the goal image?

More experiments are required to support the claims of the paper as well. 
Given my point regarding context frames, a more fair experiment would be to compare the proposed method with them when they are conditioned on the goal frame as well. This explicitly has been avoided in 5.1.
 The used metrics are not a good evaluation metric for frame prediction as they both do not give us an objective evaluation in the sense of the semantic quality of predicted frames. The authors should present additional quantitative evaluation to show that the predicted frames contain useful semantic information. FVD and Inception score come to my mind as good candidates. 

On quality of writing, the paper is well written but it can use a figure that demonstrates proposed architecture. The authors provided the code which is always a plus. 

In conclusion, I believe the impact of the paper, in the current form, is marginal at best and for sure does not meet the requirements for a prestigious conference such as ICLR. However, a more clear motivation, a  concrete set of goals and claims, as well as more comprehensive experiments,  can push the quality above the bar. 
review_title: Official Blind Review #3

Meta-review: The paper addresses a video generation setting where both initial and goal state are provided as a basis for long-term prediction. The authors propose two types of models, sequential and hierarchical, and obtain interesting insights into the performance of these two models. Reviewers raised concerns about evaluation metrics, empirical comparisons, and the relationship of the proposed model to prior work.

While many of the initial concerns have been addressed by the authors, reviewers remain concerned about two issues in particular. First, the proposed model is similar to previous approaches with sequential latent variable models, and it is unclear how such existing models would compare if applied in this setting. Second, there are remaining concerns on whether the model may learn degenerate solutions. I quote from the discussion here, as I am not sure this will be visible to authors [about Figure 12]: "now the two examples with two samples they show have the same door in the middle frame which makes me doubt the method learn[s] anything meaningful in terms of the agent walking through the door but just go to the middle of the screen every time."
Decision: Reject