Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents an action-conditional recurrent network that can predict frames in video games hundreds of steps in the future. The paper claims three main contributions: 
1. modification to model architecture (used in Oh et al.) by using action at time t-1 to directly predict hidden state at t
2. exploring the idea of jumpy predictions (predictions multiple frames in future without using intermediate frames)
3. exploring different training schemes (trade-off between observation and prediction frames for training LSTM)

1. modification to model architecture
+ The motivation seems good that in past work (Oh et al.) the action at t-1 influences x_t, but not the state h_t of the LSTM. This could be fixed by making the LSTM state h_t dependent on a_{t-1}
- However, this is of minor technical novelty. Also, as pointed in reviewer questions, a similar effect could be achieved by adding a_t-1 as an input to the LSTM at time t. This could be done without modifying the LSTM architecture as stated in the paper. While the authors claim that combining a_t-1 with h_t-1 and s_t-1 performs worse than the current method which combines a_t-1 only with h_t-1, I would have liked to see the empirical difference in combining a_t-1 only with s_t-1 or only with h_t-1. Also, a stronger motivation is required to support the current formulation.
- Further, the benefits of this change in architecture is not well analyzed in experiments. Fig. 5(a) provides the difference between Oh et al. (with traditional LSTM) and current method. However, the performance difference is composed of 2 components (difference in training scheme and architecture). This contribution of the architecture to the performance is not clear from this experiment. The authors did claim in the pre-review phase that Fig. 12 (a) shows the difference in performance only due to architecture for "Seaquest". However, from this plot it appears that the gain at 100-steps (~15)  is only a small fraction of the overall gain in Fig. 5 (a) (~90). It is difficult to judge the significance of the architecture modification from this result for one game.

2. Exploring the idea of jumpy predictions:
+ As stated by the authors, omitting the intermediate frames while predicting future frames could significantly sppedup simulations.
+ The results in Fig. 5(b) present some interesting observations that omitting intermediate frames does not lead to significant error-increase for at least a few games.
- However, it is again not clear whether the modification in the current model leads to this effect or it could be achieved by previous models like Oh et al.
- While, the observations themselves are interesting, it would have been better to provide a more detailed analysis for more games. Also, the novelty in dropping intermediate frames for speedup is marginal.

3. Exploring different training schemes
+ This is perhaps the most interesting observation presented in the paper. The authors present the difference in performance for different training schemes in Fig. 2(a). The training schemes are varied based on the fraction of training phase which only uses observation frames and the fraction that uses only prediction frames.
+ The results show that this change in training can significantly affect prediction results and is the biggest contributor to performance improvement compared to Oh et al.
- While this observation is interesting, this effect has been previously explored in detail in other works like schedule sampling (Bengio et al.) and to some extent in Oh et al.

Clarity of presentation:
- The exact experimental setup is not clearly stated for some of the results. For instance, the paper does not say that Fig. 2(a) uses the same architecture as Oh et al. However, this is stated in the response to reviewer questions.
- Fig. 4 is difficult to interpret. The qualitative difference between Oh et al. and current method could be highlighted explicitly. 
- Minor: The qualitative analysis section requires the reader to navigate to various video-links in order to understand the section. This leads to a discontinuity in reading and is particularly difficult while reading a printed-copy.

Overall, the paper presents some interesting experimental observations. However, the technical novelty and contribution of the proposed architecture and training scheme is not clear.
review_title: Some interesting experimental observations, but significance and novelty of proposed architecture needs better justification

final_score: rating: 7: Good paper, accept
review_content: [UPDATE]
After going through the response from the author and the revision, I increased my review score for two reasons.
1. I thank the reviewers for further investigating the difference between yours and the other work (Scheduled sampling, Unsupervised learning using LSTM) and providing some insights about it.
This paper at least shows empirically that 100%-Pred scheme is better for high-dimensional video and for long-term predictions.
It would be good if the authors briefly discuss this in the final revision (either in the appendix or in the main text).

2. The revised paper contains more comprehensive results than before.
The presented result and discussion in this paper will be quite useful to the research community as high-dimensional video prediction involves large-scale experiments that are computationally expensive.

- Summary
This paper presents a new RNN architecture for action-conditional future prediction. The proposed architecture combines actions into the recurrent connection of the LSTM core, which performs better than the previous state-of-the-art architecture [Oh et al.]. The paper also explores and compares different architectures such as frame-dependent/independent mode and observation/prediction-dependent architectures. The experimental result shows that the proposed architecture with fully prediction-dependent training scheme achieves the state-of-the-art performance on several complex visual domains. It is also shown that the proposed prediction architecture can be used to improve exploration in a 3D environment.

- Novelty
The novelty of the proposed architecture is not strong. The difference between [Oh et al.] and this work is that actions are combined into the LSTM in this paper, while actions are combined after LSTM in [Oh et al.]. The jumpy prediction was already introduced by [Srivastava et al.] in the deep learning area. 

- Experiment
The experiments are well-designed and thorough. Specifically, the paper evaluates different training schemes and compares different architectures using several rich domains (Atari, 3D worlds). Besides, the proposed method achieves the state-of-the-art results on many domains and presents an application for model-based exploration.

- Clarity
The paper is well-written and easy to follow.

- Overall 
Although the proposed architecture is not much novel, it achieves promising results on Atari games and 3D environments. In addition, the systematic evaluation of different architectures presented in the paper would be useful to the community.

[Reference]
Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov. Unsupervised Learning with LSTMs. ICML 2016.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose a recurrent neural network architecture that is able to output more accurate long-term predictions of several game environments than the current state-of-the-art.
The original network architecture was inspired by inability of previous methods to accurately predict many time-steps into the future,
and their inability to jump directly to a future prediction without iterating through all intermediate states.
The authors have provided an extensive experimental evaluation on several benchmarks with promising results.
In general the paper is well written and quite clear in its explanations.
Demonstrating that this kind of future state prediction is useful for 3D maze exploration is a plus.

# Minor comments:
`jumpy predictions have been developed in low-dimensional observation spaces' - cite relevant work in the paper.

# Typos
Section 3.1 - `this configuration is all experiments'
review_title: Review

final_score: rating: null
review_content: Thank you for this interesting work.

1. One interesting result is that 100%-Prediction (fully prediction-dependent training) method performs best.
This seems a bit contradictory to the result of Scheduled Sampling [Bengio et al.] paper.
Have you compared your 100%-prediction method to scheduled sampling method (with epsilon annealing)?
Showing this would be interesting.

2. Figure 9 shows that the wall configurations are same as previously generated walls after 360-degree spin.
Does this mean that generated scenes do not necessarily match with the ground-truth, but your model can remember and preserve generated topologies? 

3. What's the difference between your model-based exploration and informed exploration in [Oh et al.]?
Can you give an intuition why your exploration method can be potentially better than [Oh et al.]? 
review_title: Some questions

final_score: rating: null
review_content: `jumpy predictions have been developed in low-dimensional observation spaces' - could you elaborate on the work that inspired you to develop jumpy predictions for higher-dimensional spaces? What is the difference in dimensionality between theirs and yours?
review_title: Jumpy predictions

final_score: rating: null
review_content: 1. The current formulation is claimed to overcome the limitation of traditional LSTM which makes accurate predictions for only a few time-steps. Why is the current formulation which is a variant of LSTM that fuses action (a_t-1) with hidden state (h_t-1) instead of (h_t) expected to overcome this limitation?

2. Also, have you considered fusing (a_t-1) with (s_t-1) instead of h_t-1? How would the performance be affected if a_t-1 is treated as an additional input along with s_t-1 for the traditional LSTM? Is there a strong reason for fusing a_t-1 with h_t-1 to change the LSTM formulation?

3. At many places, it is claimed that jumpy simulators can make make predictions at any point in the future. However, can't the prediction-dependent "single step" simulator also achieve this, since it does not require observed frames?

4. Is the color coding same for Fig. 2a and Fig. 2b?

5. The comparison of the current approach with Oh et al. in Fig. 5a is useful. However, I would have liked to see the breakdown in performance improvement due to the (a) difference in training scheme and (b) difference in architecture. What happens if the same training scheme is used for Oh et al?
review_title: Few questions about motivation of formulation

Meta-review: Quality, Clarity: 
  The paper is well written. Further revisions have been made upon the original.
 
 Originality, Significance:
  The paper presents an action-conditional recurrent network that can predict frames in video games hundreds of steps in the future. This is done using a mix of (a) architectural modifications; (b) jumpy predictions; and (c) particular training schemes. The experimental validation is extensive, now including additional comparisons suggested by reviewers.
 There is not complete consensus about the significance of the contributions, with one reviewer seeking additional technical novelty. Overall, the paper appears to provide interesting and very soundly-evaluated results, which likely promises to be the new standard for this type of prediction problem.
Decision: Accept (Poster)