Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper addresses sample-efficient learning (~2 hours of gameplay equivalent) for Atari (ALE) games. Building on the idea of training in a learned world model and the use of a u-net next-frame predictor, the approach is claimed to yield almost comparable performance to other models with only a fraction of the true-environment experience.

Sample efficiency is a major concern for DRL, particularly with an eye towards robotics and other physical domains. Although the approach is rather specific to the shapes and qualities of data in the ALE setting, the work is motivated at a high level, and the specific techniques for predicting the next frame explained in the past are explained.

This reviewer moves for a weak accept on account that the paper is well written (with quite thorough experiments explaining improvements in sample efficiency and possible limits in final task performance) but specifically targets ALE where execution is so cheap. The total number of PPO updates made in the new approach is not much reduced from before even if the number of trajectories evaluated in the true environment is very much reduced. On the problem of how much RL itself is sample efficient, not much progress is made.

Question:
- What is the impact on total wall-clock training time when using this approach? Given that the technique is centered on ALE, the characteristics of ALE compared to the learned world model are relevant (ALE executes very quickly and easily parallelizes whereas the learned world model presumably only runs where you have a GPU).
- Can this approach be stacked to benefit from training in a lighter-weight approximate model (env'') of the world model (env')?
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper covers the author’s approach to learning a model of a game, which can then be used to train a reinforcement learning agent. The major benefit of the approach is that instead of requiring millions of training steps in the game, the model can be constructed with only 1-2 hours of footage, and then train in the simulated game for millions of training steps. 

This is a well-written paper, and the results are very impressive. The approach builds upon prior work with the same general thrust, but broadly makes clear that it stands above these existing approaches. However, I would have appreciated some clarity in the comparison made to the work of Ha and Schmidhuber (2018). It is unclear if the difference given is just because of the environments employed by Ha and Schmidhuber or if the authors see the approach presented in this paper as fundamentally different or improved in some way. 

My one major technical concern comes down to how this work is framed and what that implies about appropriate baselines. The authors state clearly that the benefit of this work is that it can learn a sufficient model of a game in only 1-2 hours of gameplay footage. As I said above that is very impressive. However, the agents then requires 15.2 million interactions in this environment to learn to play the game. I would have appreciated some clarity then in the computational resource cost in this approach as opposed to just training say Rainbow in the actual game environments with 15.2 million interactions. It’s also not clear if optimizing Rainbow’s performance on 1M steps is a fair comparison. Ideally I would have liked to have seen some variance in the amount of time Rainbow was trained for compared to the associated computational costs. Clarity on this especially in sections like 6.1 would help readers better grasp the tradeoffs of the approach.

The authors could have also included reference to non-DNN work on learning forward/engine/world models that were then used to play or reason about the game. For example Guzdial and Riedl’s 2017 “Game Engine Learning from Gameplay Video” on Super Mario Bros. or Ersen and Sariel’s 2015 “Learning behaviors of and interactions among objects through spatio–temporal reasoning” on a novel game.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary

This paper proposes a model-based reinforcement learning algorithm suitable for high-dimensional visual environments like Atari. The algorithmic loop is conceptually simple and comprises 1) collecting real data with the current policy 2) updating an environment model with old and newly acquired data and 3) updating the policy "virtually" inside of the environment model using PPO. The approach is evaluated on 26 games from the Atari benchmark and compared against the model-free baselines Rainbow DQN and PPO. The newly proposed model-based method clearly outperforms both model-free baselines in low training regimes (100,000 steps). Further ablation studies are provided, e.g. similar results are obtained in a more stochastic setting of ALE with sticky actions.

Quality

This paper has a strong applied focus and needs to be judged based on its experiments. The quality of those are high. The method is evaluated on a suite of 26 games, compared to strong model-free baselines and results are averaged over 5 seeds. One concern I have is that the method is only evaluated in low training regimes. While I do understand that increasing the training horizon is computationally demanding, results in the appendix (Figure 11a) indicate that the proposed model-based method has worse asymptotic performance compared to the model-free baselines. After 500,000 training steps the effect of sample efficiency vanishes and the final performance results are far away from the final performance results of the model-free baselines after 50,000,000 training steps. Also, a plot similar to Figure 11a) from the appendix for Rainbow DQN would be good (but I do understand this might be difficult to obtain in the course of the review period should this require more experiments).

Clarity

The paper is clearly written and easy to follow. However, the authors could state in the main paper more clearly that their method excels in low training regimes and that the sample efficiency effect seems to vanish when increasing training iterations from 100,000 to 500,000 steps. In fact, Figure 11a) from the appendix should go into the main paper, and it should be also mentioned that there is a huge discrepancy between the maximum performance achieved by the proposed model-based method and the maximum performance achieved by the model-free baselines when training for 50,000,000 steps. Based on the experiments, it is not clear at all if the new method will eventually catch up with best model-free results from the literature when training time is increased, or stall in low-performance regimes indefinitely.

Originality

The originality of this paper is not very high since the proposed algorithm and its components are not novel (there might be some minor novelty in the environment model architecture). However, this paper should not be judged based on its originality but based on its significance. 

Significance

A working model-based RL algorithm for Atari is clearly a huge gap in the current literature and this paper takes an important step towards this direction. Demonstrating improved sample efficiency compared to strong model-free baselines in low training regimes is a significant result. The significance is however decreased by the fact that the paper does not answer the question how to obtain good asymptotic performance that matches (or comes close to) model-free state-of-the-art results. I therefore vote for weak accept at this stage.

Minor details

On a side note, there are two citations missing related to model-based RL in visual domains:
- S. Alaniz. Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft. In the 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2017.
- F. Leibfried and P. Vrancx. Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks. In NIPS Deep Reinforcement Learning Workshop, 2018.

Update

After the author response, my review remains the same. I think this paper is worthwhile publishing at ICLR.
review_title: Official Blind Review #2

Meta-review: This paper presents a model-based RL approach to Atari games based on video prediction. The architecture performs remarkably well with a limited amount of interactions.  This is a very significant result on a question that engages many in the research community.

Reviewers all agree that the paper is good and should be published. There is some disagreement about the novelty of it. However, as one reviewer states, the significance of the results is more important than the novelty. Many conference attendees would like to hear about it.

Based on this, I think the paper can be accepted for oral presentation.
Decision: Accept (Spotlight)