Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents an emprical study of how a properly tuned implementation of a model-free RL method can achieve data-efficiency similar to a state-of-the-art model-based method for the Atari domain. 

The paper defines r as ratio of network updates to environment interactions to describe model-free and model-based methods, and hypothesizes that model-based methods are more data-efficient because of a higher ratio r. To test this hypothesis, the authors take Rainbow DQN (model-free) and modify it to increase its ratio r to be closer to that SiMPLe (model-based). Using the modified verison of Rainbow (OTRainbow), the authors replicate an experimental comparison with SiMPLe (Kaiser et al, 2019), showing that Rainbow DQN can be a harder baseline to beat than previously reported (Figure 1). This paper raises an important point about empirical claims without properly tuned baselines, when comparing model-based to model-free methods, identifying the amount of computation as a hyperparameter to tune for fairer comparisons.

I recommend this paper to be accepted only if the following issues are addressed. The first is the presentation of the empirical results. In Figure 1, OTRainbow is compared against the reported results in (Kaiser et al, 2019), along with other baselines, when limiting the experience to 100k interactions. Then, in Figure 2, human normalized scores are reported for varying amounts of experience for the variants of Rainbow, and compared against SiMPLe with 100k interactions, with the claim that the authors couldn't run the method for longer experiences. Unless a comparison can be made with the same amounts of experience, I don't see how Figure 2 can be interpreted objectively. In any case, the results in Figure 1 and the appendix are useful for showing that the baselines used in prior works were not as strong as they could be.

The second has to do with the interpretation of the results. The paper chooses a single method class of model-based methods to do this comparison, namely dyna-style algorithms that use the model to generate new data. But models can also be used for value function estimation (Model Based Value Expansion) and reducing gradient variance(using pathwise derivatives). The paper is written as if the conclusions could be extended to model-based methods in general. Can we get the same conclusions on a different domain where other model-based methods have been successful; e.g. continuous control tasks? A way to improve the paper would be to make it clear from the beginning that these results are about Dyna-style algorithms in the Atari domain.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper presents a data-efficient version of the Rainbow DQN by Hessel et al. (2018) that matches the performance of a recent state of the art model-based method by Kaiser et al. (2019) on several Atari games. Particularly, the paper empirically shows that a simple hyper-parameter tuning, in this case increasing the ratio of number of training steps to the environment interactions as well as decreasing epsilon-decay period, can result in significant improvements in sample efficiency of the Rainbow DQN agent. They show that their method (which requires significantly less computation) can outperform the model-based variant on half of the games tested, while performing worse on the rest. 

Overall, I believe that this paper is below the acceptance threshold due to lack of 1) novelty, 2) significance and 3) depth of analysis.

The observation that more training updates with the agent’s existing experience results in  sample efficiency in DQN method has already been shown empirically by Holland et al. (2018) which has also been cited by this paper. Additionally, more recent work by Hasselt et al. (2019) which is not currently cited, explicitly addresses the same problem as this work by tuning the hyper-parameters of the Rainbow DQN to achieve significant sample efficiency, outperforming Kaiser et al. (2019) in 17 out of 26 Atari games tested. In addition, their work gave a rather detailed motivation and analysis of their findings, proposing and testing several hypotheses for how and when model-based methods could outperform replay-based model-free variants. 

In comparison to prior work, the current paper has a more limited scope and significance. Hence, I believe more work would be needed to warrant acceptance.

Hessel et al., 2018: Rainbow: Combining Improvements in Deep Reinforcement Learning https://arxiv.org/abs/1710.02298
Kaiser et al., 2019: Model-Based Reinforcement Learning for Atari https://arxiv.org/abs/1903.00374
Holland et al., 2018: The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces https://arxiv.org/abs/1806.01825
Hasselt et al., 2019: When to use parametric models in reinforcement learning? https://arxiv.org/abs/1906.05243


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Do recent advancements in model-based deep reinforcement learning really improve data efficiency?

In this paper, the authors revisit the model-free baselines used in recent model-based reinforcement learning algorithms. And after more careful tuning of the hyperparameters, the model-free baselines can obtain comparable and much better performance using the same number of samples.

The paper is well-written, and the experiments are well-designed to support the claim.
However, the research contribution of the project is limited to image-space discrete RL tasks, and does not cover the wide-range other RL. In terms of the novelty, the proposed algorithm is not fundamentally different from Rainbow. 
Therefore I tend to vote for borderline for this paper and am willing to increase the scores if more improvement is updated.

Besides, I would like to thank Mr. Ankesh Anand for mentioning the Hasselt et al. (2019) [1] paper, which is indeed very similar in terms of the topic discussed and the methods used to evaluate the algorithms. 
I have also read Hasselt et al. (2019) before this submission, but I think it would be fair to say the two papers are relatively concurrent.

Potential improvement:
- It will be great if the authors also extend the discussion to current reinforcement learning algorithms that are applied in continuous tasks from states. In [2], similar conclusion is observed in continuous control tasks, where SAC [3] / TD3 [4] perform substantially better than many of the state-of-the-art model-based baselines.

[1] van Hasselt, Hado, Matteo Hessel, and John Aslanides. "When to use parametric models in reinforcement learning?." arXiv preprint arXiv:1906.05243 (2019).
[2] Wang, Tingwu & Bao, Xuchan & Clavera, Ignasi & Hoang, Jerrick & Wen, Yeming & Langlois, Eric & Zhang, Shunshi & Zhang, Guodong & Abbeel, Pieter & Ba, Jimmy. (2019). Benchmarking Model-Based Reinforcement Learning.
[3] Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, and Sergey Levine. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." arXiv preprint arXiv:1801.01290 (2018).
[4] Fujimoto, Scott, Herke van Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." arXiv preprint arXiv:1802.09477 (2018).

review_title: Official Blind Review #2

Meta-review: The paper makes broad claims, but the depth of the experiments is very limited to a narrow combination of algorithms.
Decision: Reject