Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: #rebuttal responses
 
I change the score to be weak accept as the authors do not provide any comparison result on Rainbow without the prioritized replay buffer during the rebuttal phase. I also agree with Reviewer 1's opinion that the authors do not provide some fixing method, such as combining the noisy networks and bonus methods.


#review
This paper evaluates the recently proposed exploration methods that achieve ground-breaking performance in the difficult exploration problem, Montezuma's Revenge.  The authors combine Rainbow with different exploration methods, such as count-based bonus methods, curiosity-driven methods, and noisy networks. Results show that these methods fail to beat epsilon-greedy on other Atari games, even if the parameters of these methods are tuned. 

The paper is very well written, and they claim that evaluating the exploration methods on the Montezuma's Revenge and tuning parameters on this environment are not suitable for the total ALE environments. The claim is very interesting and important for the exploration community. 

To support their claim, the authors firstly compare bonus exploration methods, noisy networks, and epsilon-greedy on hard exploration games. Then results in easy games and other games are presented. The results are very impressive.

Question:
(1) The authors compared these methods based on Rainbow, which employs many techniques, such as the prioritized replay buffer. Can you show the comparison results on Rainbow without the prioritized replay buffer? It will strengthen the understanding of these exploration methods.
(2) The noisy networks perform well on most games, while bonus methods perform well on hard games. Is there any combination method to achieve better performance?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Updated review: I am overall happy with the response of the authors. I can appreciate the contributions of the paper and I am happy to recommend accept. The empirical study offers some insights into deep RL methods for ATARI games and raises some key questions. I feel the current version of the paper does not build upon these insights to propose a new method. 

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Summary: This paper presents a detailed empirical study of the recent bonus based exploration method on the Atari game suite. The paper concludes that methods that perform well on Montezuma’s revenge do not necessarily perform well on the other games, sometimes, even worse than the eps-greedy approach. This also leads to the conclusion that recent results on the game Montezuma’s revenge can be attributed to architectural changes instead of the exploration method. 

I think this is a-ok paper in that it does what it says it does. The paper is clear and well-written. 

I think the main contribution of the paper is that it raises some questions over existing methods/trends in solving exploration problems in reinforcement learning by comparing the performance of multiple methods across various games in ATARI suite. 
I think this is relevant to the ICLR community and will be appreciated by it. 

However, I also feel that while the paper runs a satisfactory empirical analysis, it was all too much focussed on the existing methods. Throughout the paper, the experiments and results raise questions on the robustness and generalization of existing exploration methods across various ATARI games, but the paper puts absolutely zero effort into investigating if there is a quick fix to the questions it poses. For example, one could easily investigate in the CTS method if the factor by which exploration bonus dies N^{alpha} (alpha=-1/2 by default) changes, then does it do better or worse (more below on this).
I can understand that might not be the aim of the paper but still. 

Here are a couple of points that I felt conflicted/confused about the paper: 
- The conclusion of the paper is that ‘progress of exploration in ATARI suite is obfuscated by good results in single domain’. I am confused if the paper is making a narrow point that (1) dont focus on Montezuma’s revenge OR (2) is it admitting a broader point that focussing on even ATARI is probably not a good choice. I am not saying that I know the answer to this question, but I am unclear as to what is the question the paper is trying to raise. If it is saying (1st) then I find it contradictory that it is not ok to focus on MR but it is ok to focus on ATARI as a single domain; if it is saying the second then also it is contradictory because the paper only experiments with the ATARI suite.
 
- It is interesting to note that noisy networks are most robust to hyperparameter optimization on a separate set of games when tested on a different set of games. It is also interesting to note that noisy networks are the only exploration bonus method that does not decrease/reduce exploration as the experience of the agent increases. I would have liked to see if the paper had made an attempt to investigate this. I feel such a hypothesis would have been easy to investigate with simple modifications to the CTS methods. Currently, the exploration bonus goes down by the factor of 1/sqrt(N)  in the CTS method. A comparison that showed the performance of CTS for a couple more values of factors such as (1/N) or (1/N)^{1/4} would have been nice to see if that mattered.

- One of the comparisons I did not particularly find fair was when the hyperparameters of various methods were tuned to play MR and then the hyperparameters were fixed and the method were tested on other ATARI games. 

- Another point I felt was missing was checking if rainbow DQN is really the reason behind the observed performance of the methods. It would have been interesting to know how the methods performed when combined with the original DQN algorithm. 

review_title: Official Blind Review #1

Meta-review: This paper presents a detailed comparison of different bonus-based exploration methods on a common evaluation framework (Rainbow) when used with the ATARI game suite. They find that while these bonuses help on Montezuma's Revenge (MR), they underperform relative to epsilon-greedy on other games. This suggests that architectural changes may be a more important factor than bonus-based exploration in recent advances on MR.

The reviewers commented that this paper makes no effort to present new techniques, and the insights discovered could be expanded on. Despite this, it is an interesting paper that is generally well argued and would be a useful contribution to the field. I recommend acceptance.
Decision: Accept (Poster)