Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper revisits the way RL algorithms are typically evaluated on the ALE benchmark, advocating for several key changes that contribute to more robust and reliable comparisons between algorithms. It also brings the following additional contributions: (1) a new measure of comparison to human performance based on actual human world records (which shows that RL algorithms are not as « super-human » as is generally believed), and (2) an evaluation (based on the proposed guidelines) of Rainbow as well as a Rainbow-IQN variant (replacing the C51 component of Rainbow with Implicit Quantile Networks), showing that the latter brings a significant improvement upon the original Rainbow algorithm.

Overall I am leaning towards acceptance as I believe that such papers encouraging better benchmarking practice on Atari are definitely needed. Even if the technical contribution is limited, this paper could have a positive impact on the field by providing a clearer picture of the current state of deep RL algorithms on Atari (assuming that other researchers start following these recommendations -- and if that is not the case at least it will highlight issues with the way evaluation is currently done).

I do have a few concerns / questions though:

1.	I am not convinced by the recommendation to use performance during training for evaluation purpose. In Machado et al. (2018) it is argued that « this better aligns the performance metric with the goal of continual learning », but most deep RL algorithms trained on Atari games have not been intended to be used in a continual learning setting. It definitely has the advantage of being simple, but it seems to me that it can cause some issues, like making it difficult to compare different exploration techniques for off-policy learning (the exploration may cause poor behavior during training even if it helps the agent learn a better greedy policy), and more generally not being representative of the common practical use case where the goal is to obtain the best agent possible to use in production (with no further learning). Finally, it could make results even harder to reproduce due to the potential high variance of an agent’s performance at a fixed # of timesteps (vs. considering the max performance it can reach over the whole period). As a result, I am currently reluctant to see the proposed performance measure become the standard evaluation metric on ALE, and I would appreciate some additional justification from the authors on this point.

2.	Why not suggest to remove reward clipping in the recommendations? As mentioned in Section 6, reward clipping can prevent RL algorithms from properly playing some games, and thus in my opinion should be removed if the goal is to reach the highest score possible on all games. It seems to me that the choice of clipping the reward should be part of the algorithm (if it is not able to handle the high variety of « raw » rewards) and not of the benchmark environment, thus enabling further advancements towards algorithms that are robust to a wide range of rewards.

3.	Why bother to keep the mean performance when, as mentioned, it is highly sensitive to outliers compared to the median?


Additional remarks:
•	I might have missed it but I do not see the link to the source code. Am I correct to assume it will be released, to help with reproducibility?
•	It is not clear, when reading the paper, that the distributed version of Rainbow is actually constrained to mimic a single agent sequential algorithm in the experiments. I would suggest to remove mentions of the distributed version in the main text to avoid confusion, and mention it only in the Appendix section where it is used.
•	The « infinite reward loop » point at the end of Section 6 does not seem relevant in the list of reasons why Deep RL algorithms are far from the best human performance, since with infinite playtime and an infinite reward loop, the algorithm should be guaranteed to outperform humans.
•	I would have appreciated an evaluation of Rainbow-IQN with the current most commonly used evaluation schemes (e.g. the one used in the original Rainbow paper), for comparison purpose (even if such an evaluation has flaws, it is often the only performance measure available for existing deep RL algorithms)

Review update: thank you for the response, I am currently keeping my "Weak accept" rating because I agree it is important to highlight and (try to) fix the problems with the way algorithms are currently evaluated on ALE, in spite of the limited technical contributions (and the fact that I remained unconvinced regarding #1)
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes an extension to the work of Machado et al. (2018) for standardizing training and evaluation procedures in the Arcade Learning Environment (ALE). It then introduces a collection of human world records for each Atari game to refute previous claims of superhuman performance, as well as recommend comparisons against these records. They proceed to evaluate Rainbow under their proposed evaluation procedures, as well as introduce a new algorithm, Rainbow-IQN, with similar evaluations made based on their proposal.

I'm proposing a weak rejection as I feel some of the arguments made in the paper aren't very strong. In particular, I'd like the authors to comment on the following:

1) The key difference between their evaluation benchmark and the recommendations in Machado et al. (2018) are that episodes should not have a time limit. The justification for this is that many algorithms might achieve practically optimal performance within this time limit, and so one wouldn't be able to compare algorithms on certain games within significance. They further emphasize that human high scores were achieved without limiting to 30 minutes of play. That said, several algorithms performing similarly within said limit can be instead interpreted as shifting emphasis toward comparing performance on the harder games. As the paper acknowledged, removing the maximum episode length ended up introducing more issues, such as the emulator never ending an episode (due to a supposed bug), as well as increasing the likelihood of the score overflowing. The paper suggested a trick of limiting how long an agent can go without receiving a reward, but it's unclear (1) if needing this fix is worth the proposed change, and (2) if the fix introduces additional game-specific nuances in evaluation; e.g., are there any situations where this can be detrimental to properly evaluating performance, or introduce biases based on a game's reward distribution?

2) The paper gathered a list of human world records for the Atari games in the ALE. In my opinion, this is very valuable for the literature in terms of addressing prior work misrepresenting the competency of an algorithm relative to what humans are capable of; a professional game tester is supposed to be representative of the average game player, who is typically tasked with optimizing fun, whereas speedrunners and scorerunners of games are tasked with optimizing a comparable objective to an RL agent. Beyond this though, I think an alternative conclusion would be to use this information in support of not comparing results to human scores, and to focus on comparisons between algorithms. A considerable number of the human world records have reached the maximum allowable score, over drastically variable gameplay times to achieve these scores, that it might still not be that fair a comparison. Have the authors considered this possibility?

3) Were any other algorithms evaluated on this benchmark, beyond Rainbow? While there are computational considerations, it seems odd for a benchmarking-focused paper to only evaluate one standard algorithm and slight modification of it.

Suggestions

1) The introduction of Rainbow-IQN in this paper feels a little random and out of place given the context created by the rest of the paper's contributions- I feel it might be more appropriate for a benchmarking paper to focus on a representative set of "standard" or relatively simple/trivial algorithms (Like Machado et al. (2018) did) to give a frame of reference for comparing novel ones.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
This paper proposes a new way to benchmark DRL algorithms using the Atari environment which is twofold, one part is a set of emulator recommendations, the other part is what quantity we should consider as a "human reference". The paper also compares Rainbow and Rainbow-IQN, where the IQN improvement matches the proposed human normalized score improvment.

I'm not quite sure how to rate this paper, I have put weak-reject for now, as I don't strongly disagree with anything in the paper, but at the same time:
- the difference to Machados et al. is marginal, but is a bit surprising
- the Rainbow-IQN improvement is too incremental to be considered a significant contribution
- there are some interesting remarks on why Atari is _not_ necessarily a good environment, e.g. most of Section 6, but this clashes with the paper's premise that we should be using Atari.

In a way, this paper reads like an interesting technical review of Atari, but I don't think it provides enough new knowledge to be a conference paper.

Detailed comments:
- I find it a bit weird that the many weaknesses of Atari as a platform are presented at DRL being bad at Atari. The line between environment design and algorithm design can be blurry, but in Atari's case, the weird peculiarities of each game are known to make it an inconvenient benchmark.
- In the same vein, why is Atari+SABER better than other RL environments? This is rather crucial. We should only work on improving a benchmark if it is a useful benchmark, yet, we have many clues that Atari is not.
- The link to TwinGalaxies should be a proper reference with the time of visit, especially if humans break new records in the future.
- Why only compare Rainbow and a variant of Rainbow? I understand compute resources being a limitation, but at the same time, the reasoning behind having standardized testing is to be able to compare a wide variety of algorithms. This paper would be much stronger if it focused on a few representative games (e.g. one reflex game, one hard exploration game, etc.) and tested these games with a bunch of DRL algorithms. That ranking might reveal something very interesting.
- The paper is easy to read, but there are a few grammar mistakes here and there.
review_title: Official Blind Review #1

Meta-review: This paper proposes a new benchmark that compares performance of deep reinforcement learning algorithms on the Atari Learning Environment to the best human players.  The paper identifies limitations of past evaluations of deep RL agents on Atari. The human baseline scores commonly used in deep RL are not the highest known human scores.  To enable learning agents to reach these high scores, the paper recommends allowing the learning agents to play without a time limit.  The time limit in Atari is not always consistent across papers, and removing the time limit requires additional software fixes due to some bugs in the game software.  These ideas form the core of the paper's proposed new benchmark (SABER). The paper also proposes a new deep RL algorithm that combines earlier ideas. 

The reviews and the discussion with the authors brought out several strengths and weaknesses of the proposal.  One strength was identifying the best known human performance in these Atari games.  
However, the reviewers were not convinced that this new benchmark is useful.  The reviewers raised concerns about using clipped rewards, using games that received substantially different amounts of human effort, comparing learning algorithms to human baselines instead of other learning algorithms, and also the continued use of the Atari environment. Given all these many concerns about a new benchmark, the newly proposed algorithm was not viewed as a distraction.

This paper is not ready for publication. The new benchmark proposed for deep reinforcement learning on Atari was not convincing to the reviewers.  The paper requires further refinement of the benchmark or further justification for the new benchmark.
Decision: Reject