Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
Authors: Marin Toromanoff
Emilie Wirbel
Fabien Moutarde
Venue:  (2020)
Keywords: Reinforcement Learning
Deep Learning
Atari benchmark
Reproducibility

Abstract: Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new state-of-the-art performance. Source code is available for reproducibility.

Under review as a conference paper at ICLR 2020
Is DEEP REINFORCEMENT LEARNING REALLY SUPER- HUMAN ON ATARI? LEVELING THE PLAYING FIELD
Anonymous authors Paper under double-blind review
ABSTRACT
Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters well train and test procedures. We then SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman per- formance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading new state- of-t performance. Source code is available for reproducibility.
INTRODUCTION
Human intelligence is able to solve many tasks of different natures. In pursuit of generality in artificial intelligence, video games have become an important testing ground: they require wide set of skills such as perception, exploration and control. Reinforcement Learning (RL) is at the forefront of this development, especially when combined with deep neural networks DRL. One of the first general approaches reaching reasonable performance on many Atari games while using the exact same hyper-parameter and neural network architecture was Deep Q-Network (DQN) (Mnih) et al. 2015), a value based DRI. algorithm which directly takes the raw image as input. This success sparked a lot of research aiming to create better, faster and more stable general algorithms. The ALE (Bellemare et al. 2013), featuring more than 60 Atari games (see Figure I is heavily used in this context. It provides many different tasks ranging from simple paddle control the ball game Pong to complex labyrinth exploration in Montezuma's 's Revenge which remains unsolved by general algorithms up to today.
As the number of contributions is growing fast, it becomes harder and harder to make a proper comparison between different algorithms. In particular, a relevant difference in the training and evaluation procedures exists between available publications. Those issues are exacerbated by the fact that training DRL agents is very time consuming, resulting high barrier for reevaluation of previous work. Specifically, even though ALE is fast at runtime, training an agent on one game takes approxi- mately one week on one GPU and thus the equivalent of more than one year to train on all 61 Atari games. standardization of the evaluation procedure is needed to make DRL that matters pointed out by Hen- derson et al. (2018) for the Mujoco benchmark (Todorov et al.] (012): the authors criticize the lack of reproducibility and discuss how to allow for a fair comparison DRL that is consistent between articles.
DOO
Figure I: ALE Space In- vaders
In this work, first discuss current issues in the evaluation procedure of different DRL algorithms on ALE and their impact. We then propose an improved evaluation procedure, extending the rec- ommendations of Machado et al. (2088), named SABER Standardized Atari BEnchmark for

---

Under review as a conference paper at ICLR 2020
Reinforcement learning. We suggest benchmarking on the world records human baseline and show that RL algorithms are in fact far from solving most of the Atari games. As an illustration of SABER, current State-of-the-ae arrt DRL algorithm Rainbow (Hessel et al. 2018) is benchmarked. Finally, introduce and henchmark on SABER a new state-of-the-art agent: a distributable combination of Rainbow and Implicit Quantiles Network (IQN) (Dabney et al. 2018).
The main contributions of this work are:
The proposal, description and justification of the SABER benchmark.
Introduction of a world records human baseline. We argue it is more representative of the human level than the one used in most previous works. With this metric, we show that the Atari benchmark is in fact a hard task for current general algorithm.
A SABER compliant evaluation current state-of-the art agent Rainbow.
A new oftte-of-thh-ar agent on Atari, Rainbow-IQN, with comparison on SABER to Rainbow, to give an improvement range for future comparisons.
For reproducibility sake, an open-source implementation of Rainbow, Rainbow-IQN, dis- tributed following the idea from Horgan al. (2018).
1.1 RELATED WORK
Reproducibility and comparison in DRL Deep Reinforcement Learning that malters (Hender- son et 2018) is one of the first works warn about reproducibility crisis in the field of DRL. This article relies on the MuJoCo (Todorov et al.] (012)) benchmark to illustrate how some com- mon practices can bias reported results. As continuation the work Hendersso et al.) (2018), J. Pineau introduced a Machine Learning reproducibility checklist (Pineaul (209)) to allow for re- producibility and fair comparison. |aacado et al. (2018) deal with the Atari benehmark. They describe the divergence in training and evaluation procedures and how this could lead to difficulties to compare different algorithms. A first set recommendations to standardize them is introduced, constituting the basis of this work and will be summarized in the next section. Finally, the Github Dopamine (Castro et al., 2018) provides an open-source implementation of some of the current state-of-the-arr the-art algorithms
Atari benchmark, including Rainbow and IQN. An evaluation follow- ing almost all guidelines from Machado et al. (2018) are provided in Castro et al. (2018). However the implementation of Rainbow is partial, and the recommendation of using the full action set not applied. This is why our work contains a new evaluation of Rainbow.
state-of
Value based RL DQN (Miih et al. 2015) is the first value based DRL algorithm benchmarked on all Atari games with the exact same set of hyperparameters (although previous work by Hausknecht ict al. (2014) already performed such benchmark with neural networks). This algorithm relies on the well known Q-Learning algorithm (Watkins & Dayan 1992) and incorporates neural network. Deep Q-learning is quite unstable and the main success of this work is to introduce practical tricks to make it converge. Mainly, transitions stored in replay memory and sampled to avoid correlation in training batch, and a separate targel network is used to avoid oscillations. Since then, DQN has been improved and extended to make it more robust, faster and better. Rainbow (Hessel et al. 2018) is the combination of 6 of these improvements (Van Hasselt et al.] 2016 Schaul et al. 2015 Bellemare et 2017 Wang et al. 2015 Fortunato et al. [2017] Mnih et al. 2016) implemented in a single algorithm. Some ablations studies showed that the most important components were Prioritized Experience Replay (PER) (Schaul et al. (015) and C51 (Bellemare et al., 2017). The idea behind PER is to sample transitions according to their surprise, i.e. the worse the network is at predicting the Q-value of a specific transition. the more we sample it. C51 is the first algorithm in Distributional RL which predicts the full distribution of the Q-function instead of predicting only the mean of it. Finally, IQN (Dabney et al. 2018) is an improvement over C51. It almost reaches on its the performance of the full Rainbow with all components. In C51 the distribution of the Q-function is represented as a categorical distribution while in IQN, it is represented by implicit quantiles.
2

---

Under review as a conference paper at ICLR 2020
2 CHALLENGES WHEN COMPARING PERFORMANCE ON THE ATARI BENCHMARK
In this section we discuss several challenges make a proper comparison between different algo- rithms trained on the Atari benchmark. First, we briefly summarize the initial problems and their solution as proposed by Machado et al. (2018). Then we detail remaining issue not handled by those initial standards, the maximum length time allowed for an episode. Finally, we introduce readable metric, representative of actual human level and allowing meaningful comparison.
2.1 REVISITING ALE: AN INITIAL STEP TOWARDS STANDARDIZATION
Machado et al. (2018) discuss about divergence of training and evaluation procedures on Atari. They show how those divergences are making comparison extremely difficult. They establish recommen- dations that should be used order to standardize the evaluation process.
Stochasticity The ALE environment is fully deterministic, i.c. leading the exact same state if the exact same actions are taken at each state. This is actually an issue for general algorithm evaluation. For example, an algorithm learning by heart good trajectories can actually reach a high score with an open-loop behaviour. To handle this issue, Machado et al. (2018) introduce sticky actions: actions coming from the agent are repeated with a given probability &. leading a non deterministic behavior. They show that sticky actions are drastically affecting performance of an algorithm exploiting the environment determinism without hurting algorithms learning more robust policies like DQN (Mnih et al., (215)). We use sticky actions with probability & 0.25 (Machado ct al. 2018) in our experiments.
End of an episode: Use actual game over In most of the Atari games the player has multiple lives and the game is actually over when all lives are lost. But some articles, e.g. DQN, Rainbow, IQN, end a training episode after the loss of the first life but still use the standard game over signal while testing. This can in fact help the agent to learn how avoid death and is an unfair comparison to agents which are not using this game-specific knowledge. Machado et al. (2018) recommend to use only the standard game over signal for all games while training.
Action set Following the recommendation of Machado et al. (2018) we do not use the minimal useful action set (the set of actions having an effective impact on the current game) as used by many previous works (Mnih et al., 2015 Hessel et al. (0118). Instead we always use all 18 possible actions on the Atari Console. This removes some specific domain knowledge and reduces the complexity of reproducibility. For some games, the minimal useful action set is different from one version to another of the standard Atari library; an issue to reproduce result on breakout was coming from this
GGaetzz 2018).
Reporting performance As proposed by Machado et al. (2018). we report our score while training by averaging consecutive episodes (we have set le 100). This gives information about the stability of the training and removes the statistical bias induced when reporting score of the best policy which is today common practice (Mnih et al. 2015 Hessel al. 2018).
2.2 MAXIMUM EPISODE LENGTH
major parameter left out of the work of Machado et al. (2018): the maximum number of frames allowed per episode. This parameter ends the episode after a fixed number of time steps even if the game is not over. In most of recent works (Hessel al. 2018 Dabney et al., 2018), this set to 30 min of game play and only to 5 min in Revisiting ALE Machado et al. 2018). This means that the reported scores can not be compared fairly. For example, in easy games (e.g. Atlantis, Enduro), the agent never dies and the score is more less linear the allowed time: the reported score will be 6 times higher if capped at 30 minutes instead of 5 minutes.
We argue that the time cap can make the performance comparison non significant. On many games (c.g. Atlantis, Video Pinball) the scores reported of Ape-X (Horgan et al. 2018)), Rainbow (Hessel et al. 2018) and IQN (Dabney et al., 2018) almost exactly the same. This is because all agents
3

---

Under review as a conference paper at ICLR 2020
reach the time limit and get the highest possible score in 30 minutes: the difference in scores is due to minor variations, not algorithmmic difference. As a consequence, the more successful agents are, the more games incomparable because they reach the maximum possible score in the time cap.
This parameter can also be a source of ambiguity and error. The best score on Atlantis (2,311,815) reported by Proximal Policy Optimization by [Schulman al. (2017) but this score almost certainly wrong: it seems impossible to reach it in only 30 minutes! The first distributional paper, C51 [Bellemare al. 2017), also did this mistake and reported wrong results before adding an erratum in a later version on ArXiv.
We argue that episodes should not be capped at all. The original ALE article (Bellemare al. 2013 pg.3) states that This functionality is needed for a small number of games to ensure that they always terminate. On some famously hard games like Pitfall and Tennis, random exploration leads to much more negative reward than positive and thus the agent effectively learns to do nothing, e.g. not serving in Tennis. We claim that, even with this constraint, agents still end up learning to do nothing, and the drawback of the cap harms the evaluation of all other games. Moreover, the human high scores for Atari games have been achieved in several hours of play, and would have been unreachable if limited to 30 minutes.
To summarize, ideally one would not cap all length of episode while training and testing. However this makes some limitations of the ALE environment appear, as described in the following paragraph.
Glitch and bug in the ALE environment
When setting the maximum length of an episode to infinite time, the agent gets stuck some games, iee. the episode never ends, because of bug the emulator. this case, even doing random actions for more than 20 hours neither gives any reward nor end the game. This happens consistently BattleZone and less frequently on Yar's Revenge. One unmanaged occurrence of this problem is enough hamper the whole training of the agent. It is important note that those bugs were discovered by chance and it is probable that this could happen on some other games.
We recommend to set the maximum episode length to infinite (in practice, a limit of 100 hours used). Additionally maximum stuck time of 5 minutes. Instead of limiting the allowed time for the agent, we limit
suggest
Normalized score to beginner baseline
10k
the time without receiving any reward. This small trick handles all issues exposed above, and sets all reported scores on the same basis, making comparison to world records possible.
000
100
Other bugs or particularities harming evalua- tion were encountered while training on the full Atari benchmark: buffer rollover with sudden negative score, influence of a start key for some games, etc. They are detailed and discussed in the supplementary material and we argue that they can have drastic impact performance and explain inconsistencies.
10
20
20
40
50
Games sorted by score
Figure 2: World records scores the usual be- ginner human baseline (Mnih et al. 2015) (log scale).
2.3 HUMAN WORLD RECORDS BASELINE
common way evaluate AI for games to let agents compete against human world cham- pions. Recent examples for DRL. include the victory of AlphaGo versus Lee Sedol for Go Silver et 2016), OpenAI on Dota (OpenAl 2018) or AlphaStar versus Mana for StarCraft 2 (Vinyals et al. 2019). In the same spirit, one of the most used metric for evaluating RL agents on
Atari is to compare them to the human baseline introduced by Mnih et al. (2015). Previous works the normalized human score, ie. 0% is the score of random player and 100% is the score of the human baseline, which allows to summarize the performance on the whole Atari set one number, instead of individually comparing scores for each of the 61 games. However argue
4

---

Under review as a conference paper at ICLR 2020
that this human baseline is far from being representative of the best human player, which means that using it to claim superhuman performance is misleading. The current world records are available online for 58 of the 61 evaluated Atari game Evaluating these world records scores using the usual human normalized score has a median of 4.4k% and a mean of 99.3k% (see Figure for details), to be compared to 200% and 800% of original Rainbow (Hessel et al. 2018). As consequence, argue that using normalized human score with the world records will give a much better indication of the performance of the agents and the margin of improvement. Note that games of the ALE (double dunk, elevator action and tennis) do not have a registered world record, so all following experiments contain 58 games.
3 SABER A STANDARDIZED ATARI BENCHMARK FOR REINFORCEMENT LEARNING
In this section we introduce SABER, set of training and evaluation procedures the Atari bench- mark allowing for fair comparison and for reproducibility. Moreover, those procedures make possible compare with the human world records baseline introduced above and thus obtain accurate idea the gap between general agents and best human players.
3.1 TRAINING AND EVALUATION PROCEDURES
All recommendations stated in the previous section are summarized in Table [ to constitute the SABER benchmark. It is important note that those procedures must be used at both training and test time. The recent work Go-Explore (Fcoffet et al.) 2019) opened a debate on allowing or not stochasticity training time. They report state-of-the-ar performance on the famously hard game Montezuma'ss 's Revenge by removing stochasticity at training time. They conclude that we should have benchmarks with and without it (Ecoffet Adrien & Clune 2018). We choose to use same conditions for training and testing general agents: this is more in line with realistic tasks.
3.2 REPORTING RESULTS
In accordance with previous guidelines, we ad- vocate to report mean scores of 100 consecutive training episodes at specific time, here 10M, 50M, 100M and 200M frames. This removes the bias of reporting scores of the best agent en- countered during training and makes it possible to compare at different data regimes. Duc to the complexity of comparing 58 scores a syn- thetic manner, try to provide a single met- ric to make an effective comparison. Mean and median normalized scores to the records base- line are computed over all games. Note that the median is more relevant: the mean is highly im- pacted by outliers, in particular by games where the performance is superhuman. For the mean value, games with infinite game time and score are artificially capped to 200% of the records baseline. We propose to add a histogram of the nor- malized score, classify the games according to their performance. We define 5 classes: failing (< 1%), poor (< 10%). medium (< 50%), fair (< 100%) and superhuman (> 100%). Medians, means and histograms can be found Section 5 and the fully detailed scores are available in the supplementary materials.
Table 1: Game parameters of SABER
Parameter
Value
Sticky actions
{ = 0.25
Life information
Not allowed
Action set
18 actions
Max stuck time
5 min (18000 frames) Infinite (100 hours)
Max episode length
Initial state and ran-
Same starting state and varying seed
dom seed
RAINBOW-IQN
Two different approaches were combined to obtain improvement over Rainbow (Hessel al., 2018: Rainbow itself and IQN (Dabney et al. 2018) because of its excellent performance. Im-
on the TwinGalaxies website
twi http://wwww.w.....a.aa.a..//t/ta

---

Under review as a conference paper at ICLR 2020
plementation details and hyper- parammerraetterss are described in the supplementary material. Both our implementations of Rainbow and Rainbow-IQQ IQN are distributed 2 following Ape-X (Horgan et al, 2018) and based on the implementation of (Castro et al. 2018).
IQN is an evolution of the C51 algorithm (Bellemare et al. 2017) which is one of the 6 components of the full Rainbow, this is natural upgrade. After the implementation, preliminary tests high- lighted the impact of PER (Schaul et al. 2013): taking the initial hyper-parameters for PER from Rainbow resulted in poor performance. Transitions are sampled from the replay memory propor- tionally to the training loss to the power of priority exponent w. Reviewing the distribution of the loss shows that it significantly more spread for Rainbow-IQU training unstable, because some transitions were over-sampa sampled. To handle this issue, 4 values of w were tested on games: 0.1, 0.15, 0.2, 0.25 instead of 0.5 for original Rainbow, with 0.2 giving the best performance. The 5 games were Alien, Battle Zone, Chopper Command, Gopher and Space Invaders. All other parameters were left as Rainbow-IQN is evaluated on SABER and compared to Rainbow in the following section.
than for Rainbow, thus making the
5 EXPERIMENTS
In this section, we describe the experiments performed on SABER. For all parameters not mentioned in SABER (e.g. the action repeat, the network architecture, the image preprocessing, etc) care- fully followed the parameters used in Rainbow (Hessel et al. 2018)) and IQN (Dabney et al. 2018) papers. Those details and the scores for each agent and individual games can be found in the supple- mentary materials. Training one agent takes slightly less than a week, which makes a full benchmark use around year- As a consequence, for each algorithm benchmark, trainings were run with only one seed for the full benchmark, and seeds for 14 of the 61 games. Details on the choice of these games and the associated scores can be found in Section 5..3 The combined duration of all experiments conducted for this article is more than 4 years- years-GPP.. Agents are trained using SABER guidelines on the 61 Atari games, and evaluated with the records baseline for 58 games. Scores at both 5 minutes and 30 minutes are kept while training to compare to previous works.
5.1 RAINBOW EVALUATION
Algorithm
Original Rainbow (Hessel al. 2018)
Following (Machado et al. 2018)
Median
Mean
Superhuman
Median
Mean
Superhuman
Performance 4.20%
24.10%
2
2.61% %
17.09%
1
Table 2: Median and mean human-normalized performance and number of superhuman scores
(> 100%). Scores are coming from the original Rainbow and from our re-evaluation of Rainbow following recommendations of Machado et al. (30 minutes evaluation, at 200M training frames).
Benchmarking Rainbow makes it possible measure the impact of the guidelines of Machado al.: sticky actions, ignore life signal and full action set. Table _ compares the originally reported performance of Rainbow (Hessel et al. 2018) to an evaluation following the recommendations of Machado et al. The performance is measured with the records baseline, for 30 minutes evaluation 200M training frames, to be as close as possible to the conditions of the original Rainbow. The impact of the standardized training procedure is major: as shown in the following paragraph, the difference median (1.59%) is comparable to the difference between DQN and Rainbow (1.8%, see Figure 5) when both are trained on same training procedures. This demonstrates the importance of explicit and standardized training and evaluation procedures.
5.2 RAINBOW-IQN: EVALUATION AND COMPARISON
Influence of maximum episode length Table studies the influence of the time limit for the evaluation, by reporting performance for Rainbow and Rainbow-IQN depending on the evaluation time. A significant difference can be seen between 5, 30 minutes and without limiting time of evaluation, which confirms the discussion of Section 2.2
"ee supplementary materials for details

---

Under review as a conference paper at ICLR 2020
Time
min Median
30 min
No limit (SABER)
Mean
Super.
Median
Mcan
Super.
Median
Mcan
Super.
Rainbow
2.35%
14.86%
2.61% %
17.09%
2.83%
24.54% 3
Rainbow-IQN 2.61% %
17.62%
2.81% %
20.18%
3.13%
30.89% 4
Table Evolution of performance with evaluation time (mean, median of normalized baseline and number of superhuman agents) for Rainbow and Rainbow-IQN,
Median performance with regards to training frames
3.5%
Rainbow
3.0%
Rainbow-IQN
Normalized score
2.5%
2.0%
1.5%
1.0%
0.5%
0.0%
50M
100M
150M
200M
Training frames
Figure 3: Comparison of Rainbow and Rainbow-IQN SABER: Median normalized scores with regards to training steps.
Comparison to Rainbow As introduced in Section 3.2 we compare Rainbow and Rainbow-IQU IQN with median and mean metrics on SABER conditions, and with a classification of the performance of the agents in Figure Jisure3]shows that Rainbow-IQN performance during training is consistently higher than Rainbow. One can notice on Figure that the majority agents are in the poor and failing categories, showing the gap that must be crossed to achieve superhuman performance on the ALE.
Rainbow
20
Rainbow IQN
Number of games
15
(11844))
Performance range (in normalized professional baseline)
Figure 4: Comparison of Rainbow and Rainbow-IQN on SABER: classifying performance of agents relatively to the records bascline (at 200M training frames).
Comparison to DQN Figure 5 provides comparison between DQN, Rainbow and Rainbow- IQN. The evaluation time is set at 5 minutes to consistent with the reported score of DQN by Machado et al. (018). As expected, DQN is outperformed for all training steps. As aforementioned.

---

Under review as a conference paper at ICLR 2020
the difference between DQN and Rainbow is in the range as the difference coming from divergent training procedures, showing again the necessity for standardization.
DQN
Normalized score
Rainbow-IQN
1.0%
0.5%
0.0%
10M
50M
1CDM
2COM
Training frames
Figure 5: Median performance comparison for DQN, Rainbow and Rainbow-ION with regards to training frames. Evaluation time is set at 5 minutes to allow a comparison to DQN.
5.3 STABILITY BOTH RAINBOW AND RAINBOW-IQN
Machado et al. (2018) use 5 different seeds for training to check that the results are reproducible and stable. For this article, these 5 runs are conducted on both Rainbow and Rainhow-IQN for 14 games (around 25% of the whole benchmark). It would be best to have the whole benchmark on seeds but this was way above our computational resources. Still, these games allow us to make first step of stability studies. They chosen according the results of the first seed, with the idea of prioritizing games on which scores were most notably different between Rainbow and Rainbow-IQN. We also try choose diverse games from different categories (from failing to superhuman) and we removed the 5 games used for the hyperparameter tuning. Games that were either hard (such Montezuma's 's Revenge Pitfall) too simple (such as Pong Atlantis) intentionally excluded to make the additional tests as significant as possible. For each game with seeds conducted, we computed the median and mean Human-normalizer performance averaged over the 5 trials. This way, we can both have a reasonable estimation of the stability of the trainings, and a comparison as fair possible between Rainbow and Rainbow-IOQ
Median per formancce with regards tnaining frames
Rambow
3.0%
Rambow seeds
Raibbow-igN
Rembow-iQN
2.5%
Nornalized soore
2036
2.5%
2.0%
0.5%
Training frames
Figure 6: Median normalized scores with regards training steps averaged over 5 seeds for both Rainbow and Rainbow-IQN, Only the 14 games which 5 seeds have been conducted were used for this figure.
Figure shows the median averaged over 5 trials for both Rainbow and Rainbow-IQN We also plot each seed separately and the standard deviation over the 5 seeds. This has been computed only
8

---

Under review as a conference paper at ICLR 2020
the 14 games which we succeeded conduct 5 runs. This shows that standard deviations roughly similar for Rainbow and Rainbow-IQN, around 0.2 % on the world record baseline. As these standard deviations are rather small for 25% of the Atari games, can assume they would be still small on the whole benchmark. We think that this reveals that both Rainhow and Rainbow-IQN quite stable on Atari and strengthens our confidence the-art on the Atari benchmark. In particular, Rainbow-IQN reaches infinite game time on Asteroids on all 5 trials whereas Rainbow fails for each seed.
Rainbow-IQN being the new state-of-
6 CONCLUSION: WHY IS RL THAT BAD AT ATARI GAMES?
In the current work, we confirm the impact of standardized guidelines for DRL evaluation, and build consolidated benchmark, SABER. The importance of the play time is highlighted: agents should be trained and evaluated with no time limitation. To provide a more significant comparison, new baseline is built, based on human world records. Following these recommendations, we show that the state-of-the-arr Rainbow agent in fact far from human world records performance. As a further illustration, we provide an improvement, Rainbow-IQN, and use it to measure the impact of the evaluation time over performance.
The striking information from these results is that general DRL algorithms far from best human performance. The median of world records human normalized score for Rainbow-IQN is 3,1 %, meaning that for half of the games, the agent is only 3% of the way from random play to the actual best human play. There are many possible reasons for this failure, which we will briefly discuss here to give an intuition of the current limitations of general DRL algorithm.
Reward clipping In some games the optimal play for the RL algorithm is not the same as for the human player. Indeed, all rewards are clipped between -1 and I so RL agents will prefer to obtain many small rewards over a single large one. This problem is well represented in the game Bowling: the agent learns to avoid striking or sparing. Indeed the actual optimal play is to perform 10 strikes in a row leading to one big reward of 300 (clipped to 1 for the RL. agent) but the optimal play for the RL agent is to knock off bowling pins one by one. This shows the need of better way to handle reward of different magnitude, by using an invertible value function as suggested by PPollen et al. 20188 or using Pop-Art normalization (van Hasselt et al. 2016).
Exploration Another common reason for failure is a lack of exploration, resulting in the agent getting stuck in a local minimum. Random exploration or Noisy Networks (Fortunato et al. 2017) are far from being enough to solve most of Atari games. In Kangaroo for example, the agent learns to obtain rewards easily on the first level but never tries to go to the next level. This problem might be exacerbated by the reward clipping: changing level may yield a higher reward, but for the RL algorithm all rewards are the same. Exploration is one of the most studied field in Reinforcement Learning, so possible solutions could rely on curiosity (Pathak et al. 2017) or count- based explo- ration (Ostrovski et al. 2017).
Human basic knowledge Atari games designed for human players, so they rely implicit prior knowledge. This will give a human player information on actions that are probably positive, but with no immediate score reward (climbing ladder, avoiding skull etc). The most representative example can be seen in Riverraid: shooting a fuel container gives an immediate score reward, but taking it makes it possible to play longer. Current general RL agents do not identify it as a potential bonus, and so die quickly. Even with smart exploration, this remains an open challenge for any general agent.
Loop on a sub-optimal policy Finally, we discovered that on some games the agent finds quickly a loop continuously giving small amount of reward and spends the whole training this loop. In Bank Heist for example, the agent understood that bonus were respawning when changing level. Therefore the agent learned to just take over and over the same bonus until game timeout, failing to reach a good score. A very similar behaviour was discovered on Elevator Action, Kangaroo, Krull and Titankham.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marta Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Martin Wicke, Yuan Yu, Xiaoqiang Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2016.
Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 47:253 17:253-2799 2013,
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 449-458. JMLR. org, 2017.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G Bellemare. Dopamine: research framework for deep reinforcement learning. arXiv preprint arXiv: 10, 2018.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks distributional reinforcement learning. arXiv preprint arXiv:1806.06923. 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for
problems. CoRR, abs/1901. 2019.
Lehman Joel Kenneth O. Stanley Ecoffet Adrien, Huizinga Joost and Jeff Clune. Montezumas Revenge Solved by Go-Explore, New Algorithm for Hard Exnd--xplooation Problems. httpa://eng uber com/
go-explore/ 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: 1706. 10295, 2017.
Fabio M. Gractz. How to match DeepMinds Deep Q-Learning score in Breakout. https://tinyur? com com/yymceamo 2018.
Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone. A neuroevolution approach to general atari game playing. IEEE Transactions on Computational Intelligence and Al in Games, 6(4):355-366, 2014.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep rein- forcement learning that matters. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Mattco Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv: 103.00093,, 2018
Kaixhin. Open source implementation of rainbow by kaixhin, 2018.
URL https:://gtthtii com/
Ka xhi n /Raai nbow
Marlos C Machado, Mare G Bellemare, Erik Talvitic, Joel Veness, Matthew Hausknceht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Maih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Interna- tional conference on machine learning, pp. 1928 1937, 2016.
OpenAL. OpenAl Five. htttp:://ppnai. cowp com/ faemm/iive] five/ 2018.
Georg Ostrovski, Mare Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2730.. JMLR. org, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation pytorch. 2017.
10

---

Under review as a conference paper at ICLR 2020
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darell. Curiosity-driven exploration by self- supervised prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17,20 2017 2017.
J. Pineau. Reproducibility checklist.
https://www.co wttp///wwccc..c mmw. wt..ggg11.caa/a/o jeaaa-jpneaas
2019.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Veerk, Matteo Hessel, Rmi Munos, and Olivier Pietquin. Observe and look further: Achieving consistent performance atari. 2018.
Redis. Redis database. https://redis. io7 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952.222 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 17Xiv:170..06377, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panncershelvam, Marc Lanctot, al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 IEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Hado P van Hasselt, Arthur Guez, Arthur Guez, Mattco Hessel, Volodymyr Mnih, and David Sil- Learning values across many orders of magnitude. In D. D. Lee, M. Sugiyama. U. V. Luxburg. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29. pp. 4287-4295, Curran Associates. Inc.. 2016. URL
pta.pps.ccpappr/ pdf
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg. Wojeiech M. Czar- necki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Fwalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu. Demis Hassabis, and David Silver. AlphaStar: Mastering the Time Strategy Game StarCraft II. hhttps: /ttp://eeeeeeee///// ndd. com/blog/ aophastar-msstereeee
real-eraeeeooooe imm- www.atooyy.aaee-ffarrrreeeeee-ee wite---ttrrrrfftt-e waw.ayy--aaa----e
2019.
Ziyu Wang, Tom Schaul, Mattco Hessel, Hado Van Hasselt, Marc Lanetot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 1. 2015.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-29,, 1992.
A SUPPLEMENTARY MATERIALS: IMPLEMENTATION DETAILS
A.1 RAINBOW APE-X
Practically, we started with the Porcrch Paszke et al. 2017) open source implementation of Rain- bow coming from Kaikhin (Kaixhin 2018). We tested this initial implementation on some games with the exact same training conditions as in the original Rainbow to ensure our results were consis- tent. After this sanity check, we implemented a distributed version of Rainbow following the paper Distributed Prioritized Experience Replay (Ape-X) (Horgan et al.) 2018). Ape-X (Horgan et al.) 2018) is a distributed version of Prioritized Experience Replay (PER) but which can be adapted on any value-based RL algorithm including PER, e.g. Rainbow. There is study of this in the main article because we lacked time and computing resources to run experiments on whole Atari set with distributed actors. However, some experiments were conducted to ensure implementation was working as expected. These experiments are detailed in the next section. We claim that our Ape-X implementation is an important practical improvement compared to the single
distributed
11

---

Under review as a conference paper at ICLR 2020
agent implementation of both Dopamine (Castro al. 2018) and Kaikhin (Kixhhin 2018). It is important to note that all the experiments detailed in the main paper have been made with a single actor and thus do not really show the interest of distributed Rainbow Ape-X. A lock was added to synchronize all single-agent experiments to ensure that one step of learner is done every 4 steps of actor as in the original Rainbow (Hessel al. 2018). All hyperparameter values match closely those reported in Rainbow (Hessel et al. 20188). There is still one difference coming from our Ape-X implementation (even using single actor). Indeed, we compute priorities before putting transitions in memory instead of putting new transitions with the maximum priorities seen as in the original Rainbow (Hessel et al., 2018). We argue that this should not have much impact on single-actor setting and that it is straightforward to implement for each algorithm using Prioritized Experience Replay (Schaul et al. 2015).
For the distributed memory implementation, we use key-memory database with REDIS (Redis) 2019]. The database kept in RAM, which makes access faster and possible for the ALE consid- ering the size of the images and the replay memory size.
A.2 RAINBOW-IQN APE-X
We combined our Rainbow Ape-X implementation with IQN (Dabney al., 2018) coming from the TensorFlow (Abadi et al. (016) open source implementation of Dopamine (Castro et al. 2018) to obtain a PyTorch (Paszke al. 2017) implementation of Rainbow-IQN Ape-X. All our hyperparam- eter values match closely those reported in IQN. As indicated in the main paper. we had to tune the priority exponent coming from Prioritized Experience Replay (Schaul al. (2015] the training stable. We tested both value of learning rate and epsilon of the adam optimizer from Rainbow and from IQN. A minor improvement in performance was found with the learning rate of IQN (Dabney et al. (018) (tested only on 3 games for computational reasons), which was then used for our experiments.
order make
B EXPERIMENTS
B. IMAGE PREPROCESSING AND ARCHITECTURE
We used the same preprocessing procedure used in Rainbow and IQN, i.c an action repeat of 4, frames are converted grayscale. resized to 84#84 with bilinear interpolation p and max-pooled over 2 adjacent frames. The actual input to our network consists in 4 stacked frames.
Our architecture followed carefully the one from the original DQN for the main branch which was also used in Rainbow and IQN. The branch responsible of implicit quantiles is made exactly as the one from the original implementation section of IQN (Dabney al., 2018 p.g. 5)
B.2 TRAINING INFRASTRUCTURE
The training of the agents was split over several computers and GPUs, containing in total:
3 Nvidia Titan X and 1 Nvidia Titan V (training computer)
1 Nvidia 1080 Ti (local workstation)
2 Nvidia 1080 (local workstations)
3 Nvidia 2080 (training computer)
4 Nvidia P100 (in remote supercomputer)
2 Nvidia V100 (in a remote supercomputer)
4 Nvidia Tesla V-100 (DGX station)
4 Nvidia Quadro M2000 (local workstations)
for some experiments we made this interpolation using the Python image library PIL instead of OpenCV because OpenCV was not available the remote supercomputer. This was leading to small differences in the final resized image.
12

---

Under review as a conference paper at ICLR 2020
B.3 RAINBOW-IQN APE-X
To ascertain our distributed implementation of Rainbow-IQN was functional, experiments were conducted with multiple actors (10 actors instead of one). All locks and synchronization processes are removed to let actors fill the replay memory as fast as possible. The experiments are stopped when the learner reaches the same number of steps as in single-agent experiments.
Table reports the raw scores obtained by the agents on the selected games. Although the same number of batches is used in the training, there is huge improvement in performances for the 3 games tested over the single agent version. This confirms the results coming from the Ape-X X (Horgan et al. 2018) paper. Even at same learner step, the agent can benefit greatly from more experiences coming from multiple actors. Thanks to PER, the learner focuses on the most important transitions in the replay memory. Moreover this could avoid being stuck in local minimum as assumed in Ape-X X (Horgan et al.) 2018). For the 3 experiments done, all actors together played around 6 times more than in our single-agent setup, leading to 1,2B frames instead of 200M.
Table 4: Raw agents scores after training Rainbow-IQ2 Ape-X with 10 actors or single synchro- nized actor
Raw score Game
Multi-agent
Single agent
Asterix Ms Pacman
274,491
28,015 6,090.74
9,901
Space Invaders
24,183
7,385.4
C GLITCH AND BUG IN THE ALE
Inconsistent game behaviors and bugs were encountered while benchmarking Rainbow and Rainbow-IQN on all Atari games. The most damageable is the one described in the main arti- cle: games getting stuck forever even doing random actions. This one of the main reasons why the maximum stuck length parameter is introduced.
Another issue is the buffer rollover: the emulator sends a reward of -IM when reaching 1M, effec- tively making the agent goes to 0 score over and over. For example, for our first implementation of Rainbow on Asterix, the scores were going up to MM, then suddenly collapsing to random values between 0 and IM. However, the trained agent was in fact playing almost perfectly and indeed resolving the game many times before dying. This can also be observed in the reported score of Asterix by both Ape-X (Horgan et al.) 2018) and Rainbow (Hessel et al. 2018): the score goes up to IM and then varies randomly. This is an issue to compare agent, because weaker agent could actually be reported with higher score. We found this kind of buffer rollover bug 2 others games: Video Pinball and Defender. To detect this in potential other games, we advocate to keep track of really high negative rewards. Indeed the 61 games evaluated, there are no game which there is reward inferior to 1000. And if it happens, most probably this is a buffer rollover and this reward should be ignored.
Additionally, on many games (such as Breakout for example), a specific key must be pressed to start the game (most of the time the Fire button). This means that agent can easily get stuck for long time because it does not press the key. This impacts the stability of the training because the replay memory is filled with useless transitions. We argue that this problem is exacerbated by not finishing episode loss of life. Indeed there are many games where specific key must be pressed, but only after losing a life to continue the game. Moreover this is probably harder to learn with the whole action set available, because the number of actions iterate is higher than with the minimal useful action set. This is definitely not a bug, and a general agent should learn to press fire to restant start game.
13

---

Under review as a conference paper at ICLR 2020
D DETAILED EXPERIMENTAL FIGURES
In this section, we provide more detailed versions of the figures in the main article. The structure of this section follows the one of Section 5 of the main article.
As reminder, all normalized world record baseline scores are reported according to the following equation, where we note r the score random agent, w the score of the world record, and a the score of the agent to be evaluated:
F
(I)
W
D.1 RAINBOW EVALUATION
Figure illustrates in more details the difference between the reported original performance of (Hessel et al. 2018) (reported in the world record baseline), and the one obtained when applying
the recommendations of (Machado al. 2018). In particular, the number of failing games is much lower for the original implementation. Higure gives the breakdown for each game of the ALE.
Agents performance classification for Rainbow with and without recommendations
Original Rainbow
Ramnbow recommendations
20
Number agents
15
Failing (<136)
Poor (1-10ke)
(10-50le) (50-100%) Superhuman (-100%)
range professional baseline)
Figure 7: Agents performance comparison for the original Rainbow (Hessel et al., 2018 versus Rainbow trained with (Machado et al. 2018) guidelines (30 minutes evaluation time to align with original conditions)
D.2 RAINBOW-IQN: EVALUATION AND COMPARISON
Influence of maximum episode length Figure details the influence of evaluation time the performance range of the agents. As expected and discussed in the main article, evaluation time has strong impact the normalized performance of the agents. In particular, agent reaches superhuman performance before 30 minutes evaluation. More agents reach superhuman performance when the evaluation time is not capped (in particular the ones that never stop playing, see next paragraph).
Comparison of Rainbow and Rainbow-IQN Figure TO details the difference in performance between Rainbow and Rainbow-IQN on SABER conditions, 200M training frames. Note that superhuman, never ending scores artificially capped at 200% of the bascline. The most drastic difference is found on the game asteroids, which goes from failing to superhuman performance.
Some failing games are still significantly improved: for example, space invaders is increased of roughly a factor 3. To highlight these improvements, we compare Rainbow-IQN to Rainbow by using normalized baseline similar to the world record baseline, but using Rainbow scores reference. So if we note r the score a random agent, R the score of a Rainbow agent and the score of Rainbow-IQN agent, then the normalized score is:
14

---

Under review as a conference paper at ICLR 2020
Origial Ra nbew recemmendations
190.0%
100.0%
60.0%
-50.0%
-100.0%
Figure 8: Performance comparison per game between the original Rainbow (Hessel et al. 2018 versus Rainbow trained with (Machado et al. 2018) guidelines (30 minutes evaluation time to align with original conditions)
Agents performance classification Rainbow depending on evaluation time
30 minutes
Number of agents
Infinite
Falling (c19))
Poor (1-10%)
Medium (10-50%)
Fair (50-1009)
Superhumen (>1 (10%))
Performance range (in normalized professional baseline)
Figure 9: Evolution of agents performance classification with evaluation time: Rainbow-IQN, 200M training frames, evaluation time ranging from 5min SABER conditions
I --n r R
Note that we use the absolute value because the game Skiing, the Rainbow agent is worse than the random agent. The details per game be found in Figure 11 Note that games that are already superhuman in Rainbow are skipped, and that the Asteroids games, which is failing in Rainbow, becomes superhuman and is skipped in the figure for visualization purposes.
15

---

Under review as a conference paper at ICLR 2020
200.0%
Rainbow Rainbow -Qo
150.0%
Normalized
00.0%
30.0%
0.0%
10.0%
0.0%
-50.0%
Blan
the
line
Figure 10: Performance comparison per game between Rainbow and Rainbow-IQN on SABER conditions (200M training frames)
Normalized performance [Rainbow bageline)
800%
500%
400%
300%
200%
100%
0%6
100%
200%
100 SCOOND
wede g KOND
waE warte
fume Taapss
Suppu 20 maneen scae Peasni cortue
gan
sasse Tam
Soad
JOAIY D00
Steps
21000
Figure 11: Rainbow-IQN normalized with regards to Rainbow baseline for each game
D.3 STABILITY OF BOTH RAINBOW AND RAINBOW-IQN
The 14 games on which we ran trials for both Rainbow and Rainbow-IQN are: Asteroids, Cen- tipede, Demon Attack, Frostbite, Gravitar, Jamesbond, Krull, Kung Fu Master, Ms Pacman, Private Eye, Seaquest, Up N Down, Yars Revenge and Zaxxon.
E RAW SCORES
For verification purposes, we provide tables containing all relevant agent scores used to build the figures from the principal article.
Baseline scores Table contains all raw game scores for ALE games, both for the previous human baseline (Mnih et al., 2015) and the new proposed world record baseline from TwinGalaxies. Note that some of the scores missing for some games (marked as NA). For the world record baseline scores, some them were extrapolated from the reported world record and are marked with a *. Indeed, some world records report the play time other metrics (e.g. the distance travelled for
16

---

Under review as a conference paper at ICLR 2020
Enduro) instead of the raw score of the game. Note that all agents trained and reported all games of the ALE, even the world record baseline computed for 58 games.
SABER raw scores for Rainbow-IQN Table 6 for Rainbow-IQN. A few of these games (Atlantis and Defender and Asteroids for Rainbow-IQN) successfully keep playing with a positive score increase after 100 hours, so their raw scores are infinite. They are marked as infinite gameplay in the table, and capped at 200% of the world record baseline for the mean computation.
contains all raw agents scores for ALE games
Evolution of scores with time Table 7 compares agents scores with increasing evaluation times for Rainbow and Rainbow-IQN, at 200M training frames.
Evolution of scores with training frames Table (resp. Table 2 contains all raw agents scores for ALE games for Rainbow-IQN. with an evaluation time of 5 minutes (resp. 30 minutes), after 10M, 50M, 100M and finally 200M training frames.
17

---

Under review as a conference paper at ICLR 2020
Agent Category
Game Name
Random
Mnih et al.
2015
World Record
air raid
579.25
NA
23050.0
alien
211.9
7127.7
251916.0
amidar
2.34
1719.5
104159.0
assault
283.5
742.0
8647.0
asterix
268.5
8503.3
1000000.0
asteroids
1008.6
47388.7
10506650.0
atlantis
22188.0
29028.1
10604840.0
bank heist
14.0
753.1
82058.0
battle zone
3000.0
37187.5
801000.0
beam rider
414.32
16926.5
999999.0
berzerk
165.6
2630.4
1057940.0
bowling
23,48
160.7
300.0
boxing
-0.69
12.1
100.0*
breakout
1.5
30.5
864.0
carnival
700.8
NA
2541440.0
centipede
2064.77
12017.0
1301709.0 O0
chopper command
794.0
7387.8
999999.0 O
crazy climber
8043,0
35829.4
219900.0
defender
4142.0
18688.9
6010500.0
demon attack
162.25
1971.0
1556345.0
double dunk
-18.14
-16.4
NA
elevator action
4387.0
NA
NA
enduro
0.01
860.5
9500.0
fishing derby
993.06
-38.7
71.0
freeway
0.01
29.6
38.0
frostbite
73.2
4334.7
454830.0
gopher
364.0
2412.5
355040.0 O
gravitar
226.5
3351.4
162850.0
hero
551.0
30826.4
1000000.0
ice hockey
-10.03
0.9
36.0
jamesbond
27.0
302.8
45550.0
journey escape
19977.0
NA
4317800 1317804.0
kangaroo krull kung fu master
54.0
3035.0
1424600.0
1566.59
2665.5
104100.0
451.0
22736.3
1000000.0
montezuma revenge
0.0
4753.3
1219200.0
ms pacman name this game
242.6
6951.6
290090.0
2404.9
8049.0
25220.0
phoenix
757.2
7242.6
4014440.0
pitfall
-265.0
6463.7
114000.0
pong
-20.34
14.6
21.0"
pooyan
371.2
NA
13025.0
private eye
34,49
69571.3
101800.0
qbert
188.75
13455.0
2400000.0
riverraid
1575.4
17118.0
1000000.0
road runner
7.0
7845.0
2038100.0
robotank
2.24
11.9
76.0
seaquest
88.2
42054.7
999999.0 O
skiing
166267.91
-4336.9
-3272.0
solaris
2346.6
12326.7
111420.0
space invaders
136.15
1668.7
621535.0
star gunner
631.0
10250.0
77400.0
tennis time pilot
-23,92
-8.3
NA
3682.0
5229.2
65300.0
tutankham
15.56
167.6
5384.0
up n down
604.7
11693.2
82840.0
venture vidco pinball wizard of wor
0.0
1187.5
38900.0
15720.98
17667.9
89218328.0
534.0
4756.5
395300.0
yars revenge
3271.42
54576.9
15000105.0
7axxon
8.0
9173.3
83700.0
Table 5: Raw scores for ALE games, for random agent, the beginner baseline and the world records. indicates games on which score has been extrapolated from the reported world record (the time of visit was 25 July 2019).
18

---

Under review as a conference paper at ICLR 2020
Training frames
Game name
10M
50M
100M
200M
raid
7765.25
11690.0
13434.25
12289.75
alien
2740.6
1878.1
5223.0
7046.4
amidar
347.13
1554,84
2129.27
3092.05
assault
966.87
2783.49
4443.03
6372.7
asterix
3467.0
9280.0
16344.5
28015.0
asteroids
1194. 16 (98.25)
3261.88 (2602.48)
Infinite gameplay Infinite gameplay
Infinite gameplay Infinite gameplay
atlantis
Infinite gameplay
Infinite gameplay
bank heist
756.4
1325.3
1402.2
1412.4
battle zone
33000.0
36730.0
33480.0
44410.0
beam rider
11510.78
11900.7
10042.74
9826.62
berzerk
546.7
697.0
640.2
892.9
bowling
29.64
30.0
29.86
29,92
boxing
92.71
98.62
98.92
98.7
breakout
53.77
121.83
132.56
175.47
carnival
5148.7
4824.1
4851.3
4566.3
centipede
2241.70 (251.56)
4099.89 (405.19)
4720.54 (626.31)
5260.96 (920.10)
chopper command
3018.0
6523.0
9053.0
11405.0
crazy climber
863
118038.0
133114.0
144437.0
defender
Infinite gameplay 3433.13 (656.97)
Infinite gameplay 6616.96 (2949.90)
Infinite gameplay 8267. .2 (3065.27)
Infinite gameplay
demon attack
24599.31 (17441.86)
double dunk
-5.54
0.3
1.52
1.3
elevator action
2.0
0.0
43490.0
77010.0
enduro
1380.05
3867.42
5014.49
5146.73
fishing derby
22.11
34.82
48.11
49.08
freeway
32.65
33.9
33,95
33,96
frostbite
4351.54 (1456.01)
9135.10 (1611.78) 1.78)
9768.22 28 (1742.88)
10002.78 78 (1752.75)
gopher
4798.4 4
15629.8 8
141 46.30.0
15797.6 6
gravitar
283.70 (56.37)
1258.90 (228.31)
1725 ,0 (471 .0)
1973.60 (61 (614,80)
hero
13728.55
27450.65
28759.85
28957.4 44
ice hockey
-2,43
1.8
-0,72
-0.07
jamesbond
445.70 (33.50)
609.70 (46.97)
605.00 (37.52)
870.80 (171.30)
journey escape
-2096.0
-1116.0
-780.0
-736.0
kangaroo
1740.0
4416.0
7088.0
9567.0
krull kung fu master
6780.10 10 (467.12)
8804.04 (97.77)
9132.15 (207
9409.73 (98.14) 14)
24102.80 (6513.61)
27867.00 .00 (5783.19) 19)
28905.80 (6570.13)
33312.00 (41 19.74)
montezuma revenge
0.0
0.0
0.0
0.0
ms pacman name this game
2276.30 (144.50)
5058.96 (602.27)
5871.52 (454.28)
6755.47 (555.18)
10702.2
9702.9
10094.5
9946.4
phoenix
4586.7
5145.4
5370.6
5505.8
pitfall
0.0
-3,95
-2.74
-21.34
pong
6.76
19.77
19.86
20.35
pooyan
4989.7
6334.05
6339.2
6776.7
private eye
99.40 (1.20)
144.64 (46.57)
173.02 (39.13)
164.31 (42.75)
qhert
4343.75
14809.5
168 12.5
18736.25
riverrai
3955.9
15068.6
15891.3
15655.7
road runner
32737.0
51383.0
54599.0
67962.0
robotank
30.66
53,55
57.18
62.68
seaquest
3077.86 (131.08)
21853,50 (4243.86)
29694.50 (6157.97)
46735.26 (10631.30)
skünn
2.7031.73
-20930.88 88
22103.779
-12295.78
solaris
2027.2
2770.2
2205.2
1495.4
space invaders
695.15
1748.45
3365.2
10110.4
star gunner
13345.0
52961.0
59574.0
72441.0
tennis
-3.19
-0.02
-0.07
-0.03
time pilot
6501.0
11598.0
13550.0
19050.0
tutankham
128.7
177.96
284.42
288.41
up n down venture video pinball wizard of wor yars revenge zaxxon
18544.78 (3272.37)
44569.10 (12243.70)
56722.56 (9966.49)
110907.70 (10256.62)
0.0
1046.0
1486.0
1679.0
40107.80 82
1784770.52
3008620.51
1254569.66 .69
4133.0
7441.0
7466.0
9369.0
11077.61 (1366.42)
72860.33 (7560.21)
84238.64 (7721.16)
93144.71 (5251.19)
8319.00 (557.20)
12494.80 (282.63)
14077.60 (917.33)
13913.40 (585.68)
Table 6: Raw scores for ALE game agents trained with Rainbow-IQN on SABER at 10M, 50M.
100M and 200M training frames. For the 14 games ran on 5 seeds, we also show the standard deviation.
19

---

Under review as a conference paper at ICLR 2020
Rainbow 30 minutes
Rainbow-IQN
Game name
minutes
SABER
5 minutes
30 minutes
SABER
raid
10549
12308.25
12308.25
11107.25
12289.75
12289.75
alien
3458.5
3458.5
3458.5
7046.4
7046.4
7046.4
amidar
2835.53
2952.43
2952.43
2601.82 822
3092.05
3092.05
assault
3779.98
3986.1
3986.1
5178.41
6372.7
6372.7
asterix
29269
29269
29269
28015.0
28015.0
28015
asteroids
1716.90 (238)
1716,90 (238)
1716.90 (238)
30838.86
159426.4
Infinite gameplay
atlantis
129392
858765
Infinite gameplay
130475.0
839433.0
Infinite gameplay
bank heist
1563.2
1563.2
1563.2
1412.4
1412.4
1412.4
battle
45610
45610
45610
44410.0
44410.0
44410
beam rider
5437.14
5542.22
5542.22
8165.14
9826.62
9826.62
berzerk
1049.3
1049.3
1049.3
888.0
892.9
892.9
bowling
29.92
29.92
29.92
29.92
29.92
29.92
boxing
98.7
98.7
98.7
98.7
98.7
98.7
breakout
173.01
173.01
173.01
175.39
175.47
175.47
carnival
4163.5
4163.5
4163.5
4566.3
4566.3
4566.3
centipede
7267.82 (265)
7267.82 (265)
7267.82 (265)
$260.96
5260.96
5260.96
chopper command
7973
7973
7973
11405.0
11405.0
11405
crazy climber
133756
144373
144373
137299.0
144437.0
144437
defender
18524.71
30976.24
Infinite gameplay
19004.03
24926.15
Infinite gameplay
demon attack
10234.20 (415)
14617.11 (2215)
14617.11 (2215)
10294.51
24596.37
24599.31
double dunk
0
0
0
1.1
1.3
1.3
elevator action
13421
85499
85499
12455.0
77010.0
77010
enduro
369.87
2332.63
6044.36
373.3
2316.67
5146.73
fishing derby
43.57
43.57
43.57
49.08
49.08
49.08
freeway
33.96
33.96
33,96
33,96
33.96
33,96
frostbite
7075.14 (656)
7075.14 (656)
7075.14 (656)
10002.78
10002.78
10002.78
gopher
12405
16736.4
16736.4
11724.8
15797.6
15797.6
gravitar
2647.50 50 (398)
2647.50 (398)
2647.50 (398)
1973.6
1973.6
1973.6
hero
28911.15
28911.15
28911.15
28957.4
28957.4
28957.4
ice hockey
-0.69
-0.69
-0.69
-0.07
-0.07
-0.07
jameshond
1421.00 (502)
1434.00 (509)
1434.00 (509)
870.8
870.8
870.8
journey escape
-645
-645
-645
-736.0
-736.0
-736
kangaroo
13242
13242
13242
9567.0
9567.0
9567
krull
4697.19 19 (273)
4697. 19 (273)
4697.19 (273)
9409.73
9409.73
9409.73
kung fu master
32265.20 (6476)
32692.80 (6790)
32692.80 (6790)
32934.8
33312.0
33312
montezuma revenge
O
O
O
0.0
0.0
0
ms pacman
4738.30 (265)
4738.30 (265)
4738.30 (265)
6755.47
6755.47
6755.47
name this game
8187.4
11787.7
11787.7
7579.8
9946.4
9946.4
phoenix
5943.9
5943.9
5943.9
$505.8
5505.8
5505.8
pitfall
0
0
0
-11.11
-21.44
-21.34
pong
20.35
20.35
20.35
20.35
20.35
20.35
pooyan
4766.3
4788.5
4788.5
6466.6
6776.7
6776.7
private eye
100.00 (0)
100.00 (0)
100.00 (0)
164.31
164.31
164.31
qhert
26116
26171.75
26171.75
18736.25
18736.25
18736.25
riverrai
18456
18456
18456
15655.7
15655.7
15655.7
road runner
66593
66593
66593
67962.0
67962.0
67962
robotank
52.34
62.99
62.99
51.35
62.68
62.68
seaquest
12281.22 (7018)
20670.40 (17377)
20670.40 (17377)
28554.0
46735.26
46735.26
sküng
-28105.83
-28134.23
-28134.23
112244.58
112295.78
12295.77
solaris
22
2779.4 44
2779.4
819.0
1495.4
1495.4
space invaders
2764.55
2764.55
2764.55
4718.2
10110.4
10110.4
star gunner
72944
73331
73331
71705.0
72441.0
72441
tennis
O
0
0
-0.03
-0.03
-0.03
time pilot
20198
20198
20198
19050.0
19050.0
19050
tutankham
177.17
177.42
177.42
288.41
288.41
288.41
up n down venture vidco pinball wizard of wor yars revenge zaxxon
52599 (4454)
105213 (23843)
105213 (23843)
56646.0
110655.76
110907.7
1781
1781
1781
1679.0
1679.0
1679
96345.36
656571.52
2197677.95
76587.14
465419.66
1254569.69
9913
9943
9943
9369.0
9369.0
9369
60913 (2342)
60913 (2342)
60913 (2342)
93144.71
93144.71
93144.71
19017 (1228)
19060 (1238)
19060 (1238)
13913.4
13913.4
13913.4
Table 7: Agent scores for Rainbow and Rainbow-IQN at 200M training frames, reported for Smin. 30min and SABER (no limit) evaluation time. Standard deviation are showed for Rainbow (for Rainbow-IQN it can be found on next tables). 20

---

Under review as a conference paper at ICLR 2020
Training frames
Game name
10M
50M
100M
200M
raid
7549.0
9168.75
10272.75
11107.25
alien
2740.6
1878.1
5223.0
7046.4
amidar
347.13
1554,84
2129.27
2601.82
assault
966.87
2783.49
4103.89
5178.41
asterix
3467.0
9280.0
16344.5
28015.0
asteroids
1194.16 16 (98.25)
3251.64 (2582.07)
12261.36 (12251.18)
30838.86 (5427.63)
atlantis
101945.0
118844.0
125696.0
130475.0
bank heist
756,4
1325.3
1402.2
1412.4
battle
33000.0
36730.0
33480.0
44410.0
beam rider
6764,82
8554.82
7818.72
8165. 14
berzerk
546.7
697.0
640.2
888.0
bowling
29.64
30.0
29.86
29.92
boxing
92.71
98.62
98.92
98.7
breakout
53.77
121.83
132.56
175.39
carnival
5148.7
4824.1
4851.3
4566.3
centipede
2241.70 (251.56)
4099.89 (405.19)
4720.54 (626.31)
5260.96 (920.10)
chopper command
3018.0
6523.0
9053.0
11405.0
crazy climber
86085.0
117582.0
130559.0
137299.0 .0
defender
36353.98
19608.36
18915.17
19004.03
demon attack
3383.66 (648.57)
5833.77 (1542.65)
7161.13 (1364.32)
10294.51 (1868.54)
double dunk
-5.24
0.3
1.52
1.1
elevator action
2.0
0.0
7360.0
12455.0
enduro
340.68
379.34
382.91
373.3
fishing derby
22.11
34.82
48.11
49.08
freeway
32.65
33,9
33.95
33,96
frostbite
4351.54 (1456.01)
9135. (1611.78)
9768.28 (1742.88)
10002.78 (1752.75)
gopher
4798.4 4
11561.0
10944.4
11724.8
gravitar
283.70 (56.37)
1258.90 (228.31)
1725. 90 (471.00)
1973.60 (614.80)
hero
13728.55
27450.65
28759.85
28957.4
ice hockey
-2,43
1.8
-0.72
-0.07
jamesbond
445.70 (33.50)
609.70 (46.97)
605.00 (37.52)
870.80 (171.30)
journey escape
-2096.0
-1116.0
-780.0
-736.0
kangaroo
1740.0
4416.0
7088.0
9567.0
krull kung fu master
6780.10 10 (467.12)
8804.04 04 (97.77)
9132.15 15 (207.84)
9409.73 (98.14)
23970.80 (6513.13)
27701 .20 (5814.09)
28708.8 80 (6580. 12)
32934.80 (41 70.04)
montezuma revenge
0.0
0.0
0.0
0.0
ms pacman name this game
2276.30 (144.50)
5058.96 (602.27)
5871.55 (454.28)
6755.47 (555.18)
8212.4
7790.3
7754.6
7579.8
phoenix
4586.7
$145.4
5370.6
$505.8
pitfall
0.0
-3.95
-2.58
-11.11
pong
6.29
19.77
19.86
20.35
pooyan
4956.6
6233.55
6183.95
6466.6
private eye
99.40 (1,20)
144.64 (46.57)
173.02 (39.13)
164,31 (42.75)
qhert
4343.75
14809.5
168 16812.5
18736.25
riverrai
3955.9
15068.6
15891.3
15655.7
road runner
32737.0
51383.0
54426.0
67962.0
robotank
25.0
42.14
45.56
51.35
seaquest
3077.86 (131.08)
18200.66 (2114.98)
21750.36 (1891.98)
28554.00 (3617.42)
sküng
-27012.53
-20923228
-21046.99
12294..8
solaris
1210.6
1552.4
1338.0
819.0
space invaders
695.15
1748.45
3347.25
4718.2
star gunner
13345.0
52961.0
59572.0
71705.0
tennis
-3.19
-0.02
-0.04
-0.03
time pilot
6501.0
11598.0
13550.0
19050.0
tutankham
128.7
177.71
284.42
288.41
up n down
14722.22 (2551.46)
35663.92 (7724.72)
42380.48 (4978.69)
56646.00 (2541.90)
venture vidco pinball wizard of wor yars revenge zaxxon
0.0
1046.0
1486.0
1679.0
29524.06
122029.55 258
79508.52
76587. .14
4133.0
7441.0
7466.0
9369.0
11077.61 (1366.42)
72860.33 (7560.21)
84238.64 (7721.16)
93144.71 (5251.19)
8319.00 (557.20)
12494.80 (282.63)
14073.20 (910.64)
13913.40 (585.68)
Table 8: Raw scores for ALE game agents trained for Rainbow- at 10M, 50M, 100M and 200M training frames for 5 minutes evaluation. For the 14 games ran on 5 seeds, we also show the standard deviation.
21

---

Under review as a conference paper at ICLR 2020
Training frames
Game name
10M
50M
100M
200M
raid
7765.25
11690.0
13434.25
12289.75
alien
2740.6
1878.1
5223.0
7046.4
amidar
347.13
1554,84
2129.27
3092.05
assault
966.87
2783.49
4443.03
6372.7
asterix
3467.0
9280.0
16344.5
28015.0
asteroids
1194.16 16 (98.25)
3261.88 (2602.48)
48027.06 (56599.58)
159426.40 (56987.42)
atlantis
261697.0
788006.0
817118.0
839433.0
bank heist
756.4
1325.3
1402.2
1412.4
battle
33000.0
36730.0
33480.0
44410.0
beam rider
11510.78
11900.7
10042.74
9826.62
berzerk
546.7
697.0
640.2
892.9
bowling
29.64
30.0
29.86
29.92
boxing
92.71
98.62
98.92
98.7
breakout
53.77
121.83
132.56
175.47
carnival
5148.7
4824.1
4851.3
4566.3
centipede
2241.70 (251.56)
4099.89 (405.19)
4720.54 (626.31)
5260.96 (920.10)
chopper command
3018.0
6523.0
9053.0
11405.0
crazy climber
86310.0
118038.0
133114.0
144437.0
defender
49409.81
35899.7
24663.27
24926.15
demon attack
3433.13 (656.97)
6616.96 (2949.90)
8267.82 (3065.27)
24596.37 (17442.46)
double dunk
-5.54
0.3
1.52
1.3
elevator action
2.0
0.0
43490.0
77010.0
enduro
1378.3
2242.11
2307.42
2316.67
fishing derby
22.11
34.82
48.11
49.08
freeway
32.65
33.9
33,95
33,96
frostbite
4351.54 (1456.01)
9135. (161 1.78)
9768.28 (1742.88)
10002. 78 (1752.75)
gopher
4798.4 4
15629.8
14136.0
15797.6
gravitar
283.70 (56.37)
1258.90 (228.31)
1725. 90 (471.00)
1973.60 (614.80)
hero
13728.55
27450.65
28759.85
28957.4
ice hockey
-2,43
1.8
-0,72
-0.07
jamesbond
445.70 (33.50)
609.70 (46.97)
605.00 (37.52)
870.80 (171.30)
journey escape
-2096.0
-1116.0
-780.0
-736.0
kangaroo
1740.0
4416.0
7088.0
9567.0
krull kung fu master
6780.10 10 (467.12)
8804.04 (97.77)
9132.15 (207.84)
9409.73 73 (98, 14)
24102.80 (6513.61)
27867.00 .00 (5783.19) 19)
28905.80 (6570.13)
33312.00 (4119.74)
montezuma revenge
0.0
0.0
0.0
0.0
ms pacman name this game
2276.30 (144.50)
5058.96 (602.27)
5871.52 (454.28)
6755.47 (555.18)
10702.2
9702.9
10094.5
9946.4
phoenix
4586.7
5145.4
5370.6
5505.8
pitfall
0.0
-3,95
-2.74
-21.34
pong
6.76
19.77
19,86
20.35
pooyan
4989.7
6334.05
6339,2
6776.7
private eye
99.40 (1.20)
144.64 (46.57)
173.02 (39.13) 13)
164.31 (42.75)
qhert
4343.75
14809.5
16.5
18736.25
riverrai
3955.9
15068.6
15891.3
15655.7
road runner
32737.0
51383.0
54599.0
67962.0
robotank
30.66
53,55
57.18
62.68
seaquest
3077.86 (131.08)
21853,50 (4243.86)
29694,50 (6157.97)
46735.26 (10631.30)
sküng
2.7031.73
-20930.88 88
221053.79
-12295.78
solaris
2027.2
2770.2
2205.2
1495.4
space invaders
695.15
1748.45
3365.2
101
star gunner
13345.0
52961.0
59574.0
72441.0
tennis
-3.19
-0.02
-0.07
-0.03
time pilot
6501.0
11598.0
13550.0
19050.0
tutankham
128.7
177.96
284.42
288.41
up n down
18516.40 (3286.13)
44569.10 (12243.70)
56722.56 (9966.49)
110655. (10325.07)
venture vidco pinball wizard of wor yars revenge zaxxon
0.0
1046.0
1486.0
1679.0
40107 82
798642.24
565903.1 18
465419.66
4133.0
7441.0
7466.0
9369.0
11077.61 (1366.42)
72860.33 (7560.21)
84238.64 (7721.16)
93144.71 (5251.19)
8319.00 (557.20)
12494.80 (282.63)
14077.60 (917.33)
13913.40 (585.68)
Table 9: Raw scores for ALE game agents trained for Rainbow-IQN at 10M, 50M, 100M and 200M training frames for 30 minutes evaluation. For the games ran on 5 seeds, we also show the standard deviation.
22


Reviews:
[{"review_id": "rylUiKKiYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a new way to benchmark DRL algorithms using the Atari environment which is twofold, one part is a set of emulator recommendations, the other part is what quantity we should consider as a \"human reference\". The paper also compares Rainbow and Rainbow-IQN, where the IQN improvement matches the proposed human normalized score improvment.\n\nI'm not quite sure how to rate this paper, I have put weak-reject for now, as I don't strongly disagree with anything in the paper, but at the same time:\n- the difference to Machados et al. is marginal, but is a bit surprising\n- the Rainbow-IQN improvement is too incremental to be considered a significant contribution\n- there are some interesting remarks on why Atari is _not_ necessarily a good environment, e.g. most of Section 6, but this clashes with the paper's premise that we should be using Atari.\n\nIn a way, this paper reads like an interesting technical review of Atari, but I don't think it provides enough new knowledge to be a conference paper.\n\nDetailed comments:\n- I find it a bit weird that the many weaknesses of Atari as a platform are presented at DRL being bad at Atari. The line between environment design and algorithm design can be blurry, but in Atari's case, the weird peculiarities of each game are known to make it an inconvenient benchmark.\n- In the same vein, why is Atari+SABER better than other RL environments? This is rather crucial. We should only work on improving a benchmark if it is a useful benchmark, yet, we have many clues that Atari is not.\n- The link to TwinGalaxies should be a proper reference with the time of visit, especially if humans break new records in the future.\n- Why only compare Rainbow and a variant of Rainbow? I understand compute resources being a limitation, but at the same time, the reasoning behind having standardized testing is to be able to compare a wide variety of algorithms. This paper would be much stronger if it focused on a few representative games (e.g. one reflex game, one hard exploration game, etc.) and tested these games with a bunch of DRL algorithms. That ranking might reveal something very interesting.\n- The paper is easy to read, but there are a few grammar mistakes here and there.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1gj4nbAKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper revisits the way RL algorithms are typically evaluated on the ALE benchmark, advocating for several key changes that contribute to more robust and reliable comparisons between algorithms. It also brings the following additional contributions: (1) a new measure of comparison to human performance based on actual human world records (which shows that RL algorithms are not as « super-human » as is generally believed), and (2) an evaluation (based on the proposed guidelines) of Rainbow as well as a Rainbow-IQN variant (replacing the C51 component of Rainbow with Implicit Quantile Networks), showing that the latter brings a significant improvement upon the original Rainbow algorithm.\n\nOverall I am leaning towards acceptance as I believe that such papers encouraging better benchmarking practice on Atari are definitely needed. Even if the technical contribution is limited, this paper could have a positive impact on the field by providing a clearer picture of the current state of deep RL algorithms on Atari (assuming that other researchers start following these recommendations -- and if that is not the case at least it will highlight issues with the way evaluation is currently done).\n\nI do have a few concerns / questions though:\n\n1.\tI am not convinced by the recommendation to use performance during training for evaluation purpose. In Machado et al. (2018) it is argued that « this better aligns the performance metric with the goal of continual learning », but most deep RL algorithms trained on Atari games have not been intended to be used in a continual learning setting. It definitely has the advantage of being simple, but it seems to me that it can cause some issues, like making it difficult to compare different exploration techniques for off-policy learning (the exploration may cause poor behavior during training even if it helps the agent learn a better greedy policy), and more generally not being representative of the common practical use case where the goal is to obtain the best agent possible to use in production (with no further learning). Finally, it could make results even harder to reproduce due to the potential high variance of an agent’s performance at a fixed # of timesteps (vs. considering the max performance it can reach over the whole period). As a result, I am currently reluctant to see the proposed performance measure become the standard evaluation metric on ALE, and I would appreciate some additional justification from the authors on this point.\n\n2.\tWhy not suggest to remove reward clipping in the recommendations? As mentioned in Section 6, reward clipping can prevent RL algorithms from properly playing some games, and thus in my opinion should be removed if the goal is to reach the highest score possible on all games. It seems to me that the choice of clipping the reward should be part of the algorithm (if it is not able to handle the high variety of « raw » rewards) and not of the benchmark environment, thus enabling further advancements towards algorithms that are robust to a wide range of rewards.\n\n3.\tWhy bother to keep the mean performance when, as mentioned, it is highly sensitive to outliers compared to the median?\n\n\nAdditional remarks:\n•\tI might have missed it but I do not see the link to the source code. Am I correct to assume it will be released, to help with reproducibility?\n•\tIt is not clear, when reading the paper, that the distributed version of Rainbow is actually constrained to mimic a single agent sequential algorithm in the experiments. I would suggest to remove mentions of the distributed version in the main text to avoid confusion, and mention it only in the Appendix section where it is used.\n•\tThe « infinite reward loop » point at the end of Section 6 does not seem relevant in the list of reasons why Deep RL algorithms are far from the best human performance, since with infinite playtime and an infinite reward loop, the algorithm should be guaranteed to outperform humans.\n•\tI would have appreciated an evaluation of Rainbow-IQN with the current most commonly used evaluation schemes (e.g. the one used in the original Rainbow paper), for comparison purpose (even if such an evaluation has flaws, it is often the only performance measure available for existing deep RL algorithms)\n\nReview update: thank you for the response, I am currently keeping my \"Weak accept\" rating because I agree it is important to highlight and (try to) fix the problems with the way algorithms are currently evaluated on ALE, in spite of the limited technical contributions (and the fact that I remained unconvinced regarding #1)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJxg5_Satr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an extension to the work of Machado et al. (2018) for standardizing training and evaluation procedures in the Arcade Learning Environment (ALE). It then introduces a collection of human world records for each Atari game to refute previous claims of superhuman performance, as well as recommend comparisons against these records. They proceed to evaluate Rainbow under their proposed evaluation procedures, as well as introduce a new algorithm, Rainbow-IQN, with similar evaluations made based on their proposal.\n\nI'm proposing a weak rejection as I feel some of the arguments made in the paper aren't very strong. In particular, I'd like the authors to comment on the following:\n\n1) The key difference between their evaluation benchmark and the recommendations in Machado et al. (2018) are that episodes should not have a time limit. The justification for this is that many algorithms might achieve practically optimal performance within this time limit, and so one wouldn't be able to compare algorithms on certain games within significance. They further emphasize that human high scores were achieved without limiting to 30 minutes of play. That said, several algorithms performing similarly within said limit can be instead interpreted as shifting emphasis toward comparing performance on the harder games. As the paper acknowledged, removing the maximum episode length ended up introducing more issues, such as the emulator never ending an episode (due to a supposed bug), as well as increasing the likelihood of the score overflowing. The paper suggested a trick of limiting how long an agent can go without receiving a reward, but it's unclear (1) if needing this fix is worth the proposed change, and (2) if the fix introduces additional game-specific nuances in evaluation; e.g., are there any situations where this can be detrimental to properly evaluating performance, or introduce biases based on a game's reward distribution?\n\n2) The paper gathered a list of human world records for the Atari games in the ALE. In my opinion, this is very valuable for the literature in terms of addressing prior work misrepresenting the competency of an algorithm relative to what humans are capable of; a professional game tester is supposed to be representative of the average game player, who is typically tasked with optimizing fun, whereas speedrunners and scorerunners of games are tasked with optimizing a comparable objective to an RL agent. Beyond this though, I think an alternative conclusion would be to use this information in support of not comparing results to human scores, and to focus on comparisons between algorithms. A considerable number of the human world records have reached the maximum allowable score, over drastically variable gameplay times to achieve these scores, that it might still not be that fair a comparison. Have the authors considered this possibility?\n\n3) Were any other algorithms evaluated on this benchmark, beyond Rainbow? While there are computational considerations, it seems odd for a benchmarking-focused paper to only evaluate one standard algorithm and slight modification of it.\n\nSuggestions\n\n1) The introduction of Rainbow-IQN in this paper feels a little random and out of place given the context created by the rest of the paper's contributions- I feel it might be more appropriate for a benchmarking paper to focus on a representative set of \"standard\" or relatively simple/trivial algorithms (Like Machado et al. (2018) did) to give a frame of reference for comparing novel ones.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new benchmark that compares performance of deep reinforcement learning algorithms on the Atari Learning Environment to the best human players.  The paper identifies limitations of past evaluations of deep RL agents on Atari. The human baseline scores commonly used in deep RL are not the highest known human scores.  To enable learning agents to reach these high scores, the paper recommends allowing the learning agents to play without a time limit.  The time limit in Atari is not always consistent across papers, and removing the time limit requires additional software fixes due to some bugs in the game software.  These ideas form the core of the paper's proposed new benchmark (SABER). The paper also proposes a new deep RL algorithm that combines earlier ideas. 

The reviews and the discussion with the authors brought out several strengths and weaknesses of the proposal.  One strength was identifying the best known human performance in these Atari games.  
However, the reviewers were not convinced that this new benchmark is useful.  The reviewers raised concerns about using clipped rewards, using games that received substantially different amounts of human effort, comparing learning algorithms to human baselines instead of other learning algorithms, and also the continued use of the Atari environment. Given all these many concerns about a new benchmark, the newly proposed algorithm was not viewed as a distraction.

This paper is not ready for publication. The new benchmark proposed for deep reinforcement learning on Atari was not convincing to the reviewers.  The paper requires further refinement of the benchmark or further justification for the new benchmark.
Decision: Reject