Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field

1 INTRODUCTION . Human intelligence is able to solve many tasks of different natures . In pursuit of generality in artificial intelligence , video games have become an important testing ground : they require a wide set of skills such as perception , exploration and control . Reinforcement Learning ( RL ) is at the forefront of this development , especially when combined with deep neural networks in DRL . One of the first general approaches reaching reasonable performance on many Atari games while using the exact same hyper-parameters and neural network architecture was Deep Q-Network ( DQN ) ( Mnih et al. , 2015 ) , a value based DRL algorithm which directly takes the raw image as input . This success sparked a lot of research aiming to create better , faster and more stable general algorithms . The ALE ( Bellemare et al. , 2013 ) , featuring more than 60 Atari games ( see Figure 1 ) , is heavily used in this context . It provides many different tasks ranging from simple paddle control in the ball game Pong to complex labyrinth exploration in Montezuma ’ s Revenge which remains unsolved by general algorithms up to today . As the number of contributions is growing fast , it becomes harder and harder to make a proper comparison between different algorithms . In particular , a relevant difference in the training and evaluation procedures exists between available publications . Those issues are exacerbated by the fact that training DRL agents is very time consuming , resulting in a high barrier for reevaluation of previous work . Specifically , even though ALE is fast at runtime , training an agent on one game takes approximately one week on one GPU and thus the equivalent of more than one year to train on all 61 Atari games . A standardization of the evaluation procedure is needed to make DRL that matters as pointed out by Henderson et al . ( 2018 ) for the Mujoco benchmark ( Todorov et al. , 2012 ) : the authors criticize the lack of reproducibility and discuss how to allow for a fair comparison in DRL that is consistent between articles . In this work , we first discuss current issues in the evaluation procedure of different DRL algorithms on ALE and their impact . We then propose an improved evaluation procedure , extending the recommendations of Machado et al . ( 2018 ) , named SABER : a Standardized Atari BEnchmark for 1 Reinforcement learning . We suggest benchmarking on the world records human baseline and show that RL algorithms are in fact far from solving most of the Atari games . As an illustration of SABER , current state-of-the-art DRL algorithm Rainbow ( Hessel et al. , 2018 ) is benchmarked . Finally , we introduce and benchmark on SABER a new state-of-the-art agent : a distributable combination of Rainbow and Implicit Quantiles Network ( IQN ) ( Dabney et al. , 2018 ) . The main contributions of this work are : • The proposal , description and justification of the SABER benchmark . • Introduction of a world records human baseline . We argue it is more representative of the human level than the one used in most of previous works . With this metric , we show that the Atari benchmark is in fact a hard task for current general algorithm . • A SABER compliant evaluation of current state-of-the art agent Rainbow . • A new state-of-the-art agent on Atari , Rainbow-IQN , with a comparison on SABER to Rainbow , to give an improvement range for future comparisons . • For reproducibility sake , an open-source implementation of Rainbow , Rainbow-IQN , distributed following the idea from Horgan et al . ( 2018 ) . 1.1 RELATED WORK . Reproducibility and comparison in DRL Deep Reinforcement Learning that matters ( Henderson et al. , 2018 ) is one of the first works to warn about a reproducibility crisis in the field of DRL . This article relies on the MuJoCo ( Todorov et al. , 2012 ) benchmark to illustrate how some common practices can bias reported results . As a continuation to the work of Henderson et al . ( 2018 ) , J. Pineau introduced a Machine Learning reproducibility checklist ( Pineau , 2019 ) to allow for reproducibility and fair comparison . Machado et al . ( 2018 ) deal with the Atari benchmark . They describe the divergence in training and evaluation procedures and how this could lead to difficulties to compare different algorithms . A first set of recommendations to standardize them is introduced , constituting the basis of this work and will be summarized in the next section . Finally , the Github Dopamine ( Castro et al. , 2018 ) provides an open-source implementation of some of the current state-of-the-art algorithms on Atari benchmark , including Rainbow and IQN . An evaluation following almost all guidelines from Machado et al . ( 2018 ) are provided in Castro et al . ( 2018 ) . However the implementation of Rainbow is partial , and the recommendation of using the full action set is not applied . This is why our work contains a new evaluation of Rainbow . Value based RL DQN ( Mnih et al. , 2015 ) is the first value based DRL algorithm benchmarked on all Atari games with the exact same set of hyperparameters ( although previous work by Hausknecht et al . ( 2014 ) already performed such a benchmark with neural networks ) . This algorithm relies on the well known Q-Learning algorithm ( Watkins & Dayan , 1992 ) and incorporates a neural network . Deep Q-learning is quite unstable and the main success of this work is to introduce practical tricks to make it converge . Mainly , transitions are stored in a replay memory and sampled to avoid correlation in training batch , and a separate target network is used to avoid oscillations . Since then , DQN has been improved and extended to make it more robust , faster and better . Rainbow ( Hessel et al. , 2018 ) is the combination of 6 of these improvements ( Van Hasselt et al. , 2016 ; Schaul et al. , 2015 ; Bellemare et al. , 2017 ; Wang et al. , 2015 ; Fortunato et al. , 2017 ; Mnih et al. , 2016 ) implemented in a single algorithm . Some ablations studies showed that the most important components were Prioritized Experience Replay ( PER ) ( Schaul et al. , 2015 ) and C51 ( Bellemare et al. , 2017 ) . The idea behind PER is to sample transitions according to their surprise , i.e . the worse the network is at predicting the Q-value of a specific transition , the more we sample it . C51 is the first algorithm in Distributional RL which predicts the full distribution of the Q-function instead of predicting only the mean of it . Finally , IQN ( Dabney et al. , 2018 ) is an improvement over C51 . It almost reaches on its own the performance of the full Rainbow with all 6 components . In C51 the distribution of the Q-function is represented as a categorical distribution while in IQN , it is represented by implicit quantiles . 2 2 CHALLENGES WHEN COMPARING PERFORMANCE ON THE ATARI BENCHMARK . In this section we discuss several challenges to make a proper comparison between different algorithms trained on the Atari benchmark . First , we briefly summarize the initial problems and their solution as proposed by Machado et al . ( 2018 ) . Then we detail a remaining issue not handled by those initial standards , the maximum length time allowed for an episode . Finally , we introduce a readable metric , representative of actual human level and allowing meaningful comparison . 2.1 REVISITING ALE : AN INITIAL STEP TOWARDS STANDARDIZATION . Machado et al . ( 2018 ) discuss about divergence of training and evaluation procedures on Atari . They show how those divergences are making comparison extremely difficult . They establish recommendations that should be used in order to standardize the evaluation process . Stochasticity The ALE environment is fully deterministic , i.e . leading to the exact same state if the exact same actions are taken at each state . This is actually an issue for general algorithm evaluation . For example , an algorithm learning by heart good trajectories can actually reach a high score with an open-loop behaviour . To handle this issue , Machado et al . ( 2018 ) introduce sticky actions : actions coming from the agent are repeated with a given probability ξ , leading to a non deterministic behavior . They show that sticky actions are drastically affecting performance of an algorithm exploiting the environment determinism without hurting algorithms learning more robust policies like DQN ( Mnih et al. , 2015 ) . We use sticky actions with probability ξ = 0.25 ( Machado et al. , 2018 ) in all our experiments . End of an episode : Use actual game over In most of the Atari games the player has multiple lives and the game is actually over when all lives are lost . But some articles , e.g . DQN , Rainbow , IQN , end a training episode after the loss of the first life but still use the standard game over signal while testing . This can in fact help the agent to learn how to avoid death and is an unfair comparison to agents which are not using this game-specific knowledge . Machado et al . ( 2018 ) recommend to use only the standard game over signal for all games while training . Action set Following the recommendation of Machado et al . ( 2018 ) we do not use the minimal useful action set ( the set of actions having an effective impact on the current game ) as used by many previous works ( Mnih et al. , 2015 ; Hessel et al. , 2018 ) . Instead we always use all 18 possible actions on the Atari Console . This removes some specific domain knowledge and reduces the complexity of reproducibility . For some games , the minimal useful action set is different from one version to another of the standard Atari library : an issue to reproduce result on breakout was coming from this ( Graetz , 2018 ) . Reporting performance As proposed by Machado et al . ( 2018 ) , we report our score while training by averaging k consecutive episodes ( we have set k = 100 ) . This gives information about the stability of the training and removes the statistical bias induced when reporting score of the best policy which is today a common practice ( Mnih et al. , 2015 ; Hessel et al. , 2018 ) . 2.2 MAXIMUM EPISODE LENGTH . A major parameter is left out of the work of Machado et al . ( 2018 ) : the maximum number of frames allowed per episode . This parameter ends the episode after a fixed number of time steps even if the game is not over . In most of recent works ( Hessel et al. , 2018 ; Dabney et al. , 2018 ) , this is set to 30 min of game play and only to 5 min in Revisiting ALE ( Machado et al. , 2018 ) . This means that the reported scores can not be compared fairly . For example , in easy games ( e.g . Atlantis , Enduro ) , the agent never dies and the score is more or less linear to the allowed time : the reported score will be 6 times higher if capped at 30 minutes instead of 5 minutes . We argue that the time cap can make the performance comparison non significant . On many games ( e.g . Atlantis , Video Pinball ) the scores reported of Ape-X ( Horgan et al. , 2018 ) , Rainbow ( Hessel et al. , 2018 ) and IQN ( Dabney et al. , 2018 ) are almost exactly the same . This is because all agents 3 reach the time limit and get the highest possible score in 30 minutes : the difference in scores is due to minor variations , not algorithmic difference . As a consequence , the more successful agents are , the more games are incomparable because they reach the maximum possible score in the time cap . This parameter can also be a source of ambiguity and error . The best score on Atlantis ( 2,311,815 ) is reported by Proximal Policy Optimization by Schulman et al . ( 2017 ) but this score is almost certainly wrong : it seems impossible to reach it in only 30 minutes ! The first distributional paper , C51 ( Bellemare et al. , 2017 ) , also did this mistake and reported wrong results before adding an erratum in a later version on ArXiv . We argue that episodes should not be capped at all . The original ALE article ( Bellemare et al. , 2013 , pg.3 ) states that This functionality is needed for a small number of games to ensure that they always terminate . On some famously hard games like Pitfall and Tennis , random exploration leads to much more negative reward than positive and thus the agent effectively learns to do nothing , e.g . not serving in Tennis . We claim that , even with this constraint , agents still end up learning to do nothing , and the drawback of the cap harms the evaluation of all other games . Moreover , the human high scores for Atari games have been achieved in several hours of play , and would have been unreachable if limited to 30 minutes . To summarize , ideally one would not cap at all length of episode while training and testing . However this makes some limitations of the ALE environment appear , as described in the following paragraph . Glitch and bug in the ALE environment When setting the maximum length of an episode to infinite time , the agent gets stuck on some games , i.e . the episode never ends , because of a bug in the emulator . In this case , even doing random actions for more than 20 hours neither gives any reward nor end the game . This happens consistently on BattleZone and less frequently on Yar ’ s Revenge . One unmanaged occurrence of this problem is enough to hamper the whole training of the agent . It is important to note that those bugs were discovered by chance and it is probable that this could happen on some other games . We recommend to set the maximum episode length to infinite ( in practice , a limit of 100 hours was used ) . Additionally we suggest a maximum stuck time of 5 minutes . Instead of limiting the allowed time for the agent , we limit the time without receiving any reward . This small trick handles all issues exposed above , and sets all reported scores on the same basis , making comparison to world records possible . Other bugs or particularities harming evaluation were encountered while training on the full Atari benchmark : buffer rollover with sudden negative score , influence of a start key for some games , etc . They are detailed and discussed in the supplementary material and we argue that they can have a drastic impact on performance and explain inconsistencies .

Summary: The paper proposes an extension to the work of Machado et al. (2018) for standardizing training and evaluation procedures in the Arcade Learning Environment (ALE). It then introduces a collection of human world records for each Atari game to refute previous claims of superhuman performance, as well as recommend comparisons against these records. They proceed to evaluate Rainbow under their proposed evaluation procedures, as well as introduce a new algorithm, Rainbow-IQN, with similar evaluations made based on their proposal.