Title: ON BONUS-BASED EXPLORATION METHODS IN THE ARCADE LEARNING ENVIRONMENT

Abstract: Research on exploration in reinforcement learning, as applied to Atari 2600 gameplaying, has emphasized tackling difficult exploration problems such as MON-TEZUMA'S REVENGE (Bellemare et al., 2016). Recently, bonus-based exploration methods, which explore by augmenting the environment reward, have reached above-human average performance on such domains. In this paper we reassess popular bonus-based exploration methods within a common evaluation framework. We combine Rainbow (Hessel et al., 2018) with different exploration bonuses and evaluate its performance on MONTEZUMA'S REVENGE, Bellemare et al.'s set of hard of exploration games with sparse rewards, and the whole Atari 2600 suite. We find that while exploration bonuses lead to higher score on MONTEZUMA'S REVENGE they do not provide meaningful gains over the simpler -greedy scheme. In fact, we find that methods that perform best on that game often underperform -greedy on easy exploration Atari 2600 games. We find that our conclusions remain valid even when hyperparameters are tuned for these easy-exploration games. Finally, we find that none of the methods surveyed benefit from additional training samples (1 billion frames, versus Rainbow's 200 million) on Bellemare et al.'s hard exploration games. Our results suggest that recent gains in MONTEZUMA'S REVENGE may be better attributed to architecture change, rather than better exploration schemes; and that the real pace of progress in exploration research for Atari 2600 games may have been obfuscated by good results on a single domain.

1. INTRODUCTION:

In reinforcement learning, the exploration-exploitation trade-off describes an agent's need to balance maximizing its cumulative rewards and improving its knowledge of the environment. While many practitioners still rely on simple exploration strategies such as the -greedy scheme, in recent years a rich body of work has emerged for efficient exploration in deep reinforcement learning. One of the most successful approaches to exploration in deep reinforcement learning is to provide an exploration bonus based on the relative novelty of the state. This bonus may be computed, for example, from approximate counts (Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017; Machado et al., 2018a) , from the prediction error of a dynamics model (ICM, Pathak et al., 2017) or by measuring the discrepancy to a random network (RND, Burda et al., 2019) . Bellemare et al. (2016) argued for the importance of the Atari 2600 game MONTEZUMA'S REVENGE as a challenging domain for exploration. MONTEZUMA'S REVENGE offers a sparse reinforcement signal and relatively open domain that distinguish it from other Atari 2600 games supported by the Arcade Learning Environment (ALE; Bellemare et al., 2013) . As a consequence, recent exploration research (bonus-based or not) has aspired to improve performance on this particular game, ranging from a much deeper exploration (6600 points; Bellemare et al., 2016) , to completing the first level (14000 points; Pohlen et al., 2018; Burda et al., 2019) , to super-human performance (400,000 points; Ecoffet et al., 2019 ).

Yet the literature on exploration in reinforcement learning still lacks a systematic comparison between existing methods, despite recent entreaties for better practices to yield reproducible research (Henderson et al., 2018; Machado et al., 2018b) . One of the original tenets of the ALE is that agent evaluation should take on the entire suite of 60 available Atari 2600 games. Even within the context of bonus-based exploration, MONTEZUMA'S REVENGE is but one of seven hard exploration, sparse rewards games (Bellemare et al., 2016) . Comparisons have been made between agents trained under different regimes: using different learning algorithms and varying numbers of training frames, with or without reset, and with and without controller noise (e.g. sticky actions Machado et al., 2018b) . As a result, it is often unclear if the claimed performance improvements are due to the exploration method or other architectural choices, and whether these improvements carry over to other domains. The main conclusion of our research is that an over-emphasis on one Atari 2600 game, combined with different training conditions, may have obfuscated the real pace of progress in exploration research.

To come to this conclusion, we revisit popular bonus-based exploration methods (CTS-counts, PixelCNN-counts, RND, and ICM) in the context of a common evaluation framework. We apply the Rainbow (Hessel et al., 2018) agent in turn to MONTEZUMA'S REVENGE, Bellemare et al.'s seven hardest Atari 2600 games for exploration, and the full suite from the ALE.

Source of exploration bonus. We find that all agents perform better than an -greedy baseline, confirming that exploration bonuses do provide meaningful gains. However, we also find that more recent exploration bonuses do not, by themselves, lead to higher performance than the older pseudocounts method. Across the remainder of hard exploration games, we find that all bonuses lead to performance that is comparable, and in one case worse, than -greedy.

Performance across full Atari 2600 suite. One may expect a good exploration algorithm to handle the exploration/exploitation trade-off efficiently: exploring in difficult games, without losing too much in games where exploration is unnecessary. We find that performance on MONTEZUMA'S REVENGE is in fact anticorrelated with performance across the larger suite. Of the methods surveyed, the only one to demonstrate better performance across the ALE is the non-bonus-based Noisy Networks (Fortunato et al., 2018) , which provide as an additional point of comparison. Noisy Networks perform worse on MONTEZUMA'S REVENGE.

Hyperparameter tuning procedure. The standard practice in exploration research has been to tune hyperparameters on MONTEZUMA'S REVENGE then evaluate on other Atari 2600 games. By virtue of this game's particular characteristics, this approach to hyperparameter tuning may unnecessarily increase the behaviour of exploration strategies towards a more exploratory behavior, and explain our observation that these strategies perform poorly in easier games. However, we find that tuning hyperparameters on the original ALE training set does not improve performance across Atari 2600 games beyond that of -greedy .

Amount of training data. By design, exploration methods should be sample-efficient. However, some of the most impressive gains in exploration on MONTEZUMA'S REVENGE have made use of significantly more data (1.97 billion Atari 2600 frames for RND, versus 100 million frames for the pseudo-count method). We find that, within our common evaluation framework, additional data does not play an important role on exploration performance. Altogether, our results suggests that more research is needed to make bonus-based exploration robust and reliable, and serve as a reminder of the pitfalls of developing and evaluating methods primarily on a single domain.

1.1. RELATED WORK:

Closest to our work, Burda et al. (2018) benchmark various exploration bonuses based on prediction error (Schmidhuber, 1991; Pathak et al., 2017) within a set of simulated environment including many Atari 2600 games. Their study differs from ours as their setting ignores the environment reward and instead learns exclusively from the intrinsic reward signal. Outside of the ALE, Osband et al. (2019) recently provide a collection of experiments that investigate core capabilities of exploration methods.

2. EXPLORATION METHODS:

We focus on bonus-based methods, that is, methods that promote exploration through a reward signal. An agent is trained with the reward r = r ext + β • r int where r ext is the extrinsic reward provided by the environment, r int the intrinsic reward computed by agent, and β > 0 is a scaling parameter. We now summarize the methods we evaluate to compute the intrinsic reward r int .

2.1. PSEUDO-COUNTS:

Pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017) were proposed as way to estimate counts in high dimension states spaces using a density model. The agent is then encouraged to visit states with a low visit count. Let ρ be a density model over the state space S, we write ρ t (s) the density assigned to a state s after training on a sequence of states s 1 , ..., s t . We write ρ t (s) the density assigned to s if ρ were to be trained on s an additional time. The pseudo-count N is then defined such that updating ρ on a state s leads to a one unit increase of its pseudo-count

EQUATION

where n, the pseudo-count total is a normalization constant. This formulation of pseudo-counts match empirical counts when the density model corresponds to the empirical distribution. Equation 1 can be rewritten as

EQUATION

)

The intrinsic reward is then given by r int (s t ) := ( Nt (s t )) -1/2 .

(3) CTS (Bellemare et al., 2014) and PixelCNN (Van den Oord et al., 2016) have been both used as density models. We will disambiguate these agent by the name of their density model.

2.2. INTRINSIC CURIOSITY MODULE:

Intrinsic Curiosity Module (ICM, Pathak et al., 2017) promotes exploration via curiosity. Pathak et al. formulates curiosity as the error in the agent's ability to predict the consequence of its own actions in a learned feature space. ICM includes three submodules: a learned embedding, a forward and an inverse model. At the each timestep the module receives a transition (s t , a t , s t+1 ) -where s t and a t are the current state and action and s t+1 is the next state. States s t and s t+1 are encoded into the features φ(s t ) and φ(s t+1 ) then passed to the inverse model which is trained to predict a t . The embedding is updated at the same time, encouraging it to only model environment features that are influenced by the agent's action. The forward model has to predict the next state embedding φ(s t+1 ) using φ(s t ) and a t . The intrinsic reward is then given by the error of the forward model in the embedding space between φ(s t+1 ), and the predicted estimate φ(s t+1 ):

r int (s t ) = φ(s t+1 ) -φ(s t+1 ) 2 2 .

(4)

2.3. RANDOM NETWORK DISTILLATION:

Random Network Distillation (RND, Burda et al., 2019) derives a bonus from the prediction error of a random network. The intuition being that the prediction error will be low on states that are similar to those previously visited and high on newly visited states. A neural network f with parameters θ is trained to predict the output of a fixed, randomly initialized neural network f . The intrinsic reward is given by

r int (s t ) = f (s t ; θ) -f (s t ) 2 2 (5)

where θ represents the parameters of the network f .

2.4. NOISY NETWORKS:

We also evaluate Noisy Networks (NoisyNets; Fortunato et al., 2018) , which is part of the original Rainbow implementation. NoisyNets does not explore using a bonus. Instead NoisyNets add noise in parameter space and replace the standard fully-connected layers, y = W s + b, by a noisy version that combines a deterministic and a noisy stream:

EQUATION

where W and b are random variables and denotes elementwise multiplication.

3. EMPIRICAL RESULTS:

In this section we present our experimental study of bonus-based exploration methods.

3.1. EXPERIMENTAL PROTOCOL:

We evaluate the three bonus-based methods introduced in Section 2 as well as NoisyNets andgreedy exploration. To do so, we keep our training protocol fixed throughout our experiments. All methods are trained with a common agent architecture, the Rainbow implementation provided by the Dopamine framework (Castro et al., 2018) . It includes Rainbow's three most important component: n-step updates (Mnih et al., 2016) , prioritized experience replay (Schaul et al., 2015) and distributional reinforcement learning (Bellemare et al., 2017) . Rainbow was designed combining several improvements to value-based agents that were developed independently. Pairing Rainbow with recent exploration bonuses should lead to further benefits. We also keep the original hyperparameters of the learning algorithm fixed to avoid introducing bias in favor of a specific bonus.

Unless specified, our agents are trained for 200 million frames. Following Machado et al. (2018b) recommendations we run the ALE in the stochastic setting using sticky actions (ς = 0.25) for all agents during training and evaluation. We also do not use the mixed Monte-Carlo return (Bellemare et al., 2016; Ostrovski et al., 2017) or other algorithmic improvements that are combined with bonus-based methods (e.g. Burda et al., 2019) .

3.2. COMMON EVALUATION FRAMEWORK:

Our first set of results compare exploration methods introduced in Section 2 on MONTEZUMA'S REVENGE. We follow the standard procedure to use this game to tune the hyperparameters of each exploration bonus. Details regarding implementation and hyperparameter tuning may be found in Appendix A. Figure 1 shows training curves (averaged over 5 random seeds) for Rainbow augmented with the different exploration bonuses.

As anticipated, -greedy exploration performs poorly here and struggles in such a hard exploration game. On the other hand, exploration bonuses have a huge impact and all eventually beat the baselines -greedy and NoisyNets. Only ICM is unable to surpass the baselines by a large margin. RND, CTS and PixelCNN all average close to 5000 points. Interestingly, we observe that a more recent bonus like RND does not lead to higher performance over the older pseudo-count method. Of note, the performance we report at 200 millions frames improves on the performance reported in the original paper for each method. As expected, Rainbow leads to increased performance over older DQN variants previously used in the literature.

3.3. EXPLORATION METHODS EVALUATION:

It is standard in the exploration community to, first tune hyperparameters on MONTEZUMA'S REVENGE, and then, evaluate these hyperparameters on the remaining hard exploration games with sparse rewards. This is not in line with the original ALE guidelines which recommend to evaluate on the entire suite of Atari 2600 games. In this section we provide empirical evidence that, failing to follow the ALE guidelines may interfere with the conclusions of the evaluation.

Hard exploration games. We now turn our attention to the set of games categorized as hard exploration games with sparse rewards in Bellemare et al.'s taxonomy (the taxonomy is available in Appendix B). It includes ALE's most difficult games for exploration, this is where good exploration strategies should shine and provide the biggest improvements. These games are: GRAVITAR, PRIVATE EYE, VENTURE, SOLARIS, PITFALL! and FREEWAY.

We evaluate agents whose hyperparameters were tuned on MONTEZUMA'S REVENGE on this set of games. Training curves averaged over 5 runs are shown in Figure 2a . We find that performance of each method on MONTEZUMA'S REVENGE does not correlate with performance on other hard exploration domains. All methods seem to behave similarly contrary to our previous observations on MONTEZUMA'S REVENGE. In particular, there is no visible difference between -greedy and more sophisticated exploration bonuses. -greedy exploration is on par with every other method and even 

252. %:

NoisyNets percentage improvement over -greedy (by AUC) outperforms them by a significant margin on GRAVITAR. These games were initially classified to be hard exploration problems because a DQN agent using -greedy exploration was unable to achieve a high scoring policy; it is no longer the case with stronger learning agents available today. Full Atari suite. The set of hard exploration games was chosen to highlight the benefits of exploration methods, and, as a consequence, performance on this set may not be representative of an algorithm capabilities on the whole Atari suite. Indeed, exploration bonuses can negatively impact performance by skewing the reward landscape. To demonstrate how the choice of particular evaluation set can lead to different conclusions we also evaluate our agents on the original Atari training set which includes the games ASTERIX, FREEWAY, BEAM RIDER, SPACE INVADERS and SEAQUEST. Except for FREEWAY all these games are easy exploration problems (Bellemare et al., 2016) .

Figure 2b shows training curves for these games.

Pseudo-counts with a CTS model appears to struggle when the environment is an easy exploration problem and end up performing worse on every game except SEAQUEST. RND and ICM are able to consistently match -greedy exploration but never beat it. It appears that bonus-based methods have limited benefits in the context of easy exploration problems. Finally, despite its limited performance on MONTEZUMA'S REVENGE, we found that NoisyNets presented the most consistent improvements.

Overall, while the Atari training set and the set of the hard exploration games both show that bonusbased method only provide marginal improvements they lead us to different conclusions regarding the best performing exploration scheme. It appears that to fully quantify the behavior of exploration methods one cannot avoid evaluating on the whole Atari suite. We do so and add the remaining Atari games to our study. See Figure 3 and 4 and for a high level comparison. Altogether, it appears that the benefit of exploration bonuses is mainly allocated towards MONTEZUMA'S REVENGE. None of them, except NoisyNets, seems to improve over -greedy by significant margin. Bonus-based methods may lead to increased performance on a few games but seem to deteriorate performance by a roughly equal amount on other games.

We could have hoped that these methods focus their attention on hard exploration games at the expense of easier ones, meaning they trade exploitation for exploration. Unfortunately, it does not seem to be the case as they do not exhibit a preference for hard exploration games (red and orange bars). We may wonder if these methods are overfitting on MONTEZUMA'S REVENGE.

3.4. HYPERPARAMETER TUNING PROCEDURE:

Previous experiments have so far depicted a somewhat dismal picture of bonus-based methods, in particular their penchant to overfit on MONTEZUMA'S REVENGE. Though it is unfortunate to see they do not generalize to the full Atari gamut, one may wonder if their tendency to overfit to MONTEZUMA'S REVENGE specifically is caused by the hyperparameter tuning procedure or their inherent design. The experiments in this section aim at addressing this issue. We ran a new hyperparameter sweep on CTS and RND using a new training set. We chose these algorithms as we found them to be particularly sensitive to their hyperparameters. The new training set includes the games PONG, ASTERIX, SEAQUEST, Q*BERT and BREAKOUT. This set as been previously used by others as a training set to tune reinforcement learning agents (Bellemare et al., 2017; Dabney et al., 2018) .

Results are depicted in Figure 5 . Both algorithm still perform much better than -greedy exploration on MONTEZUMA'S REVENGE. As it is to be expected, performance also increased for games within the training set. A notable example is CTS on SEAQUEST which now improves over the -greedy baseline by 140% instead of 120% previously. Nevertheless, the conclusions from Section 3.3 remain valid. Bonus-based exploration strategies still provide only limited value except for a few games. In addition, neither algorithm seems to achieve better results on easy exploration problems outside of the training set.

3.5. AMOUNT OF TRAINING DATA:

Current results so far showed that exploration bonuses provide little benefits in the context of exploration in the ALE. Nonetheless, our experiments have been limited to 200 millions environment frames, it is possible that with additional data exploration bonuses would perform better. This would be in line with an emerging trend of training agents an order of magnitude longer in order to produce (Espeholt et al., 2018; Burda et al., 2019; Kapturowski et al., 2019) . In this section we present results that contradict this hypothesis. We reuse agents trained in Section 4.2.2 and lengthen the amount training data they process to 1 billion environment frames. We use Atari 2600 games from the set hard exploration with sparse rewards and the Atari training set. See Figure 6 for training curves. All exploration strategies see their score gracefully scale with additional data on easier exploration problems. In hard exploration games, none of the exploration strategies seem to benefit from receiving more data. Score on most games seem to plateau and may even decrease. This is particularly apparent in MONTEZUMA'S REVENGE where only RND actually benefits from longer training. After a while, agents are unable to make further progress and their bonus may collapse. When that happens, bonus-based methods cannot even rely on -greedy exploration to explore and may therefore see their performance decrease. This behavior may be attributed to our evaluation setting, tuning hyperparameters to perform best after one billion training frames will likely improve results. It is however unsatisfactory to see that exploration methods do not scale effortlessly as they receive more data. In practice, recent exploration bonuses have required a particular attention to handle large amount of training data (e.g. Burda et al., 2019) . Moreover, it is interesting to see that the performance at 200 millions frames leads to conclusions similar to those obtained with one billion frames.

4. CONCLUSION:

Recently, many exploration methods have been introduced with confounding factors within their evaluation -longer training duration, different model architecture and other ad-hoc techniques. This practice obscures the true value of these methods and makes it difficult to prioritize more promising directions of investigation. Therefore, following a growing trend in the reinforcement learning community, we advocate for better practices on empirical evaluation for exploration to fairly assess the contribution of newly proposed methods. In a standardized training environment and context, we found that current bonus-based methods are unable to surpass -greedy exploration. As a whole this suggest progress in bonus-based exploration may have been driven by confounding factors rather than improvement in the bonuses. This shows that more work is still needed to address the exploration problem in complex environments.

We may wonder why do we not see a positive impact of bonus-based exploration. One possible explanation is our use of the relatively sophisticated Rainbow learning algorithm. The benefits that others have observed using exploration bonuses might be made redundant through other mechanisms already in Rainbow, such as a prioritized replay buffer. An important transition may only be observed with low frequency but it can be sampled at a much higher rate by the replay buffer. As a result, the agent can learn from it effectively without the need for encouragement from the exploration bonus to visit that transition more frequently. Though the merits of bonus-based methods have been displayed with less efficient learning agents, these benefits did not carry on to improved learning agents. This is disappointing given that the simple NoisyNets baseline showed efficient exploration can still achieve noticeable gains on the ALE. As of now, the exploration and credit assignment communities have mostly been operating independently. Our work suggests that they may have to start working hand in hand to design new agents to make reinforcement learning truly sample efficient.

A HYPERPARAMETER 

CTS::

We tuned the scaling factor β.

We ran a sweep for β ∈ {0.5, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001} and found that β = 0.0005 worked best.

PixelCNN: We tuned the scaling factor and the prediction gain decay constant c. We ran a sweep with the following values: β ∈ {5.0, 1.0, 0.5, 0.1, 0.05}, c ∈ {5.0, 1.0, 0.5, 0.1, 0.05} and found β = 0.1 and c = 1.0 to work best.

A.3 ICM:

We tuned the scaling factor and the scalar α that weighs the inverse model loss against the forward model. We ran a sweep with α = {0.4, 0.2, 0.1, 0.05, 0.01, 0.005} and β = {2.0, 1.0, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0005}. We found α = 0.005 and β = 0.005 to work best.

A.4 RND Following Burda et al. (2019) we did not clip the intrinsic reward while the extrinsic reward was clipped.

We tuned the reward scaling factor β and learning rate used in Adam (Kingma & Ba, 2014) by the RND optimizer.

We ran a sweep with β = {0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.0005, 0.00001, 0.000005} and lr = {0.005, 0.001, 0.0005, 0.0002, 0.0001, 0.00005}. We found that β = 0.00005 and lr = 0.0001 worked best.

A.5 HYPERPARAMETER TUNING ON SECOND TRAINING SET (SECTION 3.4)

A.5.1 PSEUDO-COUNTS CTS:

We ran a sweep for β ∈ {0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001} and found that β = 0.0001 worked best.

A.6 RND:

We did not tune the learning rate and kept lr = 0.0001. We ran a sweep for β = {[0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005} and found that β = 0.00005 worked best. 

B TAXONOMY OF EXPLORATION ON THE ALE:



C ADDITIONAL FIGURES:

The variance of the return on MONTEZUMA'S REVENGE is high because the reward is a step function, for clarity we also provide all the training curves in Figure 7 We also provide training curves presented in the main paper in larger format. 

:



Figures and Tables:

Figure 1: A comparison of different exploration methods on MONTEZUMA'S REVENGE.

Figure 2: Evaluation of different bonus-based exploration methods on several Atari games, curves are averaged over 5 runs, shaded area denotes variance.

Figure 3: Improvements (in percentage of AUC) of Rainbow with various exploration methods over Rainbow with -greedy exploration in 60 Atari games. The game MONTEZUMA'S REVENGE is represented in purple.

Figure 4: Normalized score of bonus-based methods compared to -greedy

over -greedy (by AUC)

Figure 5: Improvements (in percentage of AUC) of Rainbow with CTS and RND over Rainbow with -greedy exploration in 60 Atari games when hyperparameters have been tuned on SEAQUEST, QBERT, PONG, BREAKOUT and ASTERIX. The game MONTEZUMA'S REVENGE is represented in purple. Games in the training set have stars on top of their bar.

Figure 6: Evaluation of bonus-based methods when training is extended to one billion frames

Figure 7: Training curves on MONTEZUMA'S REVENGE

TUNINGAll bonus based methods are tuned with respect to their final performance on MONTEZUMA'S REVENGE after training on 200 million frames averaged over five runs. When different hyper parameter settings led to comparable final performance we chose the one that achieve the performance the fastest.A.1 RAINBOW AND ATARI PREPROCESSING We used the standard Atari preprocessing fromMnih et al. (2015). FollowingMachado et al. (2018b)   recommendations we enable sticky actions and deactivated the termination on life loss heuristic. The remaining hyperparameters were chosen to matchHessel et al. (2018) implementation.

Table content: <table><tr><td>Hyperparameter</td><td>Value</td></tr><tr><td>Discount factor γ</td><td>0.99</td></tr><tr><td>Min history to start learning</td><td>80K frames</td></tr><tr><td>Target network update period</td><td>32K frames</td></tr><tr><td>Adam learning rate</td><td>6.25 × 10 -5</td></tr><tr><td>Adam</td><td>1.5 × 10 -4</td></tr><tr><td>Multi-step returns n</td><td>3</td></tr><tr><td>Distributional atoms</td><td>51</td></tr><tr><td>Distributional min/max values</td><td>[-10, 10]</td></tr></table>

Bellemare et al.'s taxonomy of exploration propose a classification of Atari 2600 games in terms of difficulty of exploration. It is provided in Table 1. A classification of Atari 2600 games based on their exploration difficulty.

Table content: <table><tr><td/><td>Easy Exploration</td><td/><td colspan="2">Hard exploration</td></tr><tr><td colspan="2">Human Optimal</td><td>Score Exploit</td><td>Dense Reward</td><td>Sparse Reward</td></tr><tr><td>ASSAULT</td><td>ASTERIX</td><td>BEAM RIDER</td><td>ALIEN</td><td>FREEWAY</td></tr><tr><td>ASTEROIDS</td><td>ATLANTIS</td><td>KANGAROO</td><td>AMIDAR</td><td>GRAVITAR</td></tr><tr><td>BATTLE ZONE</td><td>BERZERK</td><td>KRULL</td><td>BANK HEIST</td><td>MONTEZUMA'S REVENGE</td></tr><tr><td>BOWLING</td><td>BOXING</td><td>KUNG-FU MASTER</td><td>FROSTBITE</td><td>PITFALL!</td></tr><tr><td>BREAKOUT</td><td>CENTIPEDE</td><td>ROAD RUNNER</td><td>H.E.R.O</td><td>PRIVATE EYE</td></tr><tr><td>CHOPPER CMD</td><td>CRAZY CLIMBER</td><td>SEAQUEST</td><td>MS. PAC-MAN</td><td>SOLARIS</td></tr><tr><td>DEFENDER</td><td>DEMON ATTACK</td><td>UP N DOWN</td><td>Q*BERT</td><td>VENTURE</td></tr><tr><td>DOUBLE DUNK</td><td>ENDURO</td><td>TUTANKHAM</td><td>SURROUND</td><td/></tr><tr><td>FISHING DERBY</td><td>GOPHER</td><td/><td>WIZARD OF WOR</td><td/></tr><tr><td>ICE HOCKEY</td><td>JAMES BOND</td><td/><td>ZAXXON</td><td/></tr><tr><td>NAME THIS GAME</td><td>PHOENIX</td><td/><td/><td/></tr><tr><td>PONG</td><td>RIVER RAID</td><td/><td/><td/></tr><tr><td>ROBOTANK</td><td>SKIING</td><td/><td/><td/></tr><tr><td>SPACE INVADERS</td><td>STARGUNNER</td><td/><td/><td/></tr></table>

Raw scores for all games after 10M training frames

Table content: <table><tr><td>Games</td><td>-greedy</td><td>RND</td><td colspan="2">CTS PixelCNN</td><td colspan="2">ICM NoisyNets</td></tr><tr><td>AirRaid</td><td>5393.2</td><td>5374.7</td><td>4084.7</td><td>5425.8</td><td>5374.7</td><td>4361.4</td></tr><tr><td>Alien</td><td>1289.6</td><td>1352.1</td><td>1250.6</td><td>1228.0</td><td>1310.6</td><td>1222.0</td></tr><tr><td>Amidar</td><td>335.0</td><td>286.8</td><td>345.1</td><td>428.0</td><td>364.9</td><td>324.7</td></tr><tr><td>Assault</td><td>1459.5</td><td>1453.4</td><td>1303.6</td><td>1641.9</td><td>1431.7</td><td>1080.4</td></tr><tr><td>Asterix</td><td>2777.5</td><td>2793.9</td><td>2841.0</td><td>2748.0</td><td>2736.7</td><td>2698.6</td></tr><tr><td>Asteroids</td><td>759.6</td><td>848.4</td><td>722.7</td><td>800.5</td><td>789.5</td><td>1117.0</td></tr><tr><td>Atlantis</td><td colspan="2">117166.5 172134.9</td><td>99510.6</td><td colspan="2">184420.5 148927.5</td><td>109464.9</td></tr><tr><td>BankHeist</td><td>901.5</td><td>896.5</td><td>426.7</td><td>756.8</td><td>904.4</td><td>1039.9</td></tr><tr><td>BattleZone</td><td>23909.5</td><td>22374.9</td><td>9734.0</td><td>22822.1</td><td>22575.3</td><td>11581.6</td></tr><tr><td>BeamRider</td><td>5475.7</td><td>5135.5</td><td>4066.8</td><td>3415.4</td><td>5796.6</td><td>3845.6</td></tr><tr><td>Berzerk</td><td>419.1</td><td>442.5</td><td>446.8</td><td>435.5</td><td>430.8</td><td>476.0</td></tr><tr><td>Bowling</td><td>40.8</td><td>39.0</td><td>92.6</td><td>47.6</td><td>35.5</td><td>27.1</td></tr><tr><td>Boxing</td><td>75.7</td><td>74.8</td><td>45.2</td><td>76.0</td><td>74.7</td><td>57.5</td></tr><tr><td>Breakout</td><td>49.2</td><td>49.5</td><td>45.6</td><td>46.4</td><td>48.8</td><td>44.7</td></tr><tr><td>Carnival</td><td>4644.9</td><td>4375.3</td><td>3658.4</td><td>4138.8</td><td>4317.0</td><td>2291.5</td></tr><tr><td>Centipede</td><td>4845.3</td><td>5009.4</td><td>4731.0</td><td>4963.4</td><td>5110.2</td><td>4912.0</td></tr><tr><td>ChopperCommand</td><td>2494.5</td><td>913.6</td><td>2293.9</td><td>2420.5</td><td>2376.6</td><td>1899.1</td></tr><tr><td>CrazyClimber</td><td colspan="2">107706.9 105940.9</td><td>90549.0</td><td colspan="2">107724.2 104783.2</td><td>96063.8</td></tr><tr><td>DemonAttack</td><td>2932.2</td><td>3135.8</td><td>1442.0</td><td>2564.6</td><td>3153.3</td><td>1864.6</td></tr><tr><td>DoubleDunk</td><td>-18.4</td><td>-18.1</td><td>-19.4</td><td>-18.2</td><td>-19.0</td><td>-8.2</td></tr><tr><td>ElevatorAction</td><td>29.1</td><td>114.7</td><td>0.0</td><td>141.8</td><td>22.9</td><td>0.0</td></tr><tr><td>Enduro</td><td>1657.1</td><td>1719.1</td><td>1230.0</td><td>1767.0</td><td>1727.4</td><td>1272.9</td></tr><tr><td>FishingDerby</td><td>15.5</td><td>9.7</td><td>13.8</td><td>15.0</td><td>14.5</td><td>10.8</td></tr><tr><td>Freeway</td><td>32.3</td><td>32.2</td><td>32.2</td><td>32.3</td><td>32.2</td><td>31.7</td></tr><tr><td>Frostbite</td><td>3035.7</td><td>3286.5</td><td>3078.9</td><td>2787.7</td><td>2854.7</td><td>2913.0</td></tr><tr><td>Gopher</td><td>4047.6</td><td>4224.1</td><td>4295.8</td><td>4349.7</td><td>4266.1</td><td>3760.1</td></tr><tr><td>Gravitar</td><td>229.4</td><td>236.8</td><td>130.0</td><td>235.1</td><td>225.8</td><td>211.3</td></tr><tr><td>Hero</td><td>10730.5</td><td>10858.1</td><td>12753.7</td><td>12999.9</td><td>11808.4</td><td>9441.1</td></tr><tr><td>IceHockey</td><td>-5.0</td><td>-4.9</td><td>-11.9</td><td>-5.9</td><td>-5.1</td><td>-7.7</td></tr><tr><td>Jamesbond</td><td>520.3</td><td>508.0</td><td>473.2</td><td>536.0</td><td>517.0</td><td>484.6</td></tr><tr><td>JourneyEscape</td><td>-2376.4</td><td>-2299.0</td><td>-9208.8</td><td>-2463.4</td><td>-2414.1</td><td>-1784.3</td></tr><tr><td>Kangaroo</td><td>5233.1</td><td>5537.0</td><td>1747.0</td><td>5266.2</td><td>5823.1</td><td>5147.0</td></tr><tr><td>Krull</td><td>5941.1</td><td>5997.8</td><td>5620.4</td><td>5999.8</td><td>5935.5</td><td>6195.1</td></tr><tr><td>KungFuMaster</td><td>19824.1</td><td>17676.9</td><td>22668.2</td><td>21673.9</td><td>19199.3</td><td>19150.5</td></tr><tr><td>MontezumaRevenge</td><td>0.5</td><td>1.5</td><td>231.8</td><td>17.8</td><td>0.0</td><td>13.2</td></tr><tr><td>MsPacman</td><td>2371.0</td><td>2271.4</td><td>2452.5</td><td>2443.6</td><td>2424.6</td><td>2436.4</td></tr><tr><td>NameThisGame</td><td>9086.7</td><td>8561.7</td><td>7402.8</td><td>9089.4</td><td>8820.6</td><td>7590.1</td></tr><tr><td>Phoenix</td><td>4758.3</td><td>4790.7</td><td>4498.7</td><td>4840.8</td><td>4788.1</td><td>4868.0</td></tr><tr><td>Pitfall</td><td>-13.3</td><td>-10.4</td><td>-2.4</td><td>-11.1</td><td>-5.6</td><td>-3.0</td></tr><tr><td>Pong</td><td>14.6</td><td>14.0</td><td>9.8</td><td>14.3</td><td>15.5</td><td>12.4</td></tr><tr><td>Pooyan</td><td>3917.3</td><td>3867.7</td><td>3885.9</td><td>3996.4</td><td>3722.1</td><td>3623.4</td></tr><tr><td>PrivateEye</td><td>77.4</td><td>180.3</td><td>-903.9</td><td>89.1</td><td>116.5</td><td>78.9</td></tr><tr><td>Qbert</td><td>7025.6</td><td>7442.4</td><td>5466.8</td><td>5885.5</td><td>8090.2</td><td>4363.0</td></tr><tr><td>Riverraid</td><td>4578.2</td><td>4904.3</td><td>6830.2</td><td>5002.2</td><td>4739.3</td><td>3820.5</td></tr><tr><td>RoadRunner</td><td>33743.4</td><td>34944.0</td><td>39145.9</td><td>35081.6</td><td>34109.7</td><td>33649.4</td></tr><tr><td>Robotank</td><td>22.5</td><td>22.7</td><td>1.8</td><td>17.9</td><td>22.6</td><td>18.5</td></tr><tr><td>Seaquest</td><td>1711.6</td><td>1744.0</td><td>1476.0</td><td>1631.9</td><td>1647.4</td><td>1155.9</td></tr><tr><td>Skiing</td><td colspan="3">-22580.5 -21378.9 -27888.8</td><td colspan="2">-23652.6 -23040.9</td><td>-22729.0</td></tr><tr><td>Solaris</td><td>1164.7</td><td>1103.4</td><td>1108.2</td><td>966.8</td><td>1254.2</td><td>1495.9</td></tr><tr><td>SpaceInvaders</td><td>793.6</td><td>764.5</td><td>641.6</td><td>800.6</td><td>780.6</td><td>687.0</td></tr><tr><td>StarGunner</td><td>1533.2</td><td>1523.3</td><td>1579.6</td><td>1530.3</td><td>1500.3</td><td>1587.5</td></tr><tr><td>Tennis</td><td>-2.8</td><td>-0.4</td><td>-21.2</td><td>-7.5</td><td>-1.0</td><td>-1.0</td></tr><tr><td>TimePilot</td><td>2969.7</td><td>3001.6</td><td>4516.1</td><td>2810.5</td><td>3049.9</td><td>2710.6</td></tr><tr><td>Tutankham</td><td>187.0</td><td>207.2</td><td>54.3</td><td>232.3</td><td>169.2</td><td>162.2</td></tr><tr><td>UpNDown</td><td>14237.9</td><td>11060.1</td><td>4373.8</td><td>12494.2</td><td>13220.7</td><td>16120.4</td></tr><tr><td>Venture</td><td>25.0</td><td>16.4</td><td>13.3</td><td>17.8</td><td>19.0</td><td>34.9</td></tr><tr><td>VideoPinball</td><td>33347.6</td><td>30284.0</td><td>31173.0</td><td>33294.5</td><td>30661.6</td><td>22802.8</td></tr><tr><td>WizardOfWor</td><td>2432.5</td><td>2865.3</td><td>1451.5</td><td>2376.6</td><td>2191.3</td><td>1568.7</td></tr><tr><td>YarsRevenge</td><td>10349.2</td><td>11161.2</td><td>9926.9</td><td>11002.0</td><td>11415.6</td><td>10930.2</td></tr><tr><td>Zaxxon</td><td>4282.7</td><td>4707.7</td><td>3603.9</td><td>3766.4</td><td>4266.8</td><td>3016.6</td></tr></table>

Raw scores for all games after 50M training frames

Table content: <table><tr><td>Games</td><td>-greedy</td><td>RND</td><td colspan="2">CTS PixelCNN</td><td colspan="2">ICM NoisyNets</td></tr><tr><td>AirRaid</td><td>12081.8</td><td>13005.2</td><td>11016.9</td><td>12399.0</td><td>11392.8</td><td>10433.7</td></tr><tr><td>Alien</td><td>2114.1</td><td>2004.9</td><td>1969.4</td><td>2200.6</td><td>1979.4</td><td>2308.3</td></tr><tr><td>Amidar</td><td>1535.6</td><td>1140.6</td><td>1469.5</td><td>1791.9</td><td>1504.0</td><td>1337.2</td></tr><tr><td>Assault</td><td>2384.1</td><td>2423.8</td><td>2265.6</td><td>2465.2</td><td>2555.6</td><td>3050.1</td></tr><tr><td>Asterix</td><td>5722.4</td><td>6488.7</td><td>4959.4</td><td>5439.3</td><td>5931.7</td><td>13981.5</td></tr><tr><td>Asteroids</td><td>1363.7</td><td>1409.2</td><td>1095.8</td><td>1315.3</td><td>1247.5</td><td>1901.6</td></tr><tr><td>Atlantis</td><td colspan="3">831286.0 807958.0 805574.0</td><td colspan="2">816578.0 763538.0</td><td>827702.0</td></tr><tr><td>BankHeist</td><td>1113.8</td><td>1070.0</td><td>1154.6</td><td>1005.5</td><td>1028.9</td><td>1104.3</td></tr><tr><td>BattleZone</td><td>30959.7</td><td>31271.5</td><td>16167.4</td><td>31011.8</td><td>32357.5</td><td>30606.6</td></tr><tr><td>BeamRider</td><td>5550.9</td><td>5782.2</td><td>4726.8</td><td>5504.9</td><td>5633.0</td><td>5636.4</td></tr><tr><td>Berzerk</td><td>697.9</td><td>643.8</td><td>700.4</td><td>662.8</td><td>667.5</td><td>731.7</td></tr><tr><td>Bowling</td><td>41.4</td><td>42.6</td><td>45.5</td><td>56.5</td><td>36.1</td><td>30.0</td></tr><tr><td>Boxing</td><td>95.7</td><td>95.8</td><td>91.2</td><td>95.8</td><td>96.0</td><td>98.0</td></tr><tr><td>Breakout</td><td>48.3</td><td>48.3</td><td>48.6</td><td>49.7</td><td>51.2</td><td>82.3</td></tr><tr><td>Carnival</td><td>4936.0</td><td>4554.8</td><td>5199.7</td><td>4600.6</td><td>4644.4</td><td>4110.6</td></tr><tr><td>Centipede</td><td>4312.1</td><td>4568.4</td><td>4235.0</td><td>4544.2</td><td>4652.0</td><td>4986.9</td></tr><tr><td>ChopperCommand</td><td>6438.2</td><td>2087.1</td><td>6504.8</td><td>6010.3</td><td>5840.4</td><td>5589.4</td></tr><tr><td>CrazyClimber</td><td colspan="3">131002.1 132982.7 126655.8</td><td colspan="2">133039.8 130491.3</td><td>133873.7</td></tr><tr><td>DemonAttack</td><td>6021.3</td><td>5590.7</td><td>4499.4</td><td>4985.7</td><td>6052.2</td><td>13754.6</td></tr><tr><td>DoubleDunk</td><td>17.9</td><td>15.4</td><td>-11.0</td><td>20.0</td><td>18.0</td><td>-0.6</td></tr><tr><td>ElevatorAction</td><td>55982.0</td><td>66996.0</td><td>0.0</td><td>52254.0</td><td>58349.4</td><td>17514.5</td></tr><tr><td>Enduro</td><td>2173.9</td><td>2076.4</td><td>2105.2</td><td>2119.4</td><td>1683.3</td><td>2180.9</td></tr><tr><td>FishingDerby</td><td>35.1</td><td>32.7</td><td>23.9</td><td>37.2</td><td>37.5</td><td>33.6</td></tr><tr><td>Freeway</td><td>33.4</td><td>33.3</td><td>33.3</td><td>33.3</td><td>33.4</td><td>33.7</td></tr><tr><td>Frostbite</td><td>5773.9</td><td>6162.9</td><td>6334.9</td><td>5759.3</td><td>5551.6</td><td>6010.9</td></tr><tr><td>Gopher</td><td>6564.7</td><td>6445.1</td><td>6111.8</td><td>6688.1</td><td>6761.3</td><td>10255.0</td></tr><tr><td>Gravitar</td><td>1148.3</td><td>990.6</td><td>635.6</td><td>953.9</td><td>950.7</td><td>903.5</td></tr><tr><td>Hero</td><td>29014.5</td><td>30679.0</td><td>29273.5</td><td>31150.1</td><td>30756.5</td><td>27165.2</td></tr><tr><td>IceHockey</td><td>1.3</td><td>1.0</td><td>-3.8</td><td>1.8</td><td>1.8</td><td>0.9</td></tr><tr><td>Jamesbond</td><td>600.8</td><td>633.0</td><td>669.5</td><td>712.6</td><td>622.9</td><td>707.7</td></tr><tr><td>JourneyEscape</td><td>-1448.3</td><td>-1422.1</td><td>-3424.2</td><td>-1437.9</td><td>-1541.1</td><td>-1304.6</td></tr><tr><td>Kangaroo</td><td>12426.5</td><td>12731.0</td><td>12333.7</td><td>11771.7</td><td>12606.5</td><td>12428.5</td></tr><tr><td>Krull</td><td>4522.9</td><td>4065.4</td><td>4041.7</td><td>4051.8</td><td>4319.1</td><td>4341.5</td></tr><tr><td>KungFuMaster</td><td>26929.2</td><td>22972.3</td><td>25573.2</td><td>26135.5</td><td>25883.8</td><td>30161.7</td></tr><tr><td>MontezumaRevenge</td><td>553.3</td><td>2789.6</td><td>2550.4</td><td>2528.5</td><td>1160.7</td><td>2492.9</td></tr><tr><td>MsPacman</td><td>3185.3</td><td>3218.8</td><td>3272.4</td><td>3400.7</td><td>3310.0</td><td>3768.5</td></tr><tr><td>NameThisGame</td><td>8304.9</td><td>8106.9</td><td>6978.1</td><td>8467.3</td><td>8222.7</td><td>10883.2</td></tr><tr><td>Phoenix</td><td>5862.1</td><td>7434.0</td><td>4781.2</td><td>7664.7</td><td>7369.7</td><td>11873.4</td></tr><tr><td>Pitfall</td><td>-4.0</td><td>-3.6</td><td>-13.9</td><td>-10.1</td><td>-13.1</td><td>-4.5</td></tr><tr><td>Pong</td><td>19.5</td><td>19.0</td><td>19.9</td><td>19.2</td><td>19.1</td><td>19.2</td></tr><tr><td>Pooyan</td><td>3703.3</td><td>3884.6</td><td>3724.3</td><td>3727.1</td><td>3775.3</td><td>4561.7</td></tr><tr><td>PrivateEye</td><td>127.6</td><td>319.6</td><td>948.6</td><td>679.7</td><td>12156.0</td><td>96.2</td></tr><tr><td>Qbert</td><td>15498.0</td><td>17014.5</td><td>14804.3</td><td>16463.1</td><td>16786.7</td><td>16780.0</td></tr><tr><td>Riverraid</td><td>14935.6</td><td>15797.5</td><td>15747.2</td><td>16137.5</td><td>15590.9</td><td>10469.4</td></tr><tr><td>RoadRunner</td><td>50559.5</td><td>50349.6</td><td>50915.7</td><td>49905.1</td><td>50042.5</td><td>46314.5</td></tr><tr><td>Robotank</td><td>62.2</td><td>62.0</td><td>4.6</td><td>59.4</td><td>60.5</td><td>63.9</td></tr><tr><td>Seaquest</td><td>4915.9</td><td>5036.7</td><td>19227.6</td><td>3708.0</td><td>3331.5</td><td>4433.2</td></tr><tr><td>Skiing</td><td colspan="3">-21899.7 -24633.1 -23028.8</td><td colspan="2">-25674.1 -24216.7</td><td>-25306.5</td></tr><tr><td>Solaris</td><td>2272.7</td><td>2347.2</td><td>1921.1</td><td>1444.2</td><td>2076.0</td><td>2063.7</td></tr><tr><td>SpaceInvaders</td><td>1904.4</td><td>1881.2</td><td>1275.9</td><td>2007.6</td><td>1827.3</td><td>1888.8</td></tr><tr><td>StarGunner</td><td>35867.0</td><td>38744.5</td><td>33868.7</td><td>39416.2</td><td>41771.6</td><td>42542.3</td></tr><tr><td>Tennis</td><td>-0.1</td><td>0.0</td><td>22.1</td><td>-14.3</td><td>-0.7</td><td>-1.2</td></tr><tr><td>TimePilot</td><td>8901.6</td><td>9295.9</td><td>4785.5</td><td>8927.9</td><td>9220.9</td><td>8956.7</td></tr><tr><td>Tutankham</td><td>227.9</td><td>235.1</td><td>96.6</td><td>234.6</td><td>220.8</td><td>201.8</td></tr><tr><td>UpNDown</td><td>15097.4</td><td>13116.4</td><td>9151.3</td><td>14436.6</td><td>14621.5</td><td>39546.4</td></tr><tr><td>Venture</td><td>1460.4</td><td>1435.4</td><td>1117.7</td><td>1426.2</td><td>1443.2</td><td>1220.3</td></tr><tr><td>VideoPinball</td><td colspan="3">398511.1 419047.9 414727.1</td><td colspan="2">386829.1 452874.2</td><td>154968.4</td></tr><tr><td>WizardOfWor</td><td>6941.0</td><td>5932.6</td><td>4108.2</td><td>6226.2</td><td>7027.8</td><td>6906.1</td></tr><tr><td>YarsRevenge</td><td>30769.2</td><td>36086.1</td><td>30596.5</td><td>32049.2</td><td>30721.4</td><td>36136.7</td></tr><tr><td>Zaxxon</td><td>13161.2</td><td>13341.5</td><td>7223.7</td><td>11332.8</td><td>13177.1</td><td>13936.2</td></tr></table>

Raw scores for all games after 100M training frames

Table content: <table><tr><td>Games</td><td>-greedy</td><td>RND</td><td colspan="2">CTS PixelCNN</td><td colspan="2">ICM NoisyNets</td></tr><tr><td>AirRaid</td><td>11744.3</td><td>12081.2</td><td>11819.6</td><td>13292.9</td><td>12744.2</td><td>10096.6</td></tr><tr><td>Alien</td><td>2696.7</td><td>2531.9</td><td>2519.4</td><td>2761.6</td><td>2593.9</td><td>3237.0</td></tr><tr><td>Amidar</td><td>2033.1</td><td>1527.9</td><td>1797.0</td><td>2295.1</td><td>2167.2</td><td>1872.9</td></tr><tr><td>Assault</td><td>2946.8</td><td>2832.5</td><td>2535.3</td><td>2839.1</td><td>2574.8</td><td>3718.9</td></tr><tr><td>Asterix</td><td>9279.4</td><td>10785.7</td><td>6083.1</td><td>8540.0</td><td>10661.5</td><td>26367.6</td></tr><tr><td>Asteroids</td><td>1374.3</td><td>1423.3</td><td>1156.8</td><td>1391.8</td><td>1266.7</td><td>2169.9</td></tr><tr><td>Atlantis</td><td colspan="3">859040.0 799144.2 747436.0</td><td colspan="2">852382.0 754256.0</td><td>857076.0</td></tr><tr><td>BankHeist</td><td>1106.0</td><td>1090.6</td><td>1131.7</td><td>932.2</td><td>1013.4</td><td>1156.8</td></tr><tr><td>BattleZone</td><td>32850.4</td><td>30523.1</td><td>19369.3</td><td>35162.6</td><td>32625.4</td><td>35483.1</td></tr><tr><td>BeamRider</td><td>6044.1</td><td>5977.9</td><td>5197.7</td><td>5644.8</td><td>6070.2</td><td>6117.3</td></tr><tr><td>Berzerk</td><td>746.4</td><td>709.1</td><td>788.6</td><td>728.7</td><td>715.5</td><td>894.2</td></tr><tr><td>Bowling</td><td>36.0</td><td>32.5</td><td>45.5</td><td>61.9</td><td>31.5</td><td>29.9</td></tr><tr><td>Boxing</td><td>97.5</td><td>97.6</td><td>96.6</td><td>96.9</td><td>97.7</td><td>98.7</td></tr><tr><td>Breakout</td><td>56.9</td><td>59.8</td><td>57.7</td><td>58.1</td><td>63.2</td><td>75.8</td></tr><tr><td>Carnival</td><td>4805.4</td><td>4743.3</td><td>5040.4</td><td>4427.9</td><td>4623.5</td><td>4480.7</td></tr><tr><td>Centipede</td><td>4460.7</td><td>4909.7</td><td>4701.9</td><td>4613.3</td><td>4700.2</td><td>5154.6</td></tr><tr><td>ChopperCommand</td><td>9812.9</td><td>2306.4</td><td>9742.8</td><td>8967.3</td><td>9238.2</td><td>7611.4</td></tr><tr><td>CrazyClimber</td><td colspan="3">135531.2 139641.8 133805.3</td><td colspan="2">139011.0 138967.9</td><td>145380.2</td></tr><tr><td>DemonAttack</td><td>7570.5</td><td>6883.6</td><td>6268.4</td><td>5736.4</td><td>7298.5</td><td>26478.1</td></tr><tr><td>DoubleDunk</td><td>21.6</td><td>20.1</td><td>-1.4</td><td>21.8</td><td>21.6</td><td>-1.3</td></tr><tr><td>ElevatorAction</td><td>74358.0</td><td>74846.0</td><td>0.0</td><td>71432.9</td><td>73506.0</td><td>64614.0</td></tr><tr><td>Enduro</td><td>2155.8</td><td>2159.9</td><td>2132.2</td><td>2112.2</td><td>237.6</td><td>2271.2</td></tr><tr><td>FishingDerby</td><td>44.3</td><td>40.1</td><td>34.2</td><td>42.4</td><td>42.9</td><td>44.0</td></tr><tr><td>Freeway</td><td>33.6</td><td>33.5</td><td>33.5</td><td>33.6</td><td>33.6</td><td>33.8</td></tr><tr><td>Frostbite</td><td>7139.6</td><td>6980.3</td><td>7758.2</td><td>6777.8</td><td>5901.5</td><td>7391.7</td></tr><tr><td>Gopher</td><td>7815.1</td><td>7421.3</td><td>7295.3</td><td>7578.1</td><td>7581.9</td><td>12069.6</td></tr><tr><td>Gravitar</td><td>1763.4</td><td>1171.2</td><td>1050.4</td><td>1132.8</td><td>1165.9</td><td>1149.1</td></tr><tr><td>Hero</td><td>38102.2</td><td>38727.3</td><td>35444.2</td><td>37496.3</td><td>38217.7</td><td>35595.3</td></tr><tr><td>IceHockey</td><td>0.4</td><td>0.7</td><td>-0.5</td><td>-0.2</td><td>-0.3</td><td>5.2</td></tr><tr><td>Jamesbond</td><td>586.0</td><td>650.6</td><td>747.1</td><td>896.6</td><td>663.7</td><td>733.9</td></tr><tr><td>JourneyEscape</td><td>-1224.9</td><td>-1261.3</td><td>-3441.8</td><td>-1241.1</td><td>-1250.5</td><td>-992.9</td></tr><tr><td>Kangaroo</td><td>12676.2</td><td>12404.3</td><td>13028.7</td><td>12691.8</td><td>12834.1</td><td>12617.5</td></tr><tr><td>Krull</td><td>4250.4</td><td>4016.2</td><td>4171.0</td><td>4242.7</td><td>4046.5</td><td>4290.8</td></tr><tr><td>KungFuMaster</td><td>26459.1</td><td>23429.2</td><td>25450.0</td><td>27472.8</td><td>26899.8</td><td>34510.8</td></tr><tr><td>MontezumaRevenge</td><td>780.0</td><td>5188.8</td><td>5118.8</td><td>4364.6</td><td>2449.8</td><td>2486.2</td></tr><tr><td>MsPacman</td><td>3774.2</td><td>3749.3</td><td>3577.0</td><td>3694.1</td><td>3694.0</td><td>4376.9</td></tr><tr><td>NameThisGame</td><td>8309.9</td><td>8086.0</td><td>7598.0</td><td>7947.5</td><td>7810.1</td><td>11602.0</td></tr><tr><td>Phoenix</td><td>6340.5</td><td>7073.2</td><td>4779.7</td><td>7576.2</td><td>6902.3</td><td>14687.3</td></tr><tr><td>Pitfall</td><td>-9.8</td><td>-1.6</td><td>-5.4</td><td>-15.1</td><td>-18.8</td><td>-2.0</td></tr><tr><td>Pong</td><td>20.1</td><td>19.4</td><td>20.1</td><td>19.6</td><td>19.6</td><td>19.7</td></tr><tr><td>Pooyan</td><td>3920.6</td><td>4125.0</td><td>3946.0</td><td>4103.6</td><td>3871.5</td><td>4731.5</td></tr><tr><td>PrivateEye</td><td>12284.7</td><td>3371.9</td><td>10081.2</td><td>26765.1</td><td>13566.5</td><td>216.6</td></tr><tr><td>Qbert</td><td>17261.8</td><td>19027.1</td><td>17926.5</td><td>17453.1</td><td>17072.6</td><td>18169.5</td></tr><tr><td>Riverraid</td><td>18317.0</td><td>19032.5</td><td>18281.6</td><td>18392.1</td><td>17705.1</td><td>13969.1</td></tr><tr><td>RoadRunner</td><td>52548.1</td><td>52626.6</td><td>52641.4</td><td>52853.6</td><td>52164.1</td><td>52048.7</td></tr><tr><td>Robotank</td><td>62.4</td><td>62.1</td><td>13.8</td><td>59.3</td><td>61.4</td><td>67.0</td></tr><tr><td>Seaquest</td><td>9605.5</td><td>7083.1</td><td>23277.5</td><td>6833.4</td><td>6403.1</td><td>9315.5</td></tr><tr><td>Skiing</td><td colspan="3">-25020.7 -24296.5 -23602.8</td><td colspan="2">-23077.5 -24954.3</td><td>-25814.9</td></tr><tr><td>Solaris</td><td>2187.8</td><td>1877.9</td><td>1951.8</td><td>1375.9</td><td>1939.5</td><td>2032.2</td></tr><tr><td>SpaceInvaders</td><td>2621.1</td><td>2417.1</td><td>1682.1</td><td>2955.8</td><td>2604.4</td><td>5527.1</td></tr><tr><td>StarGunner</td><td>45718.0</td><td>46682.3</td><td>46281.6</td><td>48348.0</td><td>51118.0</td><td>55886.7</td></tr><tr><td>Tennis</td><td>-0.8</td><td>-0.1</td><td>22.8</td><td>22.7</td><td>-1.1</td><td>-0.0</td></tr><tr><td>TimePilot</td><td>10484.4</td><td>10637.3</td><td>5421.0</td><td>10474.8</td><td>10281.4</td><td>11161.8</td></tr><tr><td>Tutankham</td><td>240.8</td><td>232.4</td><td>147.8</td><td>240.2</td><td>223.3</td><td>212.0</td></tr><tr><td>UpNDown</td><td>21518.4</td><td>15871.6</td><td>11277.8</td><td>16540.0</td><td>17411.3</td><td>36217.2</td></tr><tr><td>Venture</td><td>1541.0</td><td>1515.0</td><td>1303.8</td><td>1470.2</td><td>1529.8</td><td>1528.2</td></tr><tr><td>VideoPinball</td><td colspan="3">453118.7 431034.8 410651.4</td><td colspan="2">431198.8 414069.2</td><td>327080.8</td></tr><tr><td>WizardOfWor</td><td>7172.8</td><td>6163.0</td><td>4719.3</td><td>6580.8</td><td>7323.6</td><td>8578.7</td></tr><tr><td>YarsRevenge</td><td>36432.8</td><td>42317.0</td><td>35471.4</td><td>37539.3</td><td>36775.4</td><td>44978.0</td></tr><tr><td>Zaxxon</td><td>14106.7</td><td>13750.8</td><td>8881.5</td><td>11179.1</td><td>14333.5</td><td>18290.2</td></tr></table>

Raw scores for all games after 200M training frames

Table content: <table><tr><td>Games</td><td>-greedy</td><td>RND</td><td colspan="2">CTS PixelCNN</td><td colspan="2">ICM NoisyNets</td></tr><tr><td>AirRaid</td><td>12719.7</td><td>12035.0</td><td>12296.9</td><td>12849.5</td><td>12923.5</td><td>10097.4</td></tr><tr><td>Alien</td><td>3461.2</td><td>3393.2</td><td>3177.4</td><td>3622.7</td><td>3455.5</td><td>4562.1</td></tr><tr><td>Amidar</td><td>2411.3</td><td>2263.8</td><td>2311.2</td><td>2499.6</td><td>2601.6</td><td>2837.4</td></tr><tr><td>Assault</td><td>3215.9</td><td>3247.1</td><td>2970.2</td><td>3026.7</td><td>1829.1</td><td>4929.5</td></tr><tr><td>Asterix</td><td>17337.1</td><td>18772.5</td><td>14855.6</td><td>19092.0</td><td>17287.0</td><td>42993.4</td></tr><tr><td>Asteroids</td><td>1609.5</td><td>1456.2</td><td>1350.5</td><td>1474.3</td><td>1442.0</td><td>3392.9</td></tr><tr><td>Atlantis</td><td colspan="3">843635.3 796360.0 752722.0</td><td colspan="2">874296.2 742868.0</td><td>860326.0</td></tr><tr><td>BankHeist</td><td>1041.1</td><td>1132.0</td><td>1086.1</td><td>621.9</td><td>1137.3</td><td>1255.0</td></tr><tr><td>BattleZone</td><td>38978.1</td><td>32521.0</td><td>29587.0</td><td>38601.4</td><td>37003.4</td><td>37314.9</td></tr><tr><td>BeamRider</td><td>5791.0</td><td>6416.3</td><td>5433.5</td><td>4604.3</td><td>6499.4</td><td>6016.1</td></tr><tr><td>Berzerk</td><td>772.9</td><td>772.1</td><td>845.5</td><td>779.3</td><td>740.5</td><td>938.4</td></tr><tr><td>Bowling</td><td>39.7</td><td>34.4</td><td>41.3</td><td>49.1</td><td>42.2</td><td>33.2</td></tr><tr><td>Boxing</td><td>98.2</td><td>98.4</td><td>97.8</td><td>98.2</td><td>98.5</td><td>98.9</td></tr><tr><td>Breakout</td><td>102.3</td><td>118.8</td><td>112.8</td><td>104.6</td><td>113.7</td><td>129.2</td></tr><tr><td>Carnival</td><td>4792.3</td><td>4726.3</td><td>5058.3</td><td>4385.5</td><td>4585.1</td><td>4464.1</td></tr><tr><td>Centipede</td><td>6470.3</td><td>6933.9</td><td>6572.6</td><td>6434.8</td><td>6600.6</td><td>6466.8</td></tr><tr><td>ChopperCommand</td><td>12578.4</td><td>2674.9</td><td>12133.2</td><td>12396.4</td><td>11595.2</td><td>9556.3</td></tr><tr><td>CrazyClimber</td><td colspan="3">145511.7 146489.0 145465.8</td><td colspan="2">147683.8 145570.3</td><td>153267.5</td></tr><tr><td>DemonAttack</td><td>19737.1</td><td>14349.2</td><td>11689.9</td><td>9350.5</td><td>18477.4</td><td>48634.6</td></tr><tr><td>DoubleDunk</td><td>22.3</td><td>21.9</td><td>10.3</td><td>22.1</td><td>22.3</td><td>-0.2</td></tr><tr><td>ElevatorAction</td><td>75946.0</td><td>76118.0</td><td>0.0</td><td>67752.0</td><td>75606.0</td><td>81142.0</td></tr><tr><td>Enduro</td><td>2153.0</td><td>2175.9</td><td>2184.7</td><td>2170.9</td><td>31.6</td><td>2263.7</td></tr><tr><td>FishingDerby</td><td>46.4</td><td>41.4</td><td>44.3</td><td>44.7</td><td>47.5</td><td>45.6</td></tr><tr><td>Freeway</td><td>33.6</td><td>33.7</td><td>33.6</td><td>33.6</td><td>33.6</td><td>33.9</td></tr><tr><td>Frostbite</td><td>8048.7</td><td>8133.9</td><td>8343.3</td><td>7859.6</td><td>6909.5</td><td>9012.5</td></tr><tr><td>Gopher</td><td>10709.6</td><td>9630.3</td><td>9844.3</td><td>10547.0</td><td>10551.9</td><td>13382.3</td></tr><tr><td>Gravitar</td><td>1999.3</td><td>1245.6</td><td>1442.2</td><td>1338.2</td><td>1332.6</td><td>1337.8</td></tr><tr><td>Hero</td><td>43671.9</td><td>47802.0</td><td>41244.6</td><td>44286.2</td><td>45974.3</td><td>40075.3</td></tr><tr><td>IceHockey</td><td>-1.0</td><td>-1.0</td><td>2.7</td><td>-0.9</td><td>-1.4</td><td>13.6</td></tr><tr><td>Jamesbond</td><td>721.6</td><td>839.5</td><td>1094.4</td><td>1168.6</td><td>935.1</td><td>828.4</td></tr><tr><td>JourneyEscape</td><td>-1185.4</td><td>-1215.0</td><td>-1992.6</td><td>-1112.0</td><td>-1199.6</td><td>-986.2</td></tr><tr><td>Kangaroo</td><td>12917.7</td><td>12606.1</td><td>12521.5</td><td>12611.1</td><td>12593.4</td><td>13981.5</td></tr><tr><td>Krull</td><td>4249.5</td><td>4202.2</td><td>4153.0</td><td>4075.4</td><td>4276.6</td><td>4371.8</td></tr><tr><td>KungFuMaster</td><td>28741.8</td><td>24087.6</td><td>26439.4</td><td>27234.6</td><td>28168.3</td><td>39382.1</td></tr><tr><td>MontezumaRevenge</td><td>780.0</td><td>5199.2</td><td>5028.2</td><td>4652.3</td><td>2795.9</td><td>2435.4</td></tr><tr><td>MsPacman</td><td>4063.2</td><td>4008.6</td><td>3964.5</td><td>3993.7</td><td>3861.0</td><td>5051.5</td></tr><tr><td>NameThisGame</td><td>9496.4</td><td>9202.3</td><td>8779.3</td><td>9147.4</td><td>9247.3</td><td>11499.7</td></tr><tr><td>Phoenix</td><td>6672.1</td><td>7318.6</td><td>4762.9</td><td>7671.8</td><td>8306.2</td><td>17579.5</td></tr><tr><td>Pitfall</td><td>-5.3</td><td>-1.8</td><td>-6.5</td><td>-11.4</td><td>-35.9</td><td>-15.7</td></tr><tr><td>Pong</td><td>20.3</td><td>20.1</td><td>20.2</td><td>20.3</td><td>19.9</td><td>19.9</td></tr><tr><td>Pooyan</td><td>4324.6</td><td>4353.1</td><td>4416.6</td><td>4382.3</td><td>4128.3</td><td>4974.4</td></tr><tr><td>PrivateEye</td><td>26038.5</td><td>19421.7</td><td>5276.8</td><td>12211.4</td><td>26363.8</td><td>4339.9</td></tr><tr><td>Qbert</td><td>17606.8</td><td>18369.7</td><td>17247.3</td><td>17293.6</td><td>18603.9</td><td>18785.1</td></tr><tr><td>Riverraid</td><td>20607.1</td><td>20941.0</td><td>20694.7</td><td>20699.5</td><td>20290.8</td><td>16250.6</td></tr><tr><td>RoadRunner</td><td>54381.9</td><td>53829.7</td><td>56352.5</td><td>54773.0</td><td>53630.3</td><td>62423.5</td></tr><tr><td>Robotank</td><td>64.7</td><td>66.1</td><td>34.3</td><td>60.3</td><td>61.0</td><td>70.0</td></tr><tr><td>Seaquest</td><td>16044.8</td><td>11821.2</td><td>27027.8</td><td>11185.7</td><td>15469.9</td><td>20898.3</td></tr><tr><td>Skiing</td><td colspan="3">-24988.0 -28678.5 -27801.4</td><td colspan="2">-29094.5 -28781.4</td><td>-29613.9</td></tr><tr><td>Solaris</td><td>1792.5</td><td>1684.3</td><td>1622.6</td><td>1252.0</td><td>1470.1</td><td>3024.6</td></tr><tr><td>SpaceInvaders</td><td>4172.0</td><td>3232.1</td><td>2229.4</td><td>4217.7</td><td>4936.0</td><td>9719.3</td></tr><tr><td>StarGunner</td><td>55112.9</td><td>57334.7</td><td>58005.6</td><td>56811.3</td><td>59500.1</td><td>63232.9</td></tr><tr><td>Tennis</td><td>-0.1</td><td>-0.6</td><td>23.0</td><td>23.2</td><td>2.6</td><td>-0.0</td></tr><tr><td>TimePilot</td><td>11640.6</td><td>12010.6</td><td>7430.4</td><td>11564.4</td><td>9210.8</td><td>13593.5</td></tr><tr><td>Tutankham</td><td>239.4</td><td>237.9</td><td>193.0</td><td>234.7</td><td>233.6</td><td>238.1</td></tr><tr><td>UpNDown</td><td>38975.8</td><td>22559.4</td><td>14958.5</td><td>21799.9</td><td>24751.1</td><td>43345.9</td></tr><tr><td>Venture</td><td>1591.7</td><td>1554.7</td><td>1356.3</td><td>1536.6</td><td>1555.8</td><td>1646.4</td></tr><tr><td>VideoPinball</td><td colspan="3">382581.4 447711.5 463181.6</td><td colspan="2">454007.7 432640.6</td><td>478029.4</td></tr><tr><td>WizardOfWor</td><td>7939.4</td><td>6365.5</td><td>5807.5</td><td>7330.5</td><td>7041.1</td><td>10980.4</td></tr><tr><td>YarsRevenge</td><td>44142.9</td><td>56794.0</td><td>43173.2</td><td>47749.1</td><td>45821.0</td><td>52570.1</td></tr><tr><td>Zaxxon</td><td>15101.4</td><td>14532.9</td><td>10559.9</td><td>12248.3</td><td>8451.7</td><td>19211.5</td></tr></table>

Peer review (iclr2020):
Updated review: I am overall happy with the response of the authors. I can appreciate the contributions of the paper and I am happy to recommend accept. The empirical study offers some insights into deep RL methods for ATARI games and raises some key questions. I feel the current version of the paper does not build upon these insights to propose a new method. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Summary: This paper presents a detailed empirical study of the recent bonus based exploration method on the Atari game suite. The paper concludes that methods that perform well on Montezuma’s revenge do not necessarily perform well on the other games, sometimes, even worse than the eps-greedy approach. This also leads to the conclusion that recent results on the game Montezuma’s revenge can be attributed to architectural changes instead of the exploration method. I think this is a-ok paper in that it does what it says it does. The paper is clear and well-written. I think the main contribution of the paper is that it raises some questions over existing methods/trends in solving exploration problems in reinforcement learning by comparing the performance of multiple methods across various games in ATARI suite. I think this is relevant to the ICLR community and will be appreciated by it. However, I also feel that while the paper runs a satisfactory empirical analysis, it was all too much focussed on the existing methods. Throughout the paper, the experiments and results raise questions on the robustness and generalization of existing exploration methods across various ATARI games, but the paper puts absolutely zero effort into investigating if there is a quick fix to the questions it poses. For example, one could easily investigate in the CTS method if the factor by which exploration bonus dies N^{alpha} (alpha=-1/2 by default) changes, then does it do better or worse (more below on this). I can understand that might not be the aim of the paper but still. Here are a couple of points that I felt conflicted/confused about the paper: - The conclusion of the paper is that ‘progress of exploration in ATARI suite is obfuscated by good results in single domain’. I am confused if the paper is making a narrow point that (1) dont focus on Montezuma’s revenge OR (2) is it admitting a broader point that focussing on even ATARI is probably not a good choice. I am not saying that I know the answer to this question, but I am unclear as to what is the question the paper is trying to raise. If it is saying (1st) then I find it contradictory that it is not ok to focus on MR but it is ok to focus on ATARI as a single domain; if it is saying the second then also it is contradictory because the paper only experiments with the ATARI suite. - It is interesting to note that noisy networks are most robust to hyperparameter optimization on a separate set of games when tested on a different set of games. It is also interesting to note that noisy networks are the only exploration bonus method that does not decrease/reduce exploration as the experience of the agent increases. I would have liked to see if the paper had made an attempt to investigate this. I feel such a hypothesis would have been easy to investigate with simple modifications to the CTS methods. Currently, the exploration bonus goes down by the factor of 1/sqrt(N) in the CTS method. A comparison that showed the performance of CTS for a couple more values of factors such as (1/N) or (1/N)^{1/4} would have been nice to see if that mattered. - One of the comparisons I did not particularly find fair was when the hyperparameters of various methods were tuned to play MR and then the hyperparameters were fixed and the method were tested on other ATARI games. - Another point I felt was missing was checking if rainbow DQN is really the reason behind the observed performance of the methods. It would have been interesting to know how the methods performed when combined with the original DQN algorithm.