Human-level Atari 200x faster
Authors: Steven Kapturowski
Víctor Campos
Ray Jiang
Nemanja Rakicevic
Hado van Hasselt
Charles Blundell
Adria Puigdomenech Badia
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Data-efficiency
Exploration
Off-policy

Abstract: The task of building general agents that perform well over a wide range of tasks has been an important goal in reinforcement learning since its inception. The problem has been subject of research of a large body of work, with performance frequently measured by observing scores over the wide range of environments contained in the Atari 57 benchmark. Agent57 was the first agent to surpass the human benchmark on all 57 games, but this came at the cost of poor data-efficiency, requiring nearly 80 billion frames of experience to achieve. Taking Agent57 as a starting point, we employ a diverse set of strategies to achieve a 200-fold reduction of experience needed to outperform the human baseline, within our novel agent MEME. We investigate a range of instabilities and bottlenecks we encountered while reducing the data regime, and propose effective solutions to build a more robust and efficient agent. We also demonstrate competitive performance with high-performing methods such as Muesli and MuZero. Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under a rapidly-changing policy.

Published as a conference paper at ICLR 2023
HUMAN-LEVEL ATARI 200X FASTER
Steven Kapturowski, Victor Campos", Ray Jiang", Nemanja Rakićević, Hado van Hasselt, Charles Blundell, Adrià Puigdomènech Badia DeepMind, *Equal contribution
(skapturowski, canunez, rayjiang, cblundell, adriap haap::deeepppe @deeppind. com com
hado,
ABSTRACT
The task of building general agents that perform well over a wide range of tasks has been an important goal reinforcement learning since its inception. The problem has been subject of research of a large body of work, with performance frequently measured by observing scores over the wide range of environments contained in the Atari 57 benchmark. Agent57 was the first agent to surpass the human benchmark on all 57 games, but this came at the cost of poor data-efficiency, requiring nearly 80 billion frames of experience to achieve. Taking Agent57 as starting point, we employ diverse set of strategies to achieve a 200-fold reduction of experience needed for all games to outperform the human baseline within novel agent MEME. We investigate a range of instabilities and bottlenecks we encountered while reducing the data regime, and propose effective solutions to build robust and efficient agent. We also demonstrate competitive performance with nigh-performing methods such Muesli and MuZero. Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under rapidly-changing policy.
INTRODUCTION
To develop generally capable agents, the question how to evaluate them is paramount. The Arcade I Learning Environment (ALE) (Bellemare et al., 2013) was introduced as benchmark to evaluate agents on an diverse set of tasks which are interesting to humans, and developed externally to the Reinforcement Learning (RI.) community. As a result, several games exhibit reward structures which are highly adversarial to many popular algorithms. Mean and median human normalized scores (HNS) (Mnih et al. 2015) over all games in the ALE have become standard metrics for evaluating deep RL agents. Recent progress has allowed state-of-the-as algorithms to greatly exceed average human-level performace on large fraction of the games (Van Hasselt et al. 2016 Espeholt et al.) 2018 Sc.rischhittweseer et al. 2020). However, it has been argued that mean or median HNS might not be well suited assess generality because they tend to ignore the tails of the distribution (Badia et al.) 2019). Indeed, most state-of-the-ar algorithms achieve very high scores by performing very well on most games, but completely fail learn small number of them.
Agent57 (Badia et al. 2020) was the first algorithm to obtain above human-average scores on all 57 Atari games. However, such generality came at the cost of data efficiency; requiring tens of billions of environment interactions to achieve above average-humen human performance in some games, reaching figure of 78 billion frames before beating the human benchmark in all games. Data efficiency remains a desirable property for agents to possess, as many real-world challenges are data-limited by time and cost constraints (Dulac-Aroold al. 2019). In this work, we develop an agent that is as general as Agent57 but that requires only a fraction of the environment interactions to achieve the same result.
There exist two main trends in the literature when it comes to measuring improvements in the learning capabilities of agents. One approach consists in measuring performance after limited budget of interactions with the environment. While this type of evaluation has led to important progress (Espeholt et al. 20188 van Hasselt et al.] 2019 Hessel et al.; 2021), it tends to disregard problems which are considered too hard to be solved withi the allowed budget (Kasser et al. 2019). On the other hand, one aim to achieve target end-performance with as few interactions as

---

Published as a conference paper at ICLR 2023
Frames > human score
1:ce
140
Agent57
MEME
10
390M
200M
160M
JN
Prave
Solert
Broking
Grantarr
Tome
Teo she
Rere C
poro
Beam tale
Cespa Cammi contraant
221
bonoor mesre Co Now
mrree poni puld
Ugi Done
Botre
Raboran
Kirp trox Wodoo
Rod Sarne
Oue Onber
Mall
Spacts
Figure 1: Number of environment frames required by agents to outperform the human baseline on each game (in log-scale). Lower is better. On average, MEME achieves above human scores using 62x fewer environment interactions than Agent57. The smallest improvement is 10x (Road Runner), the maximum is 734x (Skingg), and the median across the suite is 36x. We observe small variance across seeds (c.f. Figure 8
possible (Silver et al. is as general as Agent57, 7, while being more data-efficient, we focus on the latter approach.
2017 2018 Schmitt al. 2020). Since our goal is show that our new agent
be summarized follows. Building off Agent57, bottlenecks which slow down learning and address instabilities that arise when these bottlenecks are removed. We propose novel agent that call MEME, for MEME is an Efficient Memory-based Exploration agent, which introduces solutions to enable taking advantage three approaches that would otherwise lead to instabilities: training the value functions of the whole family of policies from Agent57 in parallel, on all policies' transitions (instead of just the behaviour policy transitions), bootstrapping from the online network, and using high replay ratios. These solutions include carefully normalising value functions with differing scales, as well as replacing the Retrace update target (Munos al. 2016) with soft variant of Watkins' Q(A) (Watkins & Dayan 1992) that enables faster signal propagation by performing less aggressive trace-cutting, and introducing trust-region for value updates. Moreover, we explore several recent advances in deep learning and determine which of them are beneficial for non-stationary problems like the ones considered in this work. Finally, we examine approaches to robustify performance by introducing a policy distillation mechanism that learns a policy head based on the actions obtained from the value network without being sensitive to value magnitudes. Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than Agent57 as shown in Figure
Our contributions
carefully examine
2 RELATED WORK
Large scale distributed agents have exhibited compelling results in recent years. Actor-critic (Espeholt al., 2018 Song et al. 2020) as well value-based agents (Horgan et al.) 2018 Kapturowski et al. 2018) demonstrated strong performance in a wide-range of environments. including the Atari 57 benchmark. Moreover, approaches such as evolutionary strategies (Salimans al. 2017) and large scale genetic algorithms (Such et al., (2017) presented alternative learning algorithms that achieve competitive results on Atari Finally, search-augmented distributed agents (Schrittwieser et al. 2020
Hessel et al. 2)21) also hold high performance across many different tasks, and concretely they hold the highest mean and median human normalized scores over the 57 Atari games. However, all these methods show the same failure mode: they perform poorly in hard exploration games, such as Pitfall!, and Montezuma' Revenge. In contrast, Agent57 (Badia et al. 2020) surpassed the human benchmark on all 57 games, showing better general performance. Go-Explort (Ecoffet et al. (021) similarly achieved such general performance, by relying on coarse-grained state representations via a downscaling function that is highly specific Atari.
Learning as much as possible from previous experience is key for data efficiency. Since it is often desirable for approximate methods to make small updates to the policy (Kakade & Langford 2002) Schulman et al. 2015), approaches have been proposed for enabling multiple learning steps over the same batch of experience in policy gradient methods to avoid collecting new transitions for
2

---

Published as a conference paper at ICLR 2023
every learning step (Schulman et al.,
2017). This decoupling between collecting experience and
learning occurs naturally in off policy learning agents with experience replay (Lin 1992 Mnih et al. 2013) and Fitted Q Iteration methods (Ernst al. 2005 Riedmiller] 2005). Multiple approaches for making more efficient use of a replay buffer have been proposed, including prioritized sampling of
transitions (Schaul et al. 20T6), sharing experience across populations of agents (Schmitt et al. 2020), learning multiple policies in parallel from a single stream of experience (Riedmiller et al. [20|20011, 8).. or reanalyzing old trajectories with the most recent version of a learned model generate new targets in model-basee settings (Schrittwieser et al. 2017
2020
202T
or to re-evaluate goals (Andrychowicz et al.
The ATARII0OK benchmark (Kaiser et al. 2019) was introduced to observe progress in improving
the data efficiency of reinforcement learning agents, by evaluating game scores after 100k agent steps (400k frames). Work on this benchmark has focused on leveraging the use of models Yee et al. [021] Kaiser et al. 2019: Long et al.) 2022), unsupervised learning (Hansen et al.) 2019) [Schwarzer et al.
2021:
Srinivas al. 2020 Liu & Abbeel 202I). or greater use of replay data (van Hasselt et al.
2019 Kielakj 2020) or augmentations (Kostrikov et al.] 2021 Schwarzer et al.) 2021). While we
consider this to be an important line of research, this tight budget produces incentive to focus on a subset of games where exploration is easier, and is unclear some games can be solved from
scratch with such small data budget. Such a setting is likely to prevent any meaningful learning on hard-exploration games, which is in contrast with the goal of our work.
3 BACKGROUND: AGENT57
Our work builds on top of Agent57, which combines three main ideas: (i) a distributed deep RL framework based on Recurrent Replay Distributed DQN (R2D2) (Kapturowski et al.] [2018). (ii) exploration with a family of policies and the Never Give Up (NGU) intrinsic reward (Badia ct al. 2019). and (iii) a meta-controller that dynamically adjusts the discount factor and balances exploration and exploitation during training, by selecting from a family of policies. Below, we give general introduction to the problem setting and some of the relevant components of Agent57.
Problem definition. We consider the problem of discounted infinite-horizon RL in Markov Decision Processes (MDP) (Puterman 1994). The goal is to find a policy π that maximises the expected sum of discounted future rewards, E,, L.200 7're), where 7 E [0, 1) is the discount factor, rt r(ra, 0z) the reward at time t, x, the state at time I, and a, ~ I Taa xx)) is the action generated by following some policy π. In the off- aff-cy learning setting, data generated by behavior policy μ is used to learn about the target policy и. This can be achieved by employing a variant of Q-learning (Watkins Dayan 1992) to estimate the action-value function, Q" (ar, a) = Ex [Er2 7 [a+ Ir ag = a.. The estimated action-value function can then be used to derive a new policy T(a|x) using the e-greedy operator G. (Sutton & Barto 2018) 8 This new policy can then be used as the target policy for another iteration, repeating the process. Agent57 uses deep neural network with parameters @ to estimate action-value functions, Q" (zt a; 0) trained on return estimates G, derived with Retrace from sequences of off- policy data (Munos al. 2016). In order to stabilize learning, target network is used for bootstrapping the return estimates using double Q-learning (Van Hasselt et al. (0016); the parameters of this target network, 0r, are periodically copied from the online network parameters 0 (Mnih et al. 2015). Finally, a value-function transformation is used to compress the wide range of reward scales present in Atari, as in (Pohlen et al. 2018).
Distributed RL framework. Agent57 is distributed deep RL agent based on R2D2 that decouples acting from learning. Multiple actors interact with independent copies of the environment and feed trajectories to central replay buffer. A separate learning process obtains trajectories from this buffer using prioritized sampling and updates the neural network parameters to predict action-values at each state. Actors obtain parameters from the learner periodically. See Appendix for more details.
Exploration with NGU. Agent57 uses the Never Give Up (NGU) intrinsic reward to encourage exploration. It aims at learning family of N = 32 policies which maximize different weightings of the extrinsic reward given by the environment (rf) and the intrinsic reward (rf), TG = rf + B,r (8, IR j E {0, ... N....,,N-1. 1)). The value of By controls the degree of exploration. with higher values
We also use G = Go to denote the pure greedy operator (E = 0). Forr convenience, we occasionally omit (zt, a) or @ from Q(z,, a; f), (aa on; 8) when it is unambiguous.
3

---

Published as a conference paper at ICLR 2023
Qn a;
_'(a,
Q-valuu
(k2 9;
Head
Torso
Policy
xk,,1x)
policy family members
Figure 2: MEME agent network architecture. The output of the LSTM block is passed to each of the N members of the family of policies, depicted light-grey box. Each policy consists of an Q-value and policy heads. The Q-value head similar as in Agent57 paper, while the policy head is introduced for acting and target computation, and trained via policy distillation.
encouraging exploratory behaviors, and each policy in the family is optimized with different discount factor 75- The Universal Value Function Approximators (UVFA) framework (Schaul et al.
2015) is employed to efficiently learn QT' (xx, a: Q) E, 1Ez0 7frzalze 2, a, = a] (we use shorthand notation Q' a; 0)) for j E {O,..., ., N 1} using a single set shared parameters The policy x (alar) can then be derived using the e-greedy operator G G2O' a; 0). We refer the reader to Appendix H for more details.
Meta-controller. Agent57 introduces adaptive meta-controller that decides which policies from the family of policies to use for collecting experience based on their episodic returns. This naturally creates curriculum over 3j and %/ by adapting their value throughout training. The optimization process is formulated as a non-stationary bandit problem. A detailed description about the meta-controller implementation is provided in Appendix
Q-function separation. The architecture of the Q-function in Agent57 is implemented as two separate networks in order to split the intrinsic and extrinsic components. The network parameters of Q' (zz a; 2 and Q" (z, a; ( are separate and independently optimized with rī and rj. respectively. The main motivation behind this decomposition is allow each network to adapt to the scale and variance associated with their corresponding reward, as well as preventing the gradients of the decomposed intrinsic and extrinsic value function heads from interfering with each other.
МЕME: IMPROVING THE DATA EFFICIENCY OF AGENT57
This section describes the main algorithmic contributions of the MEME agent, aimed at improving the data efficiency of Agent57. These contributions aim to achieve faster propagation of learning signals related to rare events stabilize learning under differing value scales B improve the neural For
network architecture
and make updates more robust under a rapidly-changee policy clarity of exposition, label the contributions according the type of limitation they address.
Bootstrapping with online network. Target networks are frequently used in conjunction with value-based agents due to their stabilizing effect when learning from off off- policy data (Mnih et al. 2015;|Van Van Hasselt et al.) 2016). This design choice places fundamental restriction on how quickly changes in the Q-function are able to propagate. This issue can be mitigated to some extent by simply updating the target network more frequently, but the result typically a less stable agent. To accelerate signal propagation while maintaining stability, we use online network bootstrapping, and we stabilize the learning by introducing an approximate trust region for value updates that allows us to filter which samples contribute to the loss. The trust region masks out the loss at any timestep for which both of the following conditions hold:
1Q a,; 0) Q" (21,, 0r)) ad;
sgn(QQ (Q' (Ir, oz; 0) Q' u4; 4,; 0T)) )) # (gn(Q"( ax; G.)
where a is a fixed hyperparameter, G, denotes the return estimate, 0 and 0rr denote the online and target parameters respectively, and o, the standard deviation of the TD-errors (a more precise description of which we defer until B1 Intuitively, we only mask the current value of the online network is outside of the trust region (Equation and the sign of the TD-error points away
4

---

Published as a conference paper at ICLR 2023
from the trust region (Equation trust region scheme is used for the value-function in most Proximal Policy Optimization (PPO) implementations (Schulman et al. 2017), though not described in the original paper. In contrast, the PPO version instead uses constant threshold, and thus is not able to adapt to differing scales of value functions.
depicted in Figure in red. We note that very similar
@(x,a,00)
(,,,
(xx,),0,))
Figure 3: Trust region. The position of dots is given by the relationship between the values predicted by the online network, Q' (æt, ag; 9), and the values predicted by the target network, Q' (2a, a,; 0r) (Equation T and left hand side of Equation 2). the box represents the trust region bounds defined in Equation and the direction of the arrow is given by the right hand side of Equation 2 Green-colored transitions used in the loss computation, whereas red ones masked out.
Target computation with tolerance. Agent57 uses Retrace (Munos et al. 2016 to compute return estimates from off-policy data, but we observed that it tends to cut traces too aggressively when using e-greedy policies thus slowing down the propagation of information into the value function. Preliminary experiments showed that data efficiency
improved in many dense-reward tasks when replacing Retrace with Peng's Q(A) (Peng & Williams 1994), but its lack of off-policy corrections tends result in degraded performance expected number of times that sequence is sampled from replay, or by sharing data across family of policies). This motivates us propose an alternative return estimator, which we derive from Q(A) Watkins & Dayan
data becomes more off- off-policy (e.g. by increasing the
1992):
G, max a)), +
Σ
II 1i=0
7 + max (1 max ( (3)
where IT A; E (0, 1] effectively controls how much information from the future is used in the return estimation and generally used as a trace cutting coefficient perform off off-policy correction. Peng's Q(A) does not perform any kind of off- off-policy correction and sets X; = X, whereas Watkins' Q(A) (Watkins & Dayan 1992) aggressively cuts traces whenever encounters an off-policy action by using X; XI, Eargmax, a)) where denotes the indicator function. We propose to use softer trace cutting mechanism by adding a fixed tolerance parameter x and taking the expectation of trace coefficients under
\Ea-=(()x))
[1Q(
(4)
Finally, we replace all occurrences of the max operator in Equation with the expectation under The resulting return estimator, which we denote Soft Warkins Q(A), leads to more transitions being used and increased sample efficiency. Note that Watkins Q(A) recovered when setting x. 0 and
π = G(Q).
Loss and priority normalization. As we learn family of Q-functions which vary over wide range of discount factors and intrinsic reward scales, we expect that the Q-functions will vary considerably in scale. This may cause the larger-scale Q-values to dominate learning and destabilize learning of smaller Q-values. This a particular concern in environments with very small extrinsic reward scales. To counteract this effect we introduce a normalization scheme the TD-errors similar to that used in Schaul et al. (2021). Specifically, we compute a running estimate of the standard deviation of TD-errors of the online network of
B1
running as well as a batch standard deviation a baachh and
compute oj max(o running o c), where e acts as small threshold to avoid amplification of noise past specified scale, which fix to 0.01 in all our experiments. We then divide the TD-errors by o, when computing both the loss and priorities. As opposed to [Schaul et al. (202T) we compute the running statistics the learner, and importance sampling to correct the sampling distribution.
5

---

Published as a conference paper at ICLR 2023
Cross-mixture training. Agent57 only trains the policy j which was used to collect given trajectory, but it is natural to ask whether data efficiency and robustness may be improved by training all policies at once. We propose training loss L according to the following weighting scheme between the behavior policy loss and the mean over all losses:
B2
1-7 N-1 N
= n +
Σ
L
(5)
j=0
where Lj denotes the Q-learning loss for policy j. and jn denotes the index for the behavior policy selected by the meta-controller for the sampled trajectory. We find that an intermediate value for the mixture parameter of 7 = 0.5 tends to work well. To achieve better compute-effciieccy we choose to deviate from the original UVFA architecture which fed a 1-hot encoding of the policy index to the LSTM, and instead modify the Q-value heads to output N sets of Q-values, one for each of the members in the family of policies introduced in Section Therefore, in the end output values for all combinations of actions and policies (see Figure 2). We note that in this setting, there is also less deviation in the recurrent states when learning across different mixture policies.
Normalizer-free torso network. Normalization layers are common feature of ResNet architec- tures, and which are known to aid in training of very deep networks, but preliminary investigation revealed that several commonly used normalization layers are in fact detrimental to performance in our setting. Instead, we employ a variation of the NENet architecture (Brock et al., 2021) for our policy torso network, which combines a variance-scaling strategy with scaled weight standardization and careful initialization to achieve state-of-the-an performance on ImageNet without the need for normalization layers. We adopt their use of stochastic depth (Huang et al., (016) at training- time but omit the application of ordinary dropout to dally-connected layers as we did not observe any benefit from this form of regularization. Some care is required when using stochastic depth in conjunction with multi-step returns, as resampling of the stochastic depth mask at each timestep injects additional noise into the bootstrap values, resulting in a higher-varians return estimator. As such, we employ a cemporally-consisee stochastic depth mask which remains fixed over the length of each training trajectory.
C2 Shared torso with combined loss. Agent57 decomposes the combined Q-function into intrinsic and extrinsic components, Qe and Q;, which are represented by separate networks. Such a decompo- sition prevents the gradients of the decomposed value functions from interfering with each other. This interference may occur in environments where the intrinsic reward is poorly aligned with the task objective, as defined by the extrinsie reward. However, the choice to use separate separate networks comes at an expensive computational cost, and potentially limits cample--fficiency since generic low-level features cannot be shared. To alleviate these issues, we introduce shared torso for the two Q-functions while retaining separate heads.
While the form of the decomposition in Agent57 was chosen as ensure convergence the optimal value-function Q' in the tabular setting, this does not generally hold under function approximation. Comparing the combined and decomposed losses we observe mismatch in the gradients due the
absence of cross-terms (0)
(Q_(.) and (0) (Q) in the decomposed loss:
80
00
((110) 2)2
+ 8 (()((0)) - G.) #BQ0 3C,)
(6)
combined loss
decomposed loss
[Q,(0)+8Q de [Q.(0)+Q(((0] # [Q.(0)-G.] aQ.(0) 3 (Q: (Q:(0) 00
(7)
00
Since we use a behavior policy induced by the total Q-function Q = Q. + 3Q; rather than the individual components, theory would suggest use the combined loss instead. In addition, from practical implementation perspective, this switch to the combined loss greatly simplifies the design choices involved in our proposed trust region method described in A1 The penalty paid for this choice is that the decomposition of the value function into extrinsic and intrinsic components no longer carries a strict semantic meaning. Nevertheless we do still retain implicit inductive bias induced by multiplication of Q, with the intrinsic reward weight 33,
Robustifying behavior via policy distillation. Schaul et al. (2022) describe the effect of policy churn, whereby the greedy action of value-based RL. algorithms may change frequently over

---

Published as a conference paper at ICLR 2023
consecutive parameter updates. This can have a deleterious effect on off-po correction methods: traces will be cut more aggressively than with a stochastic policy, and bootstrap values will change frequently which can result in higher variance return estimator. In addition, our choice of training stochastic depth masks can be interpreted as learning an implicit ensemble
with temporally-consisee of Q-functions; thus it is natural ask whether we may see additional benefit from leveraging the policy induced by this ensemble.
We propose to train an explicit policy head Waist (see Figure 2) via policy distillation match the e-greedy policy induced by the Q-function. In expectation over multiple gradient steps this should help to smooth out the policy over time, as well as over the ensemble, while being much faster to evaluate than the individual members of the ensemble. Similarly to the trust-region described in Al we mask the policy distillation loss at any timestep where a KL constraint CKL is violated:
L = Ee a; 0) log Tans(a/ v' s.t. KL (Tiss (a| 0r)) Tasa(aaaa 0)) S CKL a,
(8)
We use a fixed value of e = 10-1 to act as a simple regularizer to prevent the policy logits from growing too large. We then use "'igt = softmax( log Raint) ) as the target policy in our Soft Warkins Q(A) return estimator, where is fixed temperature parameter. We tested values of in the range [0, 1] and found that any choice of in this range yields improvement compared to not using the distilled policy, but values closer tend exhibit greater stability, while those closer to 0 tend to learn more efficiently. We settled on an intermediate T 0.25 to help balance between these two effects. We found that sampling from d'ist at behavior-time was less effective in some sparse reward environments, and instead opt to have the agent act according Ge (Taist)..
5 EXPERIMENTS
Methods constituting MEME proposed in Section 4 aim at improving the data efficiency of Agent57, but such efficiency gains must not come at the cost of end performance. For this reason, we train our agent with budget of 1B environment fr This budget allows asymptotic performance is maintained, i.c. the agent converges and is stable when improving data efficiency. Hyperparameters have been tuned over a subset of eight games, encompassing games with different reward density, scale and requiring credit assignment over different time horizons: Frostbite, H.E.R. O., Montezuma's 's Revenge, Pirfall!, Skiing, Solaris, Surround, and Tennis. Specifically, we selected Montezuma's 's Revenge, Pitfall!, and H.E.R.O. O. as these are games exhibiting very sparse and large scale reward signal, requiring long-term credit assignment, while being partially observable. Additionally, Solaris and Sküng also require long-term credit assignment, but have medium scale and large negative scale rewards, respectively. Having a high degree of randomness in the observations is also property of Solaris which our agents should be robust to. Finally, Surround and Tennis can be considered as fully-observable and having small-scale, moderately sparse rewards. In particular, due to the improvements in stability, find it beneficial use higher samples per insert ratio (SPI) than the published values of Agent57 or R2D2. For all our experiments SPI is fixed to 6. An ablation An exhaustive description of the hyperparameters used is provided in Appendix A and the network architecture in Appendix B We report results averaged over six seeds per game for all experiments. We also show similar results with sticky actions (Machado et al. 2018) in Appendix K] Metrics for previous methods are computed using the mean final score per game reported in their respective publications: Agent57 (Badia et al., 2020), MuZero 20B (Schritwwisser et al.] (020). MuZero 200M (Schritwieser et al.. 2021). Muesli (Hessel et al.] (021). ). Data efficiency comparisons with Agent57 are based on the results reported by Badia al. (2020).
validate that the
on the effect of different SPI values can be found in Appendix L
5.1 SUMMARY OF RESULTS
In this section, we show that the proposed MEME agent is able achieve 200-fold reduction in the number of frames required for all games to surpass the human benchmark. The last game to surpass
This corresponds to 250M agent-environment interactions due to the standard action repeat of 4 in the Atari henchmark.
7

---

Published as a conference paper at ICLR 2023
Table I: Number of games above human, capped mean, mean and median human normalized scores for the 57 Atari games. Similarly to Badia et al. (2020), we first compute the final score per game by averaging results from all random seeds, and then aggregate the scores of all 57 games. We sample three out of the six seeds per game without replacement and report the average as well as 95% confidence intervals over 10,000 repetitions of this process.
200M frames
200M frames
Statistic
MEME
Muesli
MuZero
MEME
Agent57
MuZero
Env frames
200M
200M
200M
IB
90B
20B
Number of games human
54 (63,56)
52
49
57 (57.67)
57
51
Capped mean
98 97.)
92
89
100 100)
100
87
Mean
3305 3346)
2523
2856
4087(3772..0)
4766
4998
Median
848 (229,805)
1077
1006
(185 (1005,11388 1325;
1933
2041
25th percentile
282 203)
269
153
478 429 515)
387
276
5th percentile
100 (80)
15
28
119 (119119,12))
116
0
human score with MEME is Pitfall! at 390M frames, as compared to 78B frames required for Skiing, the last game for Agent57.
Figure [ gives further details per game, about the required number of frames to reach the human baseline, and shows that hard exploration games such as Private Eye, Pitfall! and Montezuma's 's Revenge pose the hardest challenge for both agents, being among the last ones in which the human baseline is surpassed. This can be explained by the fact that in these games the agent might need very large number of episodes before it is able to generate useful experience from which to learn. Notably, our agent is able reduce the number of interactions required to outperform the human baseline in each of the 57 Atari games, by 62x on average. In addition to the improved sample- efficiency, MEME shows competitive performance compared the sate-of-theeaus methods shown in Table | while being trained for only 1B frames. When comparing with state-of- oftte-of-the-eee ff-the--rt agents such as MuZero (Schrittwieser et al. 2020) or Muesli (Hessel et al. 202T), we observe a similar pattern to that reported by Badia et al. (2020): they achieve very high scores in most games, as denoted by their high mean and median scores, but struggle to learn completely on a few of them (Agarwal et al. 2) It is important to note that our agent achieves the human benchmark in all games, as demonstrated by its higher scores on the lower percentiles.
5.2 ABLATIONS
We analyze the contribution of all the components introduced in Section 4 throuugh ablation experi- ments on the same subset of eight games. We compare methods based on the area under the score curve in order capture end performance and data efficiency under single metriq" The magnitude of this quantity varies across games, so we normalize it by its maximum value per game in order to obtain a score between 0 and 1, and report results on the eight ablation games in Figure 4 Appendix includes full learning curves for all ablation experiments as well as ablations of other design choices (e.g. the number of times each sample is replayed on average, optimizer hyperparameters).
Results in Figure 4 demonstrate that all proposed methods play role in the performance and stability of the agent. Disabling the trust region and using the target network for bootstrapping (Al produces the most important performance drop among all ablations, likely due to the target network being slower at propagating information into the targets. Interestingly, we have observed that the trust region is beneficial even when using target networks for bootstrapping (c.f. Figure 24 in Appendix NJ. which suggests that the trust region may produce additional stabilizing effect beyond what target networks alone can provide. Besides having similar stabilizing effect, policy distillation ( (D) also speeds up convergence on some games and has less tendency to converge to local optima on some others. The Soft Watkins' Q(A) loss boosts data efficiency especially in games with sparse rewards and requiring long-term credit assignment, and we have empirically verified that
'We first compute scores at 10,000 equally spaced points in [0, 1B] frames by applying piecewise linear interpolation to the scores logged during training. After averaging the scores at each of these points over all random seeds seeds, the trapezoidal rule is used compute the area under the averaged curve.
8

---

Published as a conference paper at ICLR 2023
Montezuma's Revenge
Pitfall!
Frostbite
MEME
Combined loss (C2)
Normalization (81)
Soft Watkins' Q(A! (A2)
Tennis
H.E.R.O.
Cress Cress-mixture training (B2)
NFNet torse (C1)
Policy distillation (D)
Trust: Trust-region Online boostrapping (A1)
Surround
Skiing
Solarts
Figure 4: Results of ablating individual components. For each experiment, we first average the score over three seeds up to IB frames, then compute the area under the score curve as it captures not only final performance but also the amount of interaction required achieve As absolute values vary across games, report relative quantities by dividing the maximum value obtained in each game.
it uses longer traces than other losses (c.f. Figure 5 in Appendix C Cross-mixture training and the combined loss tend to provide efficiency gains across most games. Finally, while we observe overall gains in performance from using normalization of TD errors (BL), the effect less pronounced than that of other improvements. We hypothesize that the normalization has high degree of overlap with other regularizing enhancements, such as including the trust region.
B2
We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when using the R2D2 agent (Kapturowski et al., 2018), which we present in Appendix M
6 DISCUSSION
We present novel agent MEME that outperforms the human-level baseline in data-efficient manner on all 57 Atari games. Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than Agent57, which leads to 62x speed-up on average. As Atari games are played at 60 FPS, this translates to approximately 75 days of training, compared to more than 41 years of gameplay required by Agent57.
To achieve this, the agent employs a set of improvements that address the issues apparent in the previous state-of-the art methods. Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under rapidly-changing policy. We ablation experiments to evaluate the contribution of each of the algorithmic and network improvements. Introducing online network bootstrapping with trust-region has the most impact the performance among these changes, while certain games require multiple improvements to maintain stability and performance. The increased stability enables more aggressive optimization, e.g. through higher samples per insert ratios or centralized bandit that aggregates statistics from all actors, leading to more data-efficient learning.
Although our agent achieves above average-human performance on all 57 Atari games within 390M frames with the same set of hyperparameters, the agent separately trained on each game. An interesting research direction to pursue would be to devise a training scheme such that the agent with the same set of weights can achieve similar performance and data efficiency as the proposed agent on all games. Furthermore, the improvements that propose do not necessarily only apply to Agent57, and further study could analyze the impact these components other state-of-the-att agents. We also expect that the generality of the agent could be expanded further. While this work focuses Atari due to its wide range of tasks and reward structures, future research is required to analyze the ability of the agent to tackle other important challenges, such more complex observation spaces modlll inputs), complex action spaces, or longer-temn credit assignment.
(e.g. 3D navigation, All such improvements would lead MEME towards achieving greater generality.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank Tom Schaul for his excellent feedback on improving this manuscript, and Pablo Sprechmann, Alex Vitvitskyi, Alaa Saade, Danicle Calandriello, Jake Bruce, Will Dabney, Mark Rowland, Bilal Piot, and Daniel Guo for helpful discussions throughout the development of this work.
REPRODUCIBILITY STATEMENT
In this manuscript we made additional efforts to make sure that the explanations of the proposed methods are detailed enough to be easily reproduced by the community. Regarding the training
setup, we detail the distributed setting and the computation resources used in Appendices E and G respectively. The optimiser details are presented in Appendix [ while the comprehensive list of hyperparameters used is given in Appendix A We explain the implementation details of the neural network architecture in Appendix B with the help of Figure Finally, we give additional details about the components such as the bandit (Appendix F). the NGU and RND intrinsic rewards (Appendix H target computation (Appendix C. and further clarify the proposed trust region with Figure
ETHICS STATEMENT
We would like note that large scale RL experiments require significant amount of compute, as outlined in the Appendix G In some cases the results obtained may not justify the incurred compute and environmental costs. However, to this end, our paper greatly improves the sample-efficinne of such RL methods, and we expect its development cost and environmental impact to be amortized over many subsequent applications.
We hope that in the future, researchers will leverage the contributions presented in this paper in their work on large-scale RI., and help reduce the environmental impact their research might have.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NewrIPS, 2021.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, NeurIPS, 2017.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 2002.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In ICLR, 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In ICML, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. JAIR, 2013.
Andy Brock, Soham De, Samuel Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In ICML, 2021.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Albin Cassirer, Gabriel Marth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: a framework for experience replay. arXiv preprint arXiv:2102.04736, 2021.
10

---

Published as a conference paper at ICLR 2023
Gabriel Dulac-Arnole Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. ICML, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 2021.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. JMLR, 2005.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, 2018.
Aurelien Garivier and Eric Moulines, On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805. 34155, 2008.
Aurélien Garivier and Eric Moulines. On upper-contidence bound policies for switching bandit problems. In ALT, 2011.
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. ICLR, 2019.
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theo- phane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In ICMI., 2021.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. In ICLR, 2018
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Lukasz Kaiser, Mohammad Babacizadch, Piotr Mitos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. ICLR, 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICMI., 2002.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In ICLR, 2018.
Kacper Piotr Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency?, 2020. URL https::/ wttp://wwwww.eeeeoueuet.m.
Bke9ulHFwB
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2021.
Long-Ji Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In NeurlPS, 2021.
Alexander Long, Alan Blair, and Herke van Hoof. Fast and data efficient reinforcement learning from pixels via non-parametric value approximation. In AAAI, 2022.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling, Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. JAIR, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller. Andreas Fidjeland, Georg Ostrovski, et al. Human-lev control through deep reinforcement learning. Nature, 2015.

---

Published as a conference paper at ICLR 2023
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- off-policy reinforcement learning. In NeurIPS, 2016.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. NeurIPS, 2016.
Jing Peng and Ronald J Williams. Incremental multi-step q-learning. JCML. 1994.
Alexandre Piche, Valentin Thomas, Joseph Marino, Gian Maria Marconi, Chris J. Pal, and Mo- hammad Emitiyaz Khan. Beyond target networks: Improving deep -learning with functional regularization. arXiv, 2021.
Tobias Pohlen, Bilal Piot, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth- Maron, Hado van Hasselt, John Quan, Mel Vecerik, Matteo Hessel, Rémi Munos, and Olivier Pictquin. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: 15593,
1593, 2018.
Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.105200 2019.
Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforcement learning method. In ECML, 2005.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint
2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In ICML, 2015.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.
Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.0553 2021.
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn. arXiv preprint arXiv:2206.007300 2022.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off Off-policy actor-critic with shared experience replay. In ICML, 2020.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. In NeurIPS, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In /CML, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiy preprint arXiv: 170.007077 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm. Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2023
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017.
David Silver, Thomas Hubert, Julian Schrittieserr, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018.
Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In ICLR, 2020.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML, 2020.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth Stanley, and Jeff Clune. Deep ncuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint 171.v:171..06567, 2017.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI, 2016.
Hado P Hasselt, Matteo Hessel, and John Aslanides. When use parametric models in reinforcement learning? In NeurlPS, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. NeurlPS, 2021.
13

---

Published as a conference paper at ICLR 2023
A HYPER-PARAMETERS
Table Agent Hyper- paraee-arrmettes.
Parameter
Value
Mixtures
Bandit /
1.0
Bandit e
0.5
Max Discount
0,9997
Min Discount
0.97
Replay Period
80
Burn-in
0
Trace Length
160
AP Embedding Size
RND Scale
0.5
RND Clip Threshold
5.0
IM Reward Scale 9IM
0.1
Bad
2.0
KL CKL
0.5
Cmst-Mixture
0.5
Talee Softmax Temperature
0.25
Soft Watkins Q(A) Threshold
A
0.95
Residual Drop Rate
0.25
Eval Parameter Decay Tooo
0,995
RND Stats Decuy ORRD
0,9999
TD Stats Decay oTp
0.9999
Priority Exponent
0.6
Importance Sampling Exponent
0.4
Max Priority Weight
0,9
Replay Ratio
6.0
Replay Capacity
2 10 trajectories
Value function rescaling
(v=*
0.001a
Batch
Adam
0.9
Adam 82
0.999
Adam e
10
Adam Learning Rate
10-1
AP Adam Learning Rate
6 x 10
RND Adam Learning Rate
6 x 10-4
RL Weight Decay
AP Weight Decny
0.05
RND Weight Decay
0.0
Gradient clipping percentile Pelpp
Gradient clipping decay Delip
0.999
Table 3: Environment Slyper-parameterss
Parameter
Value
Input Shape
210 160
Grayscaling
True
Action Repeat
4
Num Stacked Frames
Pooled Frames
Max Episode Length
108000 frames (30 minutes game time)
Life Loss Signal
Not used
B NETWORK ARCHITECTURE
B. TORSO
We modified version of the NENet architecture (Brock et al. (021). We use simplified stem, consisting of a single 7 x convolution with stride 4. We also forgo bottleneck residual blocks in favor of 2 layers of 3 convolutions, followed Squeeze-Excite block.
14

---

Published as a conference paper at ICLR 2023
In addition, make some minor modifications to the downsampling blocks. Specifically, apply an activation prior to the average pooling and multiply the output by the stride in order to maintain the variance of the activations. This is then followed by 3 convolution.
All convolutions use a Scaled Weight Standardization scheme (Qiao et al. 2019). The block section parameters follows:
num blocks: (2, 4)
num channels: (64, 128, 128, 64)
* strides: (1, 2, 2, 2)
B.2 NON-IMAGE FEATURES
We also feed the following features into the network:
previous action (encoded as size-32 embedding)
previous extrinsic reward
previous intrinsic reward
previous RND component of intrinsic reward
previous Episodic component of intrinsic reward
previous Action Prediction embedding
These are fed into single linear layer of size 256 and activation and then concatenated with the output of the torso and input into the recurrent core.
B.3 RECURRENT CORF
The recurrent core is composed of single LSTM with hidden size 1024. The output then concate- nated together with the core input and fed into the Action-Valee Value and Policy heads.
B.4 ACTION- VALUE AND POLICY HEADS
We utilize separate ducling heads for the intrinsic and extrinsic components of the value function, and simple MIP for the policy. All heads two hidden layers of size 1024 and output size num actions x num mixtures.
C TARGET COMPUTATION AND TRACE COEFFICIENTS
As motivated in Section we introduce Soft Watkins Q(A) as a trade-off between aggressive trace cutting used within Retrace and Watkins Q(A,, and the lack of off-policy correction in Peng's Q(A). We investigate this hypothesis by observing the average trace coefficients for each of the methods in Figure [ The lower values of the trace coefficient À lead to more aggressive trace cutting as soon as the data is off-policy. Conversely, the À value of 1 signifies no trace cutting whatsoever. As expected, Retrace's s trace coefficient significantly lower than the other methods considered. The proposed Soft Watkins Q()) has parameter that allows us to control the permissiveness of the trace-cutting which turn affects the final trace coefficient. We ran course sweep of this parameter over the values {0, 1, .01} and found values of 0.01 to generally perform best in setting. As can be seen from Figure this this choice considerably increases the average trace coefficient of Soft Watkins' Q()) relative to k 0, and concentrates the distribution of trace coefficients without causing it to collapse.
D OPTIMIZER
We use separate optimizers for each of the RL, Action Prediction, and RND Networks, with the only differences between them being the choice of learning rate and weight decay. The base optimizer
15

---

Published as a conference paper at ICLR 2023
Mean coeficcient
1.0
0.2
0.0
Retrace
Soft Watkins' Q(A) Soft Watkins' Q(A) Soft Watkins' Q(A) 0.1
Peng's Q(A)
=
0.01 (ours)
Figure 5: Average trace coefficients for each method on the of ablation games. For each method, we average the trace length for transitions generated between 200M and 250M frames, as we observe that their values tend to stabilize after an initial transient period. Each violin plot thus generated from num_seeds X num_games num_mixtures data points.
used AdamW with Nesterov Momentum. linear learning rate warmup used for the first 200 updates, to allow time for the Adam statistics to stabilize. Most notably, we employ an adaptive element- gradient clipping strategy whereby we maintain running estimates of the mean and standard deviation of the element-wise gradients using an Exponential Moving Average (EMA) with Aelip = 0.999, and only clip when the gradient magnitude exceeds the mean by some factor c of the standard deviation. Concretely, the factor c is determined by specifying percentile Pelip (that we fix to 0.99) which is then input to the inverse CDF a standard normal distribution. In general found this clipping strategy yield small but consistent improvement in stability across many environments compared to global norm clipping. We ablate some of the optimizer choices in Figure
19
E DISTRIBUTED SETTING
All experiments are run using a distributed setting. The experiment consists of the actors, learner, bandit and evaluators, as well as replay buffer.
The actors and evaluators are the two types of workers that draw samples from the environment. Since actors collect experience with non-greedy policies, we follow the common practice in this type of agent and report scores from separate evaluator processes that continually execute the greedy policy and whose experience is not added to the replay buffer (Kapturosski et al. 2018). Therefore, only the actor workers write to the replay buffer, while the evaluation workers are used purely for reporting the performance. The evaluation scheme differs from R2D2 (Kapturowski et al., (018)) in that separate set of eval parameters are maintained, which are computed as an EMA of the online network with Veeval 0.995; and these eval parameters continually updated throughout each episode. We observed that the use of these eval parameters provided a consistent performance boost across almost all environments, but we continue to the online network for the actors in order obtain more varied experience.
In the replay buffer, we store fixed-length sequences of (rf NGU Iz, xy,aa) az) tuples. These sequences never cross episode boundaries. For Atari, we apply the standard DQN pre- pre-processing, as used in R2D2. The replay buffer is split into 8 shards, to improve robustness due to computational constraints, with each shard maintaining an independent prioritisation of the entries. We use prioritized experience replay with the same prioritization scheme proposed by (Kapturowski et al. (2018) which used a weighted mixture of max and mean TD-errors over the sequence. Each of the actor workers writes to specific shard which is consistent throughout training. The replay buffer is implemented using Reverb (Cassirer et al. 2021)
Given a single batch of trajectories we unroll both online and target networks on the same sequence of states generate value estimates. These estimates are used to execute the learner update step, which updates the model weights used by the actors, and the exponential moving average (EMA) of the weights used by the evaluator models, this yields best performance which report.
Acting in the environment is accelerated by sending observations from actors and evaluators to a shared server that runs batched inference. The remote inference server allows multiple clients such as
16

---

observations 2t and Published as a conference paper at ICLR 2023
actor and evaluator workers connect to it, and executes their inputs batch the corresponding inference models. The actor and evaluator inference model parameters are queried periodically from the inference server that the actor does need to
the learner. Also, the recurrent state is persisted communicate it. However, the episodic memory lookup required to compute the intrinsic reward is performed locally actors to reduce the communication overhead.
At the beginning of each episode, parameters 3 and y are queried from the bandit worker, i.e. meta-controller. The parameters are selected from a set coefficients {(3 } N-1 with N = 16, which correspond to the N -hheads of the network. The actors query optimal (3, 7) tuples, while the evaluators query the tuple corresponding to the greedy action. After each actor episode, the bandit stats are updated based on the episode rewards by updating the distribution over actions, according to Discounted UCB (Garivier & Moulines 2011).
The following subsections describe how actors, evaluators, and learner are run in each stage.
LEARNER
Sample a sequence of extrinsic rewards rf. intrinsic rewards NGU
actions dt, from the replay buffer.
Use Q-network learn from (rf,r) r x. a) with our modified version of Watkins' Q(A) (Watkins & Dayan 1992) using the same procedure as in R2D2.
Compute the actor model weights and EMA for the evaluator model weights.
Use the sampled sequences to train the action prediction network in NGU.
Use the sampled sequences to train the predictor of RND.
ACTOR
Query optimal bandit action ().
Obtain 24 from the environment.
Obtain / and a, from the inference model.
Insert Ft, r:NG and rf in the replay buffer.
Step on the environment with de.
EVALUATOR
Query greedy bandit action (8, ?)
Obtain 2a from the environment.
Obtain NGU and from the inference model.
Step on the environment with de-
BANDIT
Periodically checkpoints bandit action values.
Queried for optimal action by actors.
Queried for greedy action by evaluators.
Updates the stats when actors pass the episode rewards for certain action.
F BANDIT IMPLEMENTATION
While Agent57 maintains separate bandit for each actor, instead utilize centralized bandit worker. The bandit selects between a family of policies generated by tuples of intrinsic reward weight and discount factor (Be, 7:) parameterized as:
17

---

Published as a conference paper at ICLR 2023
0.10
0.995
Intringk roward welghts B:
discount farttr
0.990
0.06
0.980
0.02
0.00
0.970
policy index
policy index
Figure 6: 3 and for each of the 16 policies.
0
if i 0
Bi
BIM
if = N -1 otherwise
BIMO (8 2i (N-))
N-2
N 1 N - log(1 Ymax) +
7/s = 1 exp
N - log Ymaxx
(10)
where N is the number of policies, i is the policy index, is the sigmoid function, BIM is the maximum intrinsic reward weight, and Ymax and Yint are the maximum and minimum discount factors, respectively.
At the beginning each episode an actor will sample a policy index with which to act for the duration of the episode. At the end of which, the actor will update the bandit with the obtained extrinsic return for that policy. We use a discounted variant of UCB-Tuned bandit algorithm (Garivier & Moulines (2008) and Auer important. We hypothesize that the use of cross-mixture training may reduce the sensitivity of the agent these parameters, though we have not explored this relationship thoroughly.
(2002)). In practice the bandit pypar-parameters did not seem be very
G COMPUTE RESOURCES
For the experiments we used the TPUv4, with the x 2 x topology used for the learner. Acting is accelerated by sending observations from actors to shared server that runs batched inference using a x 1 x TPUv4, which is used for inference within the actor and evaluation workers.
On average, the learner performs 3.8 updates per second. The rate which environment frames written to the replay buffer by the actors is approximately 12,970 frames per second.
Each experiment consists of 64 actors with 2 threads, each of them acting with their independent instance of the environment. The collected experience is stored the replay buffer split in 8 shards, each with independent prioritization. This accumulated experience is used by single learner worker, while the performance is evaluated on 5 evaluator workers.
H INTRINSIC REWARDS
H.1 RANDOM NETWORK DISTILLATION
The Random Network Distillation (RND) intrinsic reward (Burda et al. 2019 is computed by introducing random, untrained convolutional network g X R and training network 9 : X' + & to predict the outputs of g on all the observations that are seen during training by minimizing the prediction enror eITRNN (
[
l2 with respect 0. The intuition is
18

---

Published as a conference paper at ICLR 2023
that the prediction error will be large states that have been visited less frequently by the agent.
The dimensionality the random embedding, d, is a hyperparameter of the algorithm.
The RND intrinsic reward is obtained by normalising the prediction enror. In this work, use slightly different normalization from that reported [Burda et al. (2019). The RND reward at time t given by
eITRND((1))
RND
(11)
where og is the running standard deviation of eITRND (Iz). As with the TD-error statistics, we compute Je on the learner using importance sampling weights correct the sampling distribution.
H.2
NEVER GIVE UP
The NGU intrinsic reward modulates an episodic intrinsic reward, 7 epiisodic with a life long signal og::
.NNGU episodic min [max {a,, T
(12)
where L is a fixed maximum reward scaling. The life-long novelty signal is computed using RND with the normalisation:
(x1) He
(13)
de where emTNND (2t) is the prediction error described in Appendix H.l and Me and dg are its running mean and standard deviation, respectively. The episodic intrinsic reward at time is computed
according to formula:
episodic
1
(14)
ERENEN ((f(x)), f(x))) +
where N₂ is the set containing the k-nearest neighbors of f(x)) in M, c is a constant and K x -> + is kernel function satisfying K(x, x) (which can be thought of as approximating pseudo-counts [Badia et al.] (2019). Algorithm sshows detailed description of how the episodic intrinsic reward computed. Below we describe the different components used in Algorithm [
M: episodic memory containing at time t the previous embeddings f(z)),...
This memory starts empty each episode
k: number of nearest neighbours
Nr {f(x)))) set of k-nearest neighbours of (xx) in the memory M; we call No [a;) E N& for ease of notation
K: kernel defined as K(x v)
where is small constant, d the Euclidean L +
distance and d is rumning average of the squared Euclidean distance of the k-nearest neighbors
pseudo-counts constant
3: cluster distance
Smi maximum similarity
f(r): action prediction network output for observation x as in Badia et al. 2020
I OTHER THINGS WE TRIED
1.1 FUNCTIONAL. REGULARIZZTION IN PLACE OF TRUST REGION
Concurrentyy with the development of our trust region method we experimented with using an explicit 1.2 regularization term in the loss, acting on the difference between the online and target networks, similar to (Piche et al.) 202T). Prior to implementation of our normalization scheme we found that this method stabilized learning early on in training but was prone to eventually becoming unstable if run for long enough. With normalization this instability did not occur, but sample efficiency was worse compared to the trust region in most instances we observed.
As opposed to Agent57 which stored d² separately per actor, aggregate all
19

---

Published as a conference paper at ICLR 2023
Algorithm 1: Computation of the episodic intrinsic reward at time t: ri
episodic
Input :M; f(ze]: es m
Output:r
Compute the n-neurest neighhours of ((x,) M and store them list Nu Create list of floats de of size &
du will contain the distances between the embedding f(xe) neightours
No.
for á (I..... do do
de[s] (f(s.), Ne(il)
end
Update the moving average d', with the list of distances da
Normslize the distancea du with the updatied moving avarage de
dn + de
di
Cluater the normalized diatances da L.e. they become too small and Os is a liet of k
d, + max(d, - 0x)
Compute Kernel values between embedding ((x)) and its neighbours Nz.
K. + an=t
Compute the similarity between the embedding [(x)) and its neighbours No.
v V K=li++
/. Compute the aptcodic intrinata reward at time tt r
if 2m then
esse
postic + 1/8
I.2 APPROXIMATE THOMPSON SAMPLING
We considered using an approximate Thompson Sampling scheme scheme similar to Bootstrapped DQN (Osband et al. 2016) whereby the stochastic depth mask was fixed for some period of time at inference (such as once per episode, or every 100 timesteps). We observed some marginal benefit in certain games, but in our view this difference implementation. We hypothesize that the added exploration this provides is not significant when a strong intrinsic reward is already present in the learning algorithm, but it may have larger effect if this is not the case.
not enough to justify the added complexity of
I.3 MIXTURE OF ONLINE AND REPLAY DATA
We considered using a mixture of online and replay data as was done the Muesli agent (Hessel et al. (02T). This was beneficial for overall stability, but it also degraded performance in the harder exploration games such as Pitfall!. We were not able to find an easy remedy for this so we did not investigate further in this direction.
1.4 ESTIMATING (o) + (Q)
Agent57 uses two neural networks with completely independent weights to estimate Qe and Q, As mentioned in the work, this provides the network with more robustness to the different scales and variance that Te and r; have for many tasks.
MEME changes the separation of networks, whereby the Qe and Q, are still estimated separately, but they share a common torso and recurrent core. However, since many of the components we introduce are geared toward improving stability, even this separation may no longer be necessary. To analyze this we run experiment where the agent network has single head that estimates (. (Q)( Note that in this case we still estimate sets of Q-values. Indeed, as results of Figure 7 show, observe similar results our proposed method. This indicates that the inductive bias that was introduced in maintaining separate heads for intrinsic and extrinsic Q-values is no longer important.
20

---

Published as a conference paper at ICLR 2023
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfall!
300
15c
2004
230
43k
20k
Environment frames
Environment frames
Environment frames
Environment frames
Skiing
Surround
-54
10
-10c
-15
20c
-20c
10c
-10
.20
0.0
0.2
2.4
0,6
5.8
0.6
033
10
0.0
022
0.4
0.6
0.2
1.0
0.6
0.3
10
Environment frames
Environment frames
Environment frames 103
Environment frames
MEME
Single value head
baseline
Figure 7: Results for the two approaches to estimating the total loss.
21

---

Published as a conference paper at ICLR 2023
J
ADDITIONAL RESULTS
Frames to human score
Agent57
MEME
390M
Roat
Frothh
Tubrkhan
Roarrae
Hu TacoNan
hag
Snapeet
Spece bradders
Sant nes
We Nentue
Dette Zone
oree11
Amuus
name The The
Dunan
Weard Weard Wo
Breakek
Usee fitaa
Haling Do to
Samp sampume mmme
Assutt
Je
Dow
Oelindrr
Botts
hass Morur
Gothr
noad burter
cas cinber
Kul
Morbeauma
Figure 8: Number of environment frames required by agents to outperform the human baseline on each game (in log-scale). Lower is better. Error bars represent the standard deviation over seeds for each game. On average, MEME achieves above human scores using 62x fewer environment interactions than Agent57. The smallest improvement is 10x (Road Runner), the maximum is 734x (Skiing), and the median across the suite is 36
Garess above human Score
commannnnoralizee led score
10
20
Environment steps
MEME
Human benchmark
Average human MEME
Agent57 @ 90B Agent57 @
Ag ent57
MuZero @ 200
Figure Comparison with Agent57. Left: number of games with scores above the human benchmark. Right: human-normalized scores per game at different interaction budgets, sorted from highest to lowest. Our agent outperforms the human benchmark in 390M frames, two orders of magnitude faster than Agent57, and achieves similar end scores while reducing the training budget from 90B to 1B frames.
1000
Specdup > himan score
Sung
eiils go
boidle Dutk Bunk
horker
Vice
Chappo cammar
Freette
Batt 201e
Tonti
Sark Muiz
hong
Boam lice
janelord
space Grwitr
Word of Wer
Tatnoum
Yous Receng
Venture
Comipaee
Name The Came
Up DDon
oul
Kunp hu Manor
belncer
Gotee
Couy Ginte
loa hund?
Fanen
Figure 10: Speedup in reaching above human performance for the first time, computed as the ratio between the orange and blue bars in Figure
22

---

Published as a conference paper at ICLR 2023
Median HN5
1000
Mean NH5
MEME
600
2000
Muesii @ 200M
1000
1.0
0.4
1.0
Environment steps
Environment steps
Figure 11: Median and mean scores over the course of training.
K STICKY ACTIONS RESULTS
This section reproduces the main results of the paper, but enabling sticky actions (Machado et al. 2018) during both training and evaluation. Since our agent does exploit the determinism in the original Atari benchmark, it is still able to outperform the human baseline with sticky actions enabled. We observe slight decrease in mean scores, which we attribute label corruption in the action prediction loss used to learn the controllable representations used in the episodic reward computation: due to the implementation of sticky actions proposed by Machado et al.) (2018) the agent actions are ignored in fraction of the timesteps. This phenomenon is aggravated by the frame stacking used in the standard Atari pre-processing, as the action being executed in the environment can vary within each stack of frames. We hypothesize that the gap between the two versions of the environment would be much smaller with a different implementation of sticky actions that did not corrupt the action labels used by the representation learning module.
All reported results are the average over six different random seeds.
47
Games human
40
MEME
Human benchmark
MuZero [2021] @ 200M
20-
200M
400M Environment steps
G00M
BD0M
1B
Figure 12: Number of games with scores above the human benchmark when training and evaluating with sticky actions (Machado et al., 2018).
Frames to himan score
10OM
JN
hitll
HEROO.
Nostl Eye
Am
Roo
Chopper Commant
Telp
hame The Came
Putanhhem
Vanuee
Mc NoMtnn
mrem Daeass
Ite snccry
Bent Postt
Decure
Frodd
Witr or War
Surgerna
Vive Saurro
Zone
The
Deferdtt
turgiu Vaster
Via Fmaa Dow
onee cimbur
Ritorrk
Alonts
Tuns
Figure 13: Number of environment frames required by agents to outperform the human baseline on each game when training and evaluating with sticky actions (Machado al. 2018). The human baseline is outperformed after 505M frames.
23

---

Published as a conference paper at ICLR 2023
3500
Medann HNS
Mean NHS
3000
2500
MEME
000
2000
Muesii @ 200M
1500
400
1000
200
200
Environment steps
Environment steps
Figure 14: Median and mean scores over the course training when training and evaluating with sticky actions (Machado al. 2018).
Table 4: Number of games above human, capped mean, mean and median human normalized scores for the 57 Atari games when training and evaluating with sticky actions (Machado et al., 2018). Metrics for previous methods are computed using the final score per game reported in their respective publications: MuZero (Schrittwieser et al. 2021), Muesli (Hessel et al. 2021
200M frames
200M frames
Statistic
MuZero
MEME
Eny frames
200M
200M
200M
1B
Number games human
54
52
49
57
Capped mean
97.51
92.52
89.78
100.0
Mean
Median
830.57
1077
1006.4
1074.25
percentile
153.
5th percentile
103.86
15.91
28.76
118 78
24

---

Published as a conference paper at ICLR 2023
L EFFECT OF SAMPLES PER INSERT RATIO
Results of the ablation on the amount replay that the learner performs per sequence experience that the actors produce can be seen in Figure [5] We of 10 still provides moderate boosts in data efficiency in games such as H.E.R. O., Montezuma's Revenge and Pirfall!, it is not as pronounced the increase that is seen from SPI implies that with an SPI of 10 we obtain much worse return in terms of wall-clock time as we replay more frequently.
that, while samples per insert ratio (SPI)
6. This
Frostbite
H.E.R.O.
Montezuma's 's Revenge
Pitfall!
1.M
100k-
B0k-
2004
5004
5t
0.0
0.2
9.4
5.d
0.0
0.8
0.8
1.0
0.0
0.2
0.6
0.6
0.0
08
10
Environment frames 100
Environment frames 103
Environment frames 109
Environment frames 102
Skiling
Solaris
Surround
Tennis
30c-
20c
0
-104
Environment frames 1-9
Environment frames
Environment frames le9
Environment frames
MEME (SP)=6)
SPI=0.8
SPI=3
SPI=10
Human baseline
Figure 15: Sweep over sample per insert ratios on the full ablation set.
25

---

Published as a conference paper at ICLR 2023
M R2D2 ABLATIONS
We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when used with the R2D2 agent (Kapturowski al. 2018), shown in Figure [6 Similarly to the original ablations of MEME, we can see that the trust region and online bootstrap- ping constitute the most important component, which once removed deteriorates the performance significantly. In general, the rest of the components are beneficial as well, as MEME R2D2 can be seen to obtain human-leve performance games used in this ablation is not fully representative Montezuma's 's Revenge, Pitfall! and Skiing are not expected be solvable without intrinsic motivation the ability leverage larger discount factors.
the largest number of games; although the set Atari
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfall!
64-
4.0
36-
60c
24
200c
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2 044
0.6
08
1.0
0.0
0.2
0.4
0.6
098
1.0 lea
0.0
0.2
0.4
0.6
08
1.0
Environment frames 1e9
Environment frames 169
Environment frames
Environment frames 169
Sklling
Solaris
Surround
Tennis
30c
-20c
20c
0
-30k
10c
4
10
0.0
2.4
b2
10
0.0
0.2
0.4
0.6
109
frames
1c3
Environment frames
MEME R2D2
Policy distillation (D)
NFNet torso (CI)
- Trusttregion Online boostrapping (A1)
Soft Watkins' QUA) (A2)
Human baseline
Normalization (B1)
Figure 16: Ablation study of the proposed components while using the R2D2 agent.
26

---

Published as a conference paper at ICLR 2023
N
FULL ABLATIONS
Frostbite
H.E.R.O.
Montezuma's m Revenge
Pitfal!!
550c
15c
100k
4.04
200c
20k
0.0
0.2
0.4
1.0
0.0
0.2
04
10
169
frames
169
Environment 169
Skiing
Solaris
Surround
Tennis
-10k
30c
-15.
20c
0-
-20c
-254
10c
-30c
10
-00
0.2
0.2
Environment frames
Environment frames
Environment frames
Environment frames
MEME
Combined loss
Human baseline
Figure 17: Comparison between our combined loss and the separate losses used by Agent57 on the full ablation set.
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfal!
300c
15k
100k
400
200
20k
0
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.6
0.8
10
109
109
109
Environment le5
Skiing
Solaris
Surround
Tennis
-30k
-15
20c
6-
0-
-20c
-25
10c
-30c
10
-00
0.2
0.0
0.2
Environment frames
Environment frames
Environment frames
Environment frames
MEME
Cross-mixture training
Human baseline
Normalization
Cross-mixture training Normalization
Figure 18: Results without cross-mixture training and TD normalization on the full ablation set.
27

---

Published as a conference paper at ICLR 2023
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfal!
300c
15k
100k
6.0
200
63k
20k
0.0
0.2
2.4
0.0
0.2
0.4
10
Environment 129
frames 1e3
Environment
le3
Environment frames le3
Skiingg
Solaris
Surround
Tennis
40c
-54
-10K
-154
20c
0-
-20c
-254
10c
-304
0.2
0.0
0.2
Environment frames
Environment frames
Environment frames
Environment frames
MEME
Nesterov momentum
Percentile clipping
Human baseline
Figure 19: Ablation experiments over different optimizer features.
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfall!
100k
400
60k-
200
100c
5k
0.0
0.2
3.4
0.6
5.8
1.0
0.0
0.2
0.4
0.6
08
1.0
0.0
0.2
0.4
0.6
68
1.0
0.0
02
0.8
0.6
08
1.0
Environment frames
lea
Environment frames 169
Environment frames les
Environment frames 169
Skiing
Solaris
Surround
Tennis
40k
30c
10c
-10
30c
-10
0.0
9.2
0.4
0.0
022
0.4
0.6
Environment
129
frames 123
109
Environment 165
MEME
Policy distillation
Human baseline
Figure 20: Results without policy distillation on the full ablation set.
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfall!
15k
2004
10k
83k-
62k
100
42k
20k
0.2
0.0
0.2
Environment frames
Environment frames
Environment frames
Environment frames
Skiing
Solaris
Surround
Tennis
10
-10c
30c
10
20k
20c
-10
-30c
0.0
0.2
2.4
6/6
5.
0.0
0.2
24
0.6
1.0
0.0
02
0.4
0.6
6.
1.0
0.0
0.2
0.6
0.6
08
1.0
Environment frames les
Environment frames 169
Environment frames les
Environment frames
Weight deray
Human baseline
Stochastic
Weight decay Stochastic depth
Figure 21: Results for agents with different amounts of regularization the full ablation set.
28

---

Published as a conference paper at ICLR 2023
Frostbite
Montezuma's Revenge
15c
100k
4.04
200
10c
62k
626
2004
100c
40k
20k
0.0
0.2
3.4
9.8
0.8
1.0
0.0
0.2
04
0.6
0.3
1.0
Environnent frames
Environment frames
Environment frames I69
Environment frames
Skiing
Solaris
Surround
Tennis
10
-104
30c
20c
10c-
-10
10
0.0
0.2
9.4
0.6
5.8
1.0
0.5
0.6
0.8
10
0.0
0.2
0.6
0.6
0.8
1.0
0.0
0.6
1.0
Environment frames 100
Environment frames
100
Environment frames 169
Environment frames 163
Soft Watkins' Q(X)
Soft Watkins' QIA), -
Peng's Q(A)
Retrace(A)
Human baseline
Figure 22: Results with different learning targets on the full ablation set.
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfal!
3004
15k
100k
430
230
29k
0.0
3.2
0.4
10
0.0
0.2
0.4
0.2
10
129
frames
103
Environment
109
Environment frames
Skiing
Solaris
Surround
Tennis
40
10
30c
20c
6-
0-
30c
-20
0.5
0.2
0.0
0.2
5.4
Environment frames
Environment frames
Environment frames
Environment frames
MEME (NFNet)
ConvNet
GroupNorm ResNet
LayerNorm ResNet
RMSNorm ResNet
Human baseline
Figure 23: Results for agents with different torsos on the full ablation set.
29

---

Published as a conference paper at ICLR 2023
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfal!!
330c
B5
1006
400
2004
54
0
0.0
0.2
0/4
0.6
3.0
3.0
0.5
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.
1.
Environment frames 1ea
Environment frames leq
Environment frames lea
Environment frames 102
Skiing
Solaris
Surround
Tennis
40c
30c
20
0
10c
-304
-10
Environment frames
Environment frames
Environment frames
Environment frames
MEME
Online boostrapping
Human baseline
Trust-region
Trust-region Online boostrapping
Figure 24: Results for agents without online bootstrapping and trust-region on the full ablation set.
Frostbite
H.E.R.O.
Montezuma's Revenge
Pitfall!
300
150c
12.c
10c
230c
100c
40k
x00
2.54
0.0
Environment frames
Environment frames
Environment frames
Environment frames
Skiling
Solaris
Surround
Tennis
5c-
-10c
-154
-204
-20
0.0
0.2
2.4
0.6
3.8
1.0
0.5
0.2
0.5
0.
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.6
0.6
0.3
10
Enviranment frames led
Environment frames
Environment frames 203
Environment frames
Multi-headed architecture
Single-headed
baseline
Figure 25: Comparison of head architectures. The headed architecture is similar to the one in Agent57, where the network is conditioned on a one-ho hot encoding of the mixture id. All experiments without cross-mixture training and TD normalization for fairness. These results demonstrate that our multi-headed architecture, introduced to enable efficient computation Q values for all mixtures in parallel, does not degrade performance.
30

---

Published as a conference paper at ICLR 2023
O SCORES PER GAME
Table 5: Scores per game.
Game
Agent57
Agent57
MEME
MEME
Alien
3094.46 + 1682.49
252889.992 02-.999222222 31085.48 .48
41048.76 + 8702.03
83683.43 + 10688.58
Amidar
$29.99
28671.19 1046.52
7361.47 1 1031144
14368.90 2775.86
Assault
1183. 25 1: 482.14
49198.37
9469.92
33266667 1: 6143.71
46615,86 86 1: 14846.53
Asterix
2777.61 +
763176.87
105395.98
861780.67
769803 113061.91
Asteroids
2422 64 + 412.96
105058.07 07 = 46390.23
217686.98 = 17304.43
364492 07 13982.31
Atlantis
44844.67 15898.85
19.97 249133.00
1535634.17
20700.44
1669226.35
3906,17
Bank Heist
364.26 1: 156.87
17274.04 12144.00
15563.35 :: 14565.74
87792.55 55 104611.7
Battle Zone
19005, 86 +: 999.82
8685 57 26523.78
733216.67 $1205-30 20
776770.02 19731 15
Beam Rider
2056. + 131.88
2838455 01 112899 64
68534-71 + 1143-31
51870. 2906,
Berzerk
614.26 - 64.32
31565.12 12 + 34278.20
7003. 12 + 2302261
38838.35 + 14783.99
Bowling
34.00
240.31 18.96
261.83 2,30
Boxing
97. 92.922222 +: 1.08
99. 93 :: 0.03
99.77 + 01.17
99 85 = 0..14
Breakout
62.09
696.91 63.20
747.62 + 61.06
08 6.
Centipede
1441 1.24 + 1780.00
348288.7 74 119557.45
112009.74 74 + 12701-73
245892. 18 + 30000.78
Chopper Command
3335.40 1: 1031.08
959747 29 10225,76
842327. 17 168080.15
91 .0 11 112908.65
Crazy Climber
85166.13 1: 9121.39
456853 80 13192.11
295419-67 1: 59774667
339274.67 1: 14518.41
Defender
33613.13 13 6856.21
666433.14 14 17:93.30 30
518605.50 .50 18011 9.
513979 .0 + 7639.27 27
Demon Attack
1786. + 91111.22
140-74.21 21- + 2931.07
139340.76 + 1927-91
142176.56 58 + 1223 59
Double Dunk
21.76 0.52
23. 33.64 + 0.06
20,00 + 0000
23. 70 + 0.44
Enduro
437.34 97.19
2349.003
2138. T 38.90
2360.66 64
Fishing Derby
- 13.36 = 19.65
83. 12 4.11
G7 19 + 5.30
77.05 = 3.97
Frocway
22 22.16-22 +
32.
33,82 + 0.10
33.9
Frostbite
980. 15 + 581.61
507775 65 + 35925.95
13660 777 35672.33
(26239.50
18289.50
Gopher
9760.20 1. 1790.84
98786. 1 4600.57
117557.53 1 3264.72
119457.55 1. 4077.33
Gravitar
666. +: 304,08
18180 +: 627.48
18049966 67 :: 272 66
20875. 00 :: 8414.41
H.E.R.O.
8850.63 = 1313.70
1021 96 50561.27
33872 .9 + G917. 11
199880.60
14074.56
Hockey
14.87
62.33 5.77
26.07 +
47.22 4.41
Jumeshond
534.53 1: 408.19
107266. 11 1: 15584.58
137313.1 17 21939.77 77
117009.92 J. 55411.1111
Kangaroo
3178.86 + 996.26
18505.30 37 5016.06666
18863,0 675.46
17311.17 17 + 419.17 17
Krull
9179.71 + 1222.12
194179.21
21451.96
157943.83 +. 26800.67
155915.32 + 43127.45
Kung Fu Master
31613.73 1. 8854.35
192616. 60 1. 9019.89
364755.65 65 387274.47
476539.55 53 518479..8
Moniezumna's Revenge
200. + 192.50
8666, = 2928.92
2863-00 + 117.91
12437.00 00 + 16118 44
Ms. Pac-Man
3348. 12 + 630.07
57402. 56 + 3077.22
22833, 12 + 28 13.55
29747.9 91 + 2472.333
Name This
4463.97 1747.50
48644.27 27 23900 83
31300.93 + 2037
40077. 2274.25
Phoenix
7359.04 1 5542.07
858909.18 13 37669.01 0.6
602303.55 53 433067. 79
849969 25 43571.52
413.21
19665.00
574 + 811-43
467344.9
Pong
-15.02 + 1.96
20.29 +: 0.65
17. 17.91 + 6.61
19:31 = 2.42
Private Eye
5727.66 1619.10
79347.0 98 25315.82
64 64145.31 211000
100798.00 90 1.07
Q"BERT
3806.99 1: 1654.31
437607 43 111087.57
96189.83 1: 17377.21
238451. 50 272386.91
Riverraid
6077.47 : 1810.98
16276.56-20 1: 6513.66 69
40266.92 1 4087.60 .60
90333. 12 1. 4694.40
Road Runner
25303.07 = 1360.56
168085. 10 = 10390.00 00
147833.33 33 = 128698.32
399511 83 + 111036.39
Robotank
13. + 2.55
116.93 10.64
87. 77.70-5 + 5.85
114.46 = 3.71
Seaquest
2146.63 1574.51
999063.77 7T 1160.16
577162.47 56047.06
960181.39 1311 25453.79
Skiing
-25261.49 1: 1193.77
42899 49 1. 628.37
-341 .6 1. 185.93 93
3273 43 1. 4.67
Solaris
29688.5 1470.62
39841108 08 6788.17
13514.80 + 1231.11
4859.26
Space Invaders
640. 13 185.45
35150. 40 = 3388 53
33214.80 80 6372. 10
57828 45 + 7551.63
Stargunner
11214.14 14+4 4667.
796115 15, 73384.04
221215.33 13074.19 19
264286.33
10019.21
Surround
8..77 1. 0.69
5.83 1 0.58
9.64 1. 0.17
9.82 0.05
Tennis
-18.31 1: 2.11
23. 0.15
23.18 1 01.33
22.79 :: 0.65
Time Pilot
3561.51 = 1114.00
382111 86 = 17388,79 79
169812.33 = 37012.22 23
404751.6 1.67 17305 23
Tutankham
106.08 = 13.87
2012 2 54 + 2853.4 44
402. 16 + 22.73
1030.27 + 11.88
Up'n Down
15986.44 2213.66
80 32333.66
472253.82
524631 20108.60
Venture
477.71 1:111::20 251.24
2544.90 1: 403.53
2261.17 1 661.39
2859. 83 1: 195.14
Vidco Pinball
18012.16 = 2773.10
885718
778530. 78 79125.55
617640.99 127005.
Wizard Of Wor
3402.48 + 1210.12
134441. 09 + 8913.67
G7072.67 = 13768 12
71942 00 6562 86
Yars' Revenge
26310.18 + 64422.63
976142.42
3219.52
654338.02
100597.12 12
633867.66 128524.41
Zaxxon
7323.83 1 1819.10
195043397
18131.20
79120.00 9783.55
77942.17 17 6614.61
31

---

Published as a conference paper at ICLR 2023
Table 6: Scores per game when training and evaluating with sticky actions (Machado al. 2018
MEME
MEME
Go-Explore
Alien
48076.49 + 103 10.65
68634 82 + 15653.10 10
Amidar
7280.27 27 + 808.09
20776.98 4859.39
Assault
27838.77 75 4337.41
31708.64 + 14199.48
126291.56
729820.40 82360.83
Astemidk
212460.60 6585.36
335137.55 50 + 32384.14 14
Allantis
1462275.60 1: 144898.14
1622960. 80 1958.
Bank Heist
6448.48 48 3066.16
45019.92 8611.39
Battle Zone
756298. 00 = 71092.41
763666 00 = 53978: 21
Rider
380-19.60
Berzerk
16265. 88 + 10497.83
45729.94 + 13228.22 29
197376 (Ø 10B)
Bowling
264.77 ::
212.70
I0B)
Boxing
99.77 0.10
99.86 = 0.11
Breakout
521.32 49.04
475.87 1. 53.73
Contipede
35068.44 43 6370.11
63792.64 44 + 24203.69
1422628 (@ 10R)
Chopper Command
170450. 00 + 318026.00 00
181573. 3.80 + 342149 46
Crazy Climber
Defender
521330.30 30 f. 10194.48
561521 30 1. 7955.85
Attack
130545.44
9343.72
1119.30
Double Dunk
23.12 0.55
23.78 = 0.09
Enduro
2339.31 1: 13.75
2352.94 94 : 14.33
Fishing Derby
68. 4 4.92
79. 79.65-2 2.65
Freeway
33.88 + 0.08
33 0.02
34 (@ 10B)
Frosibite
187638.50
Gopher
105836. 08 :: 13458.2 22
96034.76 76 1. 17422.
Gravitar
12864.
260.14
19489. + 825.38
7588 10B)
H.E.R.O.
27998.99
5920.99
175258 87 = 15772.55
Ice Hockey
38.14 9.23
52.51 51 10.04
Jamesbond
144864.80 8709.86
118039.20 47454.71
Kangaroo
15559. 00 1494.83
16951.61 60 + 275.12
28863.77
Kung-Fu Fu Master
182036.20 20 4195.48
208166. 80 +: 2714.94
Montezuma's Revenge
2890.40 + 42.06
9429.20 1485.32
43791 (@ 10B)
Ms. Pac-Man
25158. 68 1786, 43
27054.73 + 152.14
Name This Game
29029.20 20 2237.35
34471.38 30 2135.99
Phoenix
14035-008 70760.72
788107 33424.51
Pitfal!!
235. 235.90 1 493.77
7820.90 94 : 16815.81
6954 (Ø 10B)
Pong
19.38
Private Eye
90596.05 05 + 19942.76
100775. 10 15.57
95756 (@ I0B)
Q"BERT
137998. 80896.15
328686. 257052.TH
Riverrad
36680.36 36 1287.15
67631 t 4517.53
Roud Runner
515838.00 1: 153908.19
543316.22 20 1 64169.67
Robetank
Seaquest
474164.66 86 i. 62059.77 .3
744392 28 1 41259.26
Skiing
3339.21 14.59
3305.
-3660 (Ø 10B)
Solaris
13124. 24 + 657.30
28386 28 + 2381 29
19671 (@:00B
Space Invaders
26243. 6053.57
522541 64 4421 24
Stargunner
173677.66 60 19678.82
190235 40 1: 6141.42
Surround
9.60 1: 0.24
9.66 1: 0.23
Time Pilot
159728. 20 1. 39442.90
3545559 80 1. 22172.70
Turankham
383.85 +
924.47 +
Up' n Down
478535. 61 = 15969, 41
528786.11 12 + 5200.79
Venture
2518.20 20088.0044 + 56.39
2583. 40 + 175.95
2281 (Ø 10B)
Video Pinhall
750858.98 98 1: 115759.28
759284.66 69 1. 37
Wizard Of War
65005.80 80 7034.75
66627. 00 9196.92
Revenge
854251.
7axxon
85322.00 = 12413.86
69809.20 20 +: 2229.49
32

---

Published as a conference paper at ICLR 2023
300
15A
8.50
Bank Heist
Battee Zone
309
Bewn Rider
Berzerk
19
100
758
Breacout
Centipede
Chopper Command
Crazy Clmber
Defender
Demon Attace
150t
600
10t
3054
200
02
675
Double Dunk
Enduro
Fishing Deby
Freesey
Frostbile
2.5
0.5
3.00
120 ude
6.5 45S
3.99
875
1.00
0.2 035
659
005 6.22
0.0
Hockey
Kangaroo
675
Master
Montecumals
J00
Pac-Man
140
Name Tris Game
Phomie
Pitill
3.50
15
Pong
Private Rye
Q"BERT
Riverned
Road Runner
Robstank
200
109 av
10
-10
30c
0.0
4.20 US4
22
015
655 Surmund
+.5
Seargeet
Sking
Saarts
Space Invaders
Stargumnwr
6.6
0.5
0.5
1.0
620 u5e
A75
3.9
8.00 4.5
o50
875
(50
015
006 6.25 1.0
0.0
005
150-
00
154-
uSe
675
1.0
Wiowrd Wer
Marti Revenge
300
Zasson
NEM
NTM un
Human baselie
Figure 26: Score per game as a function of the number of environment frames, both with and without sticky actions (Machado al. 2018). Shading shows maximum and minimum 6 runs, while dark lines indicate the mean.
33


Reviews:
[{"review_id": "yvp4dY1gqh", "reviewer": "Reviewer_zSus", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces MEME, a method built off of Agent57 that enables exceeding the human baseline results with 200x fewer samples than Agent57 required.  This is accomplished by identifying bottlenecks and proposing solutions along with strategies to ensure stability and robustness.  The paper includes thorough experiments demonstrating the benefits of the proposed method and ablations providing intuition about each of the components.", "summary_of_the_review": "This paper had a clear and significant goal, dramatically improving the sample efficiency of Agent57, which it states concisely and accomplishes through a wide range of well motivated modifications, as demonstrated through extensive experimentation.  I recommend acceptance.", "strength_and_weaknesses": "Strengths:\n- Identify a range of issues and bottlenecks in the original Agent57 design and systematically address them in addition to any side effects of their interventions, providing a set of tools that could be applied to a variety of problems along with an evaluation of their effectiveness.\n- The result of these proposed methods is strong, at a 200 fold improvement in sample efficiency and competitive performance to MuZero and Muesli.\n- Very thorough ablations, evaluation of methods on R2D2 in addition to Agent57, and an extended appendix including a list of things the authors tried but didn’t work, facilitating future research.\n\nWeaknesses:\n- This is a strong paper which claims impressive improvements on sample efficiency against Agent57 and supports those claims with thorough experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "tmRGBDicuBw", "reviewer": "Reviewer_SNik", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission introduces an agent called MEME, which is built on Agent57, which in turn is a super inefficient atari player that requires a ridiculous number of training samples, not even remotely being sensible. Numerous ad-hoc or semi ad-hoc techniques are used (most of which with little to no formal ground) to mitigate various issues of Agent57. The resulting agent (MEME) is able to reduce the training samples by a factor of 200.", "summary_of_the_review": "The paper provides several different modifications to Agent57, nearly none of which is formally studied or justified. As a reader, when I went through different sections, they only raised the question of \"why\". What is clear is that this manuscript is far from a research paper. ", "strength_and_weaknesses": "Strength:\n\n- An obvious improvement over the Agent57.\n\n- Perhaps the only interesting part of the paper is the trust region for the value function. \n\nWeaknesses:\n\n- Atari is simply a benchmark and not a goal. I honestly do not see any merit in these types of so-called research, especially when considering the tremendous amount of carbon emission and un-necessary energy consumption. \n\n- Most of the presented techniques are ad-hoc with little formal analysis and reasoning. This hardly adds to the common knowledge of the community, and in light of the demanding computational requirements in the development stage of such projects, it is unlikely to open up new avenues for research either. \n\n- Approximate GPU hours for the entire project not reported. As \"responsible research\" has become an important part of the AI community at large, such information helps the community to evaluate the merit of this work aside from possible hypes. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aKhUdokexa", "reviewer": "Reviewer_tLk9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves Agent57 in the following four aspects, faster propagation of learning signals, handling different value scales, efficient NN architecture, robustness about policy update. It achieves great speedup on training human-level Atari AIs, and 200-fold reduction. \nThe ablation experiments are complete and careful. Also, the authors provide very detailed information in the appendix.\n", "summary_of_the_review": "Overall, this paper is good, providing several improvement techniques for Agent57 with significant speedups. Also, these techniques are well studied with ablations.\n", "strength_and_weaknesses": "**Strength:**\nThe quality of writing and survey are good, and the result is state-of-the-art in terms of training speedups at human level. Data efficiency is a major problem in practical DRL cases; the proposed methods exhibit good results.\nSince this paper has shown great improvement in the aspect of the game frame, I would also like to know how fast it is if we use wall time for counting the speedup (like many distributed methods focused).\n\n**Weaknesses:**\nIt is an unclear part to me about the Retrace update: does MEME use transformed Bellman operator? (There is a “Value function rescaling” in the appendix-A)\nWould the transformed value function be related to the part “B” (handling different value scales) or not? I do not see any mention of the transformed value function in both the main paper or appendix.\n\nA minor comment is about references: it is better to cite papers to conferences, instead of arXiv as listed as follows. \n- Distributed prioritized experience replay => ICLR 2018\n- Image augmentation is all you need: Regularizing deep reinforcement learning from pixels => ICLR 2021\n- Fast and Data Efficient Reinforcement Learning from Pixels via Non-parametric Value Approximation => AAAI 2022\n- Prioritized experience replay => ICLR 2016\n- Data-Efficient Reinforcement Learning with Self-Predictive Representations => ICLR 2021\n- V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control => ICLR 2020\n- CURL: Contrastive Unsupervised Representations for Reinforcement Learning => ICML 2020\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster