Unlocking the Power of Representations in Long-term Novelty-based Exploration
Authors: Alaa Saade
Steven Kapturowski
Daniele Calandriello
Charles Blundell
Pablo Sprechmann
Leopoldo Sarra
Oliver Groth
Michal Valko
Bilal Piot
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep RL
exploration
density estimation
representation learning
Primary area: reinforcement learning
TL;DR: We introduce a new novelty estimator for exploration in deep RL which can preserve long-term memory and be used with any representation learning techniques

Abstract: We introduce Robust Exploration via Clustering-based Online Density Estimation (RECODE), a non-parametric method for novelty-based exploration that estimates visitation counts for clusters of states based on their similarity in a chosen embedding space. By adapting classical clustering to the nonstationary setting of Deep RL, RECODE can efficiently track state visitation counts over thousands of episodes. We further propose a novel generalization of the inverse dynamics loss, which leverages masked transformer architectures for multi-step prediction; which in conjunction with \DETOCS achieves a new state-of-the-art in a suite of challenging 3D-exploration tasks in DM-Hard-8. RECODE also sets new state-of-the-art in hard exploration Atari games, and is the first agent to reach the end screen in "Pitfall!"

Published as a conference paper at ICLR 2024
UNLOCKING THE POWER OF REPRESENTATIONS IN LONG-TERM NOVELTY-BASED EXPLORATION
Alaa Saade", Steven Kapturowski, Daniele Calandriello", Charles Blundell, Pablo Sprechmann, Leopoldo Sarra', Oliver Groth, Michal Valko, Bilal Piot. Google Deepmind falaas, skapturowski cblundell, paprechmann, peeoooolmmmmeeoooodooo sarra, ogroth, piatemmmmpoto@ooogoo.o com
3kkapuuroosskidddaaaddrlllll dcalandrieell,
ABSTRACT
We introduce Robust Exploration via Clustering-based Online Density Estimation (RECODE), a -on-mmmtetriic method for novelty- based exploration that estimates visitation counts for clusters of states based their similarity in chosen embed- ding space. By adapting classical clustering to the nonstationary setting of Deep RI., RECODE can efficiently track state visitation counts We further propose a novel generalization of the inverse dynamics loss, which leverages masked transformer architectures for multi-step prediction; which in conjunction with RECODE achieves a new state-of-the-art in a suite of challenging 3D-exploration tasks in DM-HARD-8, RECODE also attains state-of- oftte--f-hee--ee ff-th--rrt perfor- mance in hard exploration Atari games, and is the first agent to reach the end screen in Piffal!!
thousands of episodes.
INTRODUCTION
Exploration mechanisms key component of reinforcement learning (RI., Sutton & Barto 2018)
agents, especially in sparse-reward tasks where long sequences actions need to be executed before collecting reward. The exploration problem has been studied theoretically (Kears & Singh (002)
Azar et al. 2017 Brafman & Tennenholtz] [003 Auer et al. 2002 Agrawal & Goyal 2012 Audibert
al. 2010 Jin al. 2020) in the context of bandits (Lattimore & Szepesvári 2020) and Markov
Decision Processes (MDPs, Puterman 1990 Jaksch et al. 2010)). One simple yet theoretically- sound
approach for efficient exploration in MDPs
use decreasing function of the visitation counts as an exploration bonus (Strehl & Littman 2008 Azar et al. 2017). However, this approach becomes intractable for large or continuous state spaces, where the agent is unlikely to visit the exact same state multiple times, and some form of meaningful generalization over states is necessary. Several approximations and proxies for visitation counts and densities have been proposed to make this form of exploration applicable to complex environments. Two partially successful approaches in deep RL are: the parametric approach, which uses neural networks to estimate visitation densities directly, and the non- non-paramttric approach, which leverages memory of visited states to guide exploration.
Parametric methods either explicitly estimate the visitation counts using density models (Bellemare et al.) 2016 Ostrovski et al., (017) use proxies for visitation such as the prediction error of dynamics model (Pathak et al. 2017], 7 Guo et al.) 2022), or from predicting features of the current observation, e.g., features given by fixed randomly initialized neural network in RND (Burda ct al. 2019). While this family of methods provides strong baselines for exploration in many settings (Burda et al., 2018), they are prone to common problems of deep learning in continual learning scenarios, especially as slow adaptation and catastrophic forgetting. Parametric models trained via gradient descent are generally unsuitable for rapid adaptation (e.g., within single episode) because it requires updates to the state representation before the exploration bonus can catch up. Additionally, catastrophic forgetting makes parametric methods susceptible to the so-called "detachment" problem in which the algorithm loses track of promising areas to explore (Ostrovski ct al. 2017). Maramaraic methods rely on a memory to store encountered states (Savinov et al. 2018 Badia et al. 2020b). This facilitates responsiveness to the most recent experience as well as preserving memories without interference. However, due to computational constraints, it is necessary to limit the memory size which, in turn, requires selection aggregation mechanism for states.
Equal contributions, Department of Physies, Friedrich-Alexanee Universität Frlangen-Nürmberg, work done while interning DeepMind.

---

Published as a conference paper at ICLR 2024
To obtain the best of both worlds, Never Give Up (NGU, Badia al. 2020b) combines short-term novelty signal based on an episodic memory and a long-term novelty signal based on RMD into a single intrinsic reward. However, the need to estimate two different novelty signals simultaneously adds complexity and requires careful tuning. Moreover, as pointed out by Pathak et al. (2017), the final efficacy of any exploration algorithm strongly depends the chosen state representation. If the state encoding is susceptible to noise or uncontrollable features in the observations, it can lead to irrelevant novelty signals and prevent meaningful generalization over states. As NGU relies on RND for representation. it also inherits its encoding deficiencies in the presence of noisy observations which limits the applicability of the method in stochastic or complex environments.
In this paper, we tackle these issues by decom- posing the exploration problem into two disen- tangled sub-problems. First, (i) Representation Learning with an embedding function that en- codes a meaningful notion of state similarity while being robust to uncontrollable factors in the observations. Second, (ii) Count Estima- tion that is able to provide a long term visitation- based exploration bonus while retaining respon- siveness to the most recent experience. Address- ing (i), we extend the inverse dynamic model proposed by Pathak et al. (2017) by leveraging the power of masked sequence transformers (De- vli et al. 2018) to build an encoder which can produce rich representations over longer trajec- tories while suppressing the encoding of uncon- trollable features. We refer to our representation as CASM, for Coupled Action-State Masking. In order to deliver on (ii) we introduce a novel, non-parametrie method called Robust Exploration via Clustering- based Online Density Estimation (RECODE). In particular, RECODE estimates soft visitation counts in the embedding space by adapting density estimation and clustering techniques to online RL setting. Our approach tracks histories of interactions spanning thousands of episodes, significantly increasing memory capacity over prior art in non-parametric exploration methods which typically only store the most recent history like the current episode. In the presence of noise, we show that it strictly improves over state-of-the-ar art exploration bonuses such as NGU or RND. RECODE matches or exceeds state-of-the-ae exploration results on Atari and is the first agent to reach the end-screen in Pitfall!, a notoriously difficult task due to strict in-game time limits that require discovering an efficient route that explores and backtracks across 255 rooms. Beyond 2D, our method also performs well in much harder 3D domains and in conjunction with CASM, sets new state-of-the-e results in the challenging DM-HARD-8 suite (Fig. in terms of human normalized score (HNS, Mnih et al. (2015) )
Mean Capped HNS on DM-HARD-8
Capped HNS
90
MEME RECODE-CASM VMPO-BYOL 4Explore
Number of frames
1e9
Figure 1: A key result of RECODE is that it allows us to leverage more powerful state representations for long-term novelty estimation. This enables new state- of-the-art performances the challenging 3D task suite DM-HARD-8, where for the first time we achieve super- human performance (i.c., exceed 100 in human normal- ized score) in 6 out of 10 tasks.
2 BACKGROUND
We consider a discrete-time interaction (MCCallum 1995 Hutter 2004 Hutter et al. 2009 Daswani al., 2013) between an agent and its environment. At each time step t E the agent receives an observation o, E O, that partially captures the underlying state E S of the environment and generates an action a A. We consider policies ग O + A that map an observation probability distribution over actions. Finally, an extrinsic reward function re S x A + R maps an observation to a scalar feedback. This function can be combined with an intrinsic reward function r;; to encourage the exploratory behavior which might not be induced from r. alone.
The observations provided to the agent at each time step are used to build a representation of the state via an embedding function fo : O + E, associating o, with a vector e (oz). Typically, the embedding space & is the vector space RDD where D) E No is the embedding size. Common approaches to learn fe include using an auto-encoding loss on the observation o (Burda et al., 2018 an inverse dynamics loss (Pathak et al. 2017). a multi-step prediction loss at the latent level (Guo) other similar representation learning methods. In particular, Pathak al.
ett al., 2020; 2022), (2017) and Badia et al. (2020b) highlight the utility of the inverse-dynamics loss to filter out noisy or uncontrollable features, e.g., on-screen death timer as in Pitfall!.
2

---

Published as a conference paper at ICLR 2024
A popular and principled approach exploration in discrete settings is to provide intrinsic reward inversely proportional to the visitation count (Strehl & Littman 2008 Azar et al. (017). However, in large continuous spaces the same state may be rarely encountered twice. Badia al. (2020b) remedy this issue by introducing a slot-based memory M, which stores all past embeddings in the current episode, and replaces discrete counts with sum of similarities between queried embedding e, = fo() and its k-nearest-neighhors (ex) under the kernel KC:
F1 (N(oo()
EmeNeiihh (e.) K(er, m)
Since storing the full history of embeddings throughout training would require prohibitive amount of space, this slot-based memory is typically relegated to short-term horizons only, and in NGU it is reset at the end of every episode. As consequence, slot-based memory must be combined with a separate mechanism capable of estimating long-term novelty; resulting in additional method complexity and trads-offs. In the following, we present a simple and efficient slot-based memory which can effectively track novelty over thousands of episodes.
3 RECODE
We will now introduce our method, Robust Exploration via Clustering- based Online Density Esti- mation (RECODE), to compute intrinsic rewards for exploration. RECODE takes inspiration from the reward of NGU (Badia et al. 2020b), but while NGU stores individual embedded observations in M and uses periodic resets to limit space complexity, RECODE controls its space complexity by aggregating similar observations in memory. This requires storing separate counter associated with each element in the memory and new observations need not be directly added to the memory, but will typically be assigned to the nearest existing element whose counter is then incremented. Since the counters never reset and the merged observations have a better coverage of the embedding space, RECODE's memory is much longer-term than simple slot-based approach, yielding state-of-the-art performance in many hard card-exploration tasks, It also simplifies the estimation of novelty to only one mechanism vs. two in NGU. Moreover, the RECODE architecture is highly flexible, allowing it to be easily combined with a variety of RL agents and most importantly different representation learning methods. As we show in the experiments, methods that can better leverage priors from learned representations, such as RECODE, outperform those that need to estimate novelty directly on raw observations, like RND (and in turn NGU). We now present more in detail RECODE, summarized in. Alg. |
Approximating visitation counts. Our estimator is based on a finite slot- container M- (m,)) J.M } where M is the memory size. We refer to m, E & as atoms since they need not correspond to single embedding as in BBdia et al. (2020b) a) We also store separate count vector such that e; an estimate of the visitation count of m,, particular, G does not only reflect the number of visits to m, but also captures any previous visit sufficiently close to it.
Given a new embedding e, we estimate its soft-visitation count (Alg. L as the weighted sum of all atoms close to in the memory, according to a similarity kernel:
Nx e) Σ, (1 + ci)K(m, e; dema).
In particular, we choose our kernel function as:
K(mm,e)
le-mu ed?
(1e- (1e-mili<dd...
(3)
where E R2, is a fixed parameter. Eq. is similar to Badia et al. (2220)), but we replace their sum over e's top-k neighbors with sum over atoms within dema distance from e. This choice prevents a counter-intuitive behaviour that can occur when using the k-NN approach with counts. In particular, it is desirable that the soft-visitation count of a given embedding should increase after adding it to the memory. However, adding atoms to the memory can change the k-NN list. If an atom displaced from this list has a large count, this might actually reduce nearby soft-visitation count estimates instead of increasing them. Conversely, our approach is not affected by this issue.
Finally, we return r as in Eq. (.. but add a small constant no to the denominator for numerical stability and normalize by a running estimate of its standard-deviation as in Burda et al. 2019
3

---

Published as a conference paper at ICLR 2024
Algorithm RECODE
I:
Input:
Embedding e, Memory M = {mi)i=1 atom visitation counts {a) number of neighbors
!M
relative tolerance to decide if a candidate new atom far K, squared distance estimate d de 's decay rate т, discount 7, insertion probability n, kernel function K, intrinsic reward constant no
2:
Output: Updated memory M [mu)]=1 updated atom visitation counts {or j updated squared
distance & intrinsic reward
3: Compute Nc(M,e) = Σ M (1 (1+ at) e)m
4: Compute intrinsic reward = (v (N + +no)
5: Find nearest It atoms to the embedding e: Neigh = {m,
Update dems estimate: dems < (1 (1-77 7) dBaa + 2 EmeNcigh((e) 1| e|12
Discount all atom counts ci 6 ct W {1, MM
8: Find nearest atom m, arg m...mmm ||m ella
9: Sample uniformly real number in [0, 1]: U0,
10: if [|m. <|12 > de.n and u < then
11: Sample atom to remove my with probability PG) x 1/c3
12: Find atom my nearest to my: my = arg ME Im my|l
13: Redistribute the count of removed atom: G + e ot
Insert at index with count 1: my t C, 99 t-
15: clse
16: Update nearest position m, c.+I mi c.+Ie
17: Update nearest atom count <- +
18:
end if
Building the memory. To build our memory we rely on the same aggregation principle we used to estimate soft-visitation counts, drawing a parallel between our atoms m; and the centroids of clustering of observations. We take inspiration from classical clustering and density estimation approaches such as je-means or DP-means Kulis Jordan (201 T): and adapt them deal with the challenges posed by our large scale RL setting: memory size limited and cannot store all past data, observations arrive sequentially, their distribution is non-stationary, and even the representation used to embed them changes
time. We now describe how RECODE tackles these problems.
At every step we must update the memory M to reflect the impact of seeing e on the soft-visitation counts, while keeping the size [M] fixed. Intuitively, two possible ways come to mind: either replace an existing atom with the new embedding, or update the position and count of an existing atom to be closer Let m, be the closest atom in M. We adopt the following rules (Alg. L8-188 to integrate new embeddings into the memory, which are closely related to the DP-means clustering algorithm Kulis & Jordan (2011:
If e satisfies |mm - ell2 < kde where dema is an adaptive threshold and k > 0 a fixed parameter, it "assigned" to the cluster encoded by m, and update m, 's value according to the convex combination of the counts of the existing embedding and the new one:
1
m.
m.
Cx +
Co +
Its weight e, is also incremented by 1;
If there no close-by atom, we randomly decide whether to create a new one by flipping coin with probability 17. If the coin-flip succeeds, we introduce the embedding as new atom, and we also remove an existing atom using a procedure described in the next paragraph. If the coin-flip fails, we instead update m, in equation
The random coin-flip is introduced to increase the stability of the clustering algorithm to noise. In particular, an embedding far away from the memory will be inserted only after it is seen on average 1/7 times, making one-off outliers less of a problem. At the same time, once far away embedding is observed multiple times and becomes relevant for the soft-visitation counts, there is high chance that will be added to improve the coverage of the memory. But to keep memory size finite, an existing atom must be removed. We investigate three different strategies select atom m, for removal
4

---

Published as a conference paper at ICLR 2024
FIA-!
p(arle
transformer stack
action prediction network Io
masked
inputs
6x
c/a
encoder
fo network
-G-i
Or
Op2+1
O,i Ot
Figure 2: Coupled Action-Statt Masking (CASM) architecture used for learning representations in partially observable environments. The transformer takes masked sequences of length l consisting of actions a, and embedded observations e; fo(o;) as inputs and tries to reconstruct the missing embeddings in the output. The reconstructed embeddings at time I-step action-predee citiionpredition classifier. The embedding function used as representation for RECODE is fa. Masked inputs are shaded in pink, N = 4 masked sequences are sampled during training (indicated by the stacks of o, e and 2 in the diagram).
1 and are then used to build
based on its cluster count (a) removing with probability + (b) removing with probability + (c) removing the atom with the smallest q, An ablation study over removal strategies in App. D.2 (Figures 8 and D. empirically shows that strategy (a) works best for the settings we consider, but also that results are generally quite robust to the specific choice.
Whenever an atom i is removed, its count is redistributed to the count of its nearest neighbor in order to preserves the total count of the memory. The update rule of RECODE can be also interpreted from the theoretical point of view clustering model. We provide a more detailed connection in App.
approximate inference scheme in latent DP-meam probabilistic
Dealing with non-stationary distributions. The distance scale between embedded observations can vary considerably between environments and throughout the course of training, as result of non-stationarity in both the policy and embedding function Sa- To deal with this issue, we include an adaptive bandwidth mechanism as NGU Badia al. (2020b). In particular, we update the kernel parameter & whenever a new embedding e is received, based on the mean squared distance of the new embedding to the k-nearest existing atoms (Alg. L.56). To allow for faster adaptation of dema- we replace the running average used in NGU with an exponential moving average with parameter r.
We note, however, that this mechanism is insufficient cope with non -taatiooonarity in fo over long timescales. The original NGU memory is not strongly impacted by this issue since it is re- set after every episode, leaving little time for the representation to change significantly. How- ever, in RECODE, these changing representations can end up corrupting the long-term memory if old clusters are not updated frequently. In particular, an atom might achieve a high count un- der representation, but become unreachable (and thus uscless) under different representation while still being unlikely to be removed. To counteract this we add decay constant 7 which discounts the counts of all atoms in memory at each step G yc, with y < 1 (Alg. [L7).
This effectively decreases the counts of stale atoms over time and increases the likelihood of their removal during future insertions: clusters that do not get new observations 'assigned" to them for a long time are eventually replaced. At the same time, relevant clusters are kept alive much longer than previous methods. Fig. re- ports the histogram of cluster ages for clusters contained in the memory of an agent that has learned how to reach Pitfall!'s s end screen. The red line in Fig. B denotes the maximum possible number of steps in an single episode, which is enforced by Pitfalll's s in in-game death timer, and would represent the maximum memory horizon for methods that reset their memory every episode. As
Mazimum Epiacde Length
10a
16a
10a
10a
10a
Cluster Age in Actor Steps
Figure 3: Content of agent memory when it
learns to reach Pirfall!'s end screen.
5

---

Published as a conference paper at ICLR 2024
see, most of the clusters much older than one episode, with earliest memories reaching back thousands of episodes. We consider the effect of discounting in more detail in App. D.2 (Figures [0] to 12 and 14). Importantly, we note that unlike NGU where each actor maintains its own copy of the memory, RECODE shares the memory across all actors in a distributed agent, which greatly increases the frequency of updates to each atom resulting in less representation drift between memory updates.
Tuning RECODE. While we introduced Alg. Din in its most general form, we observe experimentally that performance is robust w.r.t. most of the hyper-parameters introduced (see App. D. In particular, we note that the choice of discount 7 and memory size have the largest impact performance. All were chosen via coarse independent sweeps on two to three values and held constant across all experiments (see Sec. 5 App. for more details).
other
4 REPRESENTATION LEARNING METHODS
As discussed in Section 2 the choice of the embedding function fe O + E can have a significant impact on the quality of exploration; with many different representation learning techniques being studied in this context (Burda et al., 2018 Guo et al. 2020; 2022 2021, Erraqabi et al., 2021).
In the following, we focus on action prediction embeddings, introducing first the standard 1-step
prediction formulation (Pathak et al., 2017: Badia et al. 2020b a). The embedding function fo is parameterized as a feed-forward neural network taking ot, the observation at time I, as input. We define a classifier 96 that, given the embeddings of two consecutive observations fo(o:), fo outputs an estimate pe.ø (ae (0:, Oc+1) = ge (fo(o:) fo (or+1)) of the probability of taking an action given two consecutive observations (o,, 0(+1). Both fo and Is are then jointly trained by minimizing an expectation of the negative log likelihood:
min [(, (a:) = - -In( (ol 01+1)), 0,0
(5)
where a, is the true action taken between o, and 0 These embeddings proved be helpful in environments with many uncontrollable features in the observation (Badia et al. (0205), such as in Atari's ' Pitfall', where the observations contain many spurious sources of novelty even when the agent is standing still.
While RECODE can be used with an arbitrary embedding function, e.g. one tailored for the domain of interest, the choice of a meaningful representation is also key factor for the final performance. A major downside of the standard, 1-step action- predicciiniiiiionnn method is the simplicity of the prediction task, which can often be solved by learning highly localized and low-level features (e.g. how single object shifts under a transition), which need not informative of the global environment structure. In contrast, an ideal embedding should capture higher-level information about the environment, such as the agent's 's position or relative location of previously observed landmarks; which might not be simultancously present in the individual observations or and 0++1. In order to achieve this, wider context of time-steps may be needed.
However, the prediction task would become even easier if we simply provided the full trajectory to the predictor. In order to address this limitation, we propose to use a stochastic context, h,, where at each timestep k < either fa(ok) or dg-1 is provided [1] The main intuition being that the model can still predict o, by learning to infer the missing information from fo(or) given (h,-1, 11, at-1). In this way, the action predictor would not solely rely on the information provided by fo())). but would also construct redundant representations within fi.
From an implementation standpoint, we first build a sequence of observation embeddings and actions. (fo(oo)) fo fo(o))..... Gt-1, (o)).. Then, inspired by masked language models (Devlin et al. 2018). at each timestep f, we randomly substitute either fo()) or a with special token indicating missing information. These masked sequences are then fed to causally-masker masked transformer, whose output is then projected down to the size of the embedding (dim 2, = dim fo (ox)), and the difference between the two is input into a final MIP classifier Ja- As with 1-step action prediction, we train the representation using maximum likelihood. We refer to this approach as Coupled Action-Staat Masking (CASM) in the following. During training, we randomly sample multiple masked sequences per trajectory (N = 4) to help reduce gradient variance. Note that the final embedding that we
We avoid masking both fo():) and 0;-1 simultancously as this would increase the likelihood that the prediction task is indeterminable.

---

Published as a conference paper at ICLR 2024
tfail
00300
mortissume reverge
pttrate CEE
50
83002
130093
X020
60002
15300
89000
30300
60099
0000
40099
19000
6.0
7.5
33
4.5
166
1ef
160
166
suaquest
q
variur
haro
MONE 4000DE
003000
5005
40030
2003
MENE DMV
MSME-8LD
Haamen
0.2
1003
X0000
03
1.5
3.5
0.0
50
2.5
1.5
3.3
6.0
7.3
3.3
1.5
4.5
Number of framess
1c5
NUm22 of TiTRiss
160
Numbor framnss
168
Figure 4: Comparison of RECODE against other exploration bonuses on Atari's hard exploration games. All agents are based MEME and the same representation learning mechanism (AP). Note that the high variance in Q "bertt is due to a bug in the game that, when exploited, allows to obtain significantly higher scores (Chrabaszez et al. 2018).
provide to RECODE is e fo(o:), i.e. the transformer inputs, to avoid leaking information about the agent's 's trajectory. Figure 2 shows diagram of the architecture.
5 EXPERIMENTS
In this section, we experimentally validate the efficacy of our approach on two established benchmarks for exploration in 2D and 3D respectively: a subset of the Atari Learning Environment (ALE, Bellemare et al. 2013) containing eight games such as Pi Pitfall and Montezuma's 's. Revenge which are considered hard exploration problems (Bellemare et al. 2016); and DM-HARD-8 (Gulcehre et al.) 2019), suite of partially observable 3D games. All games pose significant exploration challenges such as very long horizons (O(10)) steps), the necessity to backtrack, sparse rewards, object interaction and procedural environment generation. Our method achieves ctate-of-the-ar results across both benchmarks and even solves two previously unsolved games: in Atari's Pitfall! our method is the first to reach the end screen and DM-HARD-8's Push Block we the first to achieve super- human performance. We also perform set of ablations to shed more light on the influence of the representation learning mechanism and the robustness w.r.t. noisy observations.
All candidate architectures evaluated in the following experiments (and in App. L). are composed of three main modules: (I) a base agent, responsible for core RL tasks such collecting observations and updating the policy, (2) an algorithm responsible for generating the exploration bonus, and (3) an embedding mechanism responsible for learning meaningful representations of observations. Our nomenclature reflects the choice of modules as SEEET-EEXPLORATTOIEEEEEDDD
For example, the MEME agent described Kapturowski et al. (2022) is denoted as MEME-NGU-AP. We use MEME agent across all experiments, but vary the exploration and representation mechanisms. For exploration consider EMM (pure episodic memory). , NGU and RECODE whereas for representation we experiment with AP and CASM. We provide full list of typee-parameters for all agents and baselines in App.
5.1
ATARI
The hard-exploration subset of Atari as identified by Bellemare et al. (2016) poses a considerable challenge terms optimization horizon with episodes lasting up to 27, 000 steps using the standard action-repeat of four. Additionally, rewards vary considerably in both scale and density. Across all our experiments in the Atari domain, we set the memory size of our agent to 5 10a atoms. We evaluate all agents following the regime established in prior work (Mnih et al.) 2015) Van Hasselt et al.) 2016) using 30 random no-ops, no 'sticky actions' (Machado et al. 2018 and average performance over 6 seeds. We compare the game scores obtained using exploration bonus, RECODE, against other methods while keeping agent architecture and representation mechanism fixed. The results presented in Fig. show that our method achieves state-of-the-an art, super- huper-human performance across all eight games while using a conceptually simpler exploration bonus compared to MEME-NGU-AP. The MEME-EMN EMM-AP and MEME-RND ablations in Fig. reveal the respective shortcomings of short-term
7

---

Published as a conference paper at ICLR 2024
Push Blooks
Throw Across
Basebal
Drawbridge
10.0
Ed0od 2000
2.0
100
105
160
lc5
Navigate Cubes
Well Sensor
Remember Sensor
Wall Sencor Stack
12.5
MIM--ROOOE COCM
MOME-MO NIME-N3--CISMM
Ej
2.5
MOME M03--P
VOTO BYGL Explon
2.5
fiamec
fremes
Number
frames
Figure 5: Performance of RECODE compared to NGU and BYOL-Explore on the single-task version of HM-HHRD-8. The BYOL-Explore results correspond to the final performance reported in (uoo et al. (2022) after 1e10 environment frames. All results have been averaged over 3 seeds.
and long-term novelty when used in standalone fashion. EMM its cannot solve Montezuma's Revenge because requires long-term memory. Conversely, RND on its own cannot solve Pitfall! because of the presence of many uncontrollable features in the observations and its inability to leverage the AP embeddings. In contrast, RECODE is able to leverage the AP representation for short- term and long-term novelty due to the clustering-basse -base memory integrating over long horizon which enables solving both games with a single intrinsic reward.
5.2 DM-HARD-8
DM--HARD- (Gulcehre al. 2019) consist of eight exploration tasks, designed to challenge an RL agent in procedurally--enenea rewards, and highly variable initial conditions. Each task requires the agent to interact with specific objects in its environment in order to reach a large apple that provides reward (cf. Fig. 16 in the Appendix for example). The procedural generation randomizes object shapes, colors, and positions at every episode. Across all our experiments in the DM-HARD-8 8 domain, we set the memory size of our agent 2 10 atoms. We also use the more powerful CASM representation over AP as the default in these experiments but present an ablation on the influence of the representation in Sec. 5.3 All performances reported for evaluation are averaged across three seeds.
3D worlds with partial observability, continuous control, sparse
We compare RECODE with NGU and the recently proposed BYOL-Explore (Guo et al. 2022) in this domain. The results presented in Fig. 5 show that our method is able to solve six out of eight tasks with super-human performance which sets new state-of-ine toitee-of-hh-aat fh-theerrt art on this benchmark and marks the first time that the human baseline has been beaten on Push Blocks. To control for the contribution of the representation, we also run version of NGU which uses the more powerful CASM representation instead of its default AP one. Switching AP with CASM improves NGU's performance significantly and stresses the importance of incorporating information over longer trajectories in the representation mechanism for this domain to combat the challenge of partial observability. However, only RECODE is able to take full advantage of the representational power afforded by CASM it is able to leverage it for both short-term and long-term novelty bonuses.
5.3 ABLATIONS
Concluding our experiments, we perform two ablation studies to gauge the sensitivity of our approach to the presence of noisy observations and the choice of the underlying representation mechanism.
Robustness to observation noise. Noise in the observation space is one of the most significant adversarial conditions exploration methods must to overcome to deliver utility for any practical scenario which always features imperfect sensors. The 'noisy TV problem' (Schmidhuber 2010 Pathak et al.] (017) is common metaphor which describes failure mode of exploration methods getting stuck on the prediction of noise as a meaningless signal of novelty. In order to assess our method's robustness w.r.t. observation noise, we construct noisy version of Montezuma's 's Revenge
8

---

Published as a conference paper at ICLR 2024
Notpy Marhourno
Meen Capped HNE on CM-HARD
19110
19/00
Capped INS
8000
RENIERECCEEAASM NEVE NDU
Humber
Number of fruines
1e6
Figure 6: Robustness observation noisc. Top: Performance of RECODE compared to NGU Noisy Montezuma. Bottom: A frame Noisy Montezuma where the noise is concatenated the original frame.
Figure 7: Comparing AP CASM on - for both RECODE and NGU.
by concatenating frame containing white noise in the range [0, 255] to the game's sume's original 210 x 160 greyscale observations along the image height dimension. We compare RECODE to NGU this setting using the same AP backbone to suppress uncontrollable noise on the representation level and assess the sensitivity of the exploration bonus to it. The results of this experiment are presented in Fig. 6 We find that the performance of MEME-NGU-AP deteriorates significantly in the presence of noise. This can be attributed to the fact that NGU relies on RND to compute the long-term exploration bonus, which degenerates to random exploration in the presence of uncontrollable noise (Kapturowski et al., 2018). This effectively restricts the baseline to short-term exploration within one episode. In contrast, RECODE's mean performance is not degraded significantly and achieves similar score in Fig. albeit with a higher variance.
Leveraging different representation mechanisms. The experiments on DM-HARD-8 demonstrate the importance of employing more powerful representation learning techniques in more complex, partially observable environments. However, while a richer representation often provides flat boost to downstream task learning, it cannot solve the exploration problem in itself. In Fig. 7 compare the contribution of AP and CASM to the aggregated performance of NGU and RECODE on DM--HAR--8. The results consistently demonstrate that CASM is superior representation to AP in this domain, leading to significant performance gains with both exploration methods. However, RECODE outperforms NGU for both representations, indicating that leveraging the representational power for both short-term and long-term novelty signals is a key benefit of our proposed method.
6 CONCLUSION
In this paper we introduce RECODE, a principled yet conceptually simple exploration bonus for deep RL agents that allows to perform robust exploration by estimating visitation counts from slot-based memory. RECODE improves over prior non-parametric exploration methods by increasing the effective memory span by several orders of magnitude using an online clustering mechanism. Our method sets a new statc-of-the-ar in task performance on two established exploration benchmarks, Atari's hard exploration subset and DM-HARD-8. It is also the first agent to reach the end screen in Pitfall! within the time limit which exemplifies RECODE's s efficiency of leveraging both long-term (i.e. previous experience) and short-term (i.e. within an episode) novelty signals. Beyond the benchmarks, RECODE's s performance also remains unaffected by noisy observations an adversarial condition which significantly degrades prior approaches such RND and NGU. Additionally, method is agnostic to the concrete representation technique chosen for embedding the observations and scales well with increasingly powerful representations, e.g. using multi-step sequence prediction transformers like our proposed CASM architecture. However, RECODE is still limited by the choice of the representation and cannot by itself overcome deficiencies stemming from an inappropriate state representation. We also acknowledge that the controllability prior chosen for CASM is a strong assumption suitable for the video game environments we experimented with, but this might need to be revisited when RECODE is deployed in more realistic, real-world domains. Further details on those limitations are provided in Appendix B In conclusion, we believe that RECODE can serve as simple yet robust drop-in exploration method compatible with any RL agent and representation learning method which directly translates improvements in representation learning to improvements in exploration performance.
show that our

---

Published as a conference paper at ICLR 2024
REFERENCES
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pp. 39-1. JMLR Workshop and Conference Proceedings, 2012.
Jean-Yver Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In COLT, pp. 41-53. Citeseer, 2010.
Peter Auer, Nicolo Cesa- Desa-Biannhhi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. Proceedings of the 34th International Conference on Machine Learning-Vonn Voluuee 70, pp. 263-272. JMLR. org, 2017.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020b.
André MS Barreto, Doina Precup, and Joelle Pineau. Practical kernel- based reinforcement learning.
The Journal of Machine Learning Research, 17(1):2372-2441, 2016.
Marc Bellemare, Joel Veness, and Erik Talvitie. Skip context tree switching. In International conference on machine learning, pp. 1458-1465 1466. PMLR, 2014.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479 2016.
Mare G Bellemare, Yavar Naddaf. Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Ronen Brafman and Moshe Tennenholtz. R-max general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research, 3:213-231, 2003.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 04355, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Seventh International Conference Learning Representations, pp. 1-17, 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. Back to basics: Benchmarking canonical evolution strategies for playing atari. arXiv preprint
2018.
Mayank Daswani, Peter Sunehag, and Marcus Hutter. Q-learning for history-based reinforcement learning. Asian Conference on Machine Learning, pp. 213-228. PMLR, 2013.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:
2018.
Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. Kernel-based reinforcement learning: A finite-time analysis. In Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Muchine Learning Research, pp. 2783-2792. PMLR, 2021a.
Omar Darwiche Domingues, Corentin Tallec, Rémi Munos, and Michal Valko. Density- bonuses on learned representations for reward-free -free exploration in deep reinforcement learning. In ICML 2021 Workshop, 2021b.
10

---

Published as a conference paper at ICLR 2024
Akram Erraqabi, Mingde Zhao, Marlos C Machado, Yoshua Bengio, Sainbayar Sukhbaatar, Ludovic Denoyer, and Alessandro Lazaric. Exploration-driven representation learning in reinforcement learning. In JCML 2021 Workshop Unsupervised Reinforcement Learning, 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Caglar Gulcehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making efficient use of demonstrations solve hard exploration problems. In International conference on learning representations, 2019.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean- Mann-Battien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In International Conference Machine Learning, pp. 3875-3886. PMLR, 2020.
Zhaohan Daniel Guo, Mohammad Gheshlagi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Geometric entropic exploration. arXiv preprint arXiv:2101.02055 2021.
Zhaohan Daniel Guo, Shantanu Thakoor, Miruma Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade. Daniele Calandriello, lean-Bastien Grill, Yunhao Tang, et al. Byol- explore: Exploration by bootstrapped prediction, arXiv preprint arXiv:2206.088:. 2022.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691 2019.
Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probabil- ity. Springer Science & Business Media, 2004.
Mareus Hutter et al. Feature reinforcement learning: Part unstructured MDPs. De Gruyter Open, 2009.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11 (A(Ap)):15331160 2010.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. International Conference on Machine Learning, pp. 4870-4879. PMLR, 2020.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99-134, 5 1998. ISSN 00043702. doi: (0006600004437722280000023333330
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018
Steven Kapturowski, Víctor Campos, Ray Jiang, Nemanja Rakićević, Hado van Hasselt, Charles Blundell, and Adrià Puigdomènech Badia. Human-level atari 200x faster. arXiv preprint arXiv:2209.07550. 2022.
Michael Kearns and Satinder Singh. Ncar-optimal reinforcement learning in polynomial time. Machine learning, 19(2-3):209-233) 232, 2002.
Brian Kulis and Michael I Jordan. Revisiting k-means: New algorithms via bayesian nonparametrics. arXiv preprint arXiv: / / 2011.
Branislav Kveton and Georgios Theocharous. Kernel-basee reinforcement learning on representative states. In Twenty-Sixth AAAI Conference Artificial Intelligence, 2012.
11

---

Published as a conference paper at ICLR 2024
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 102118849--888 2021.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Andrew McCallum. lastence-based utile distinctions for reinforcement learning with hidden state. In Machine Learning Proceedings 1995, pp. 387-395. Elsevier, 1995.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 2015.
Dirk Ormoneit and Šaunak Sen. Kernel-based reinforcement learning. Machine Learning. 2002.
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Volinn Volume 70, pp. 2721-2730. JMLR. org, 2017.
Emanuel Parzen. On estimation of probability density function and mode. The Annals of Mathe- matical Statistics, 33, 1962. ISSN 0003-4851. doi: 10.
177704472.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Guriosity-driver exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16 17, 2017.
Jason Pazis and Ronald Parr. Pac optimal exploration in continuous space markov decision processes. In Twenty-Seventh AAAI Conference on Artificial Intelligence, 2013.
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self- reinforcement learning. arXiv preprint arXiv: 2019.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. ICML, 2017.
Martin I. Puterman, Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Murray Rosenblatt. Remarks some nonparametric estimates of density function. The Annals of Mathematical Statisties, 27, 1956. ISSN 0003-4851. doi: 10. 10.1214/amm/1117777 17777288900
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference Learning Representations, 2018.
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, 0e////11..55995 2015.
Jürgen Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010). JEEE Transactions on Autonomous Mental Development, 2(3):230-22) 2010.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pp. 9443-9454, PMLR, 2021.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8): (309):130911311 2008.
12

---

Published as a conference paper at ICLR 2024
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke. OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurek, and Pieter Abbeel. #Exploration: A study of count- based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017. URL https:: https::/proceeddnggs cot http:///..o.ursscc/////////////0 / 1a20fffaaaaffaa11526700003366222
Ruo Yu Tao, Vincent François-Lavel, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Information Processing Systems, 33:81 14-66166, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
In this section, we give a brief and non-exhaustive overview of past works computing visitation counts or estimating densities in RL. We classify them either parametric or non-parametric.
Bellemare et al. 2016) and Ostrovski et al. (017) propose to com- pute pseudo-visitation counts using density estimators on images such as Context Tree Switching CTS et al., 2014) or PixeICNN (Van den Oord et al. 2016). On the other hand, Tang et al. (2017) use locality-sensitive hashing to map continuous states to discrete embeddings, where explicit visitation counts are computed. Some methods such as RND (Burda et al. 2019) can be interpreted as estimating implicitly the density of observations by training a neural network predict the output of randomly initialized and untrained neural network which operates on the observations. Hazan et al.) (2019); Pong et al. (2019): Lee et al. (2019); Guo et al. (2021) propose algorithms that search a policy maximizing the entropy of its induced state-space distribution. In particular, the loss optimized by Guo et al. (2021) allows to compute a density estimate as well as maximizing the entropy. Finally, Domingues et al. (2021b) computes density estimation top of learned representations, which are inspired by bonuses used in reward-free finite MDPs.
Parametric methods.
Non-parametia methods. Non-parametric density estimates that we build date back to Rosen blan (956); Parzen (1922) (Parzen-Rosenblat window) and are widely used in machine learning as they place very mild assumptions the data distribution. Non-parametric, kernel-based approaches have been already used in RL and shown to be empirically successful on smaller-scale environments by Kveton & Theocharous (2012) and Barreto et al. (2016) and are theoretically analyzed by Or moneit & Sen (2002); Pazis & Parr (2013): Domingues et al. (2021a). .). In NGU (Badia et al. (020b), al. (022),
non-parametric approach is used to compute short term reward at the episodic level. Liu & Abbeel] (2021) propose an unsupervised pre-training method for reinforcement learning which explores the environment by maximizing non-parametric entropy computed in an abstract representation space. The authors show improved performance on transfer in Atari games and continous control tasks. [Soo et al. (202I) use random embeddings and non-parametric approach to estimate the state-visitation entropy. but do not generalize to concurrently learned embeddings. [Tao et al. (2020) show that K-NN based exploration can improve exploration and data efficiency in model RL. While non-parametric methods are good models for complex data, they come with the challenge of storing and computing densities on the entire data set. We tackle this challenge in Be the main text by proposing method that estimates visitation counts over a long history of states, allowing our approach to scale to much larger problems than those considered in previous works, and without placing assumptions on the representation, that can be trained concurrnnlyy with the exploration process and doesn't need to be fixed a priori.
Agent57 (Badia et al., 2020a) and МEME (Kapturosski
B LIMITATIONS
While Atari possesses several challenging sparse reward tasks, the observations are quite simplistic and there is little variation in the environment between episodes. DM-HARD-8 has much richer observations and several procedurally generated elements such as colors, object shapes, and initial position and orientation; but the level layouts are essentially static. It would be important to understand what additional challenges might be encountered with more complex procedurally gelerated elements, and if different representation learning methods might be needed to obtain good generalization across episodes this setting.
Also, we are aware that concurrent training of embeddings can place some restriction on the maximum discount y that can be effectively used. While this does not limit our method on commonly used RL. benchmarks, where episodes tend to be relatively short (c.g. Atari maxes out 27, 000 steps for the standard action repeat of 4), could potentially be an issue for environments with significantly larger timescales, or if there were many more options for what the agent could try between episodes.
RECODE can be thought of as a general solution to the exploration problem in RL by biasing the policy towards uniform coverage of the representation space. If the representation does not allow for aliasing together semantically similar states (c.g. if the representation is tabular), exploration can become intractable in large scale environments, since most observations encountered throughout training
14

---

Published as a conference paper at ICLR 2024
unique. As such, RECODE does not obviate the need to up with meaningful representation for the environment at hand. CASM relies on a controllability prior that we find to be well-suited to the RL environments considered in this work, but the question of determining which priors useful for more general classes of environments remains largely open.
C GENERAL NOTATION
We consider the usual Reinforcement Learning setting, where agent interacts with environment to maximize the sum of discounted rewards, with discount y E [0, 1), as in Sutton 1988. In Partially-Observable Markov Decision Process
particular, the environment can be described (POMDP) Kaelbling et al. (1998). First, we define a Markov Decision Process (MDP) through a tuple (S, A, T, R). where is the set of states, A the of possible actions, T transition function, which maps state-actions to distributions over next states, and R S x A + IR is the discrete- time interaction process McCallum (995); Hutter (2004); Hutter et al. (2009); Daswani et al. (2013) between agent and MDP, the agent does not receive a state from S, but
reward function. In particular, Markov Decision Process
its environment. In Partially-Obeevall an observation E Ω2 where O is the function mapping unobserved states to distributions observations. An observation will only contain partial observations about the underlying state E S. This function can be combined with intrinsic reward function r; enable the exploratory behavior. The environment responds to an agent's action E A by performing a transition to a state s' ~ s- a); the agent receives new observation o' (t(1s') and reward i, we can indicate with h4 foo, 01 .1...... Gr. 0z E He the history of past observations-actions where H₂ H4-1 A O, Ho O and the overall history space is H ULEN Hg. We consider
R(s, a). At step
policies H. + A that map history of past observations-action to probability distribution over actions.
D RECODE FROM A CLUSTERING POINT OF VIEW
The update rules RECODE's memory structure in Algorithm of the main text can be interpreted as an approximate inference scheme latent probabilistic clustering model. We explore this connection here as means to better understand and justify the proposed algorithm as density estimator. The rule has a close connection to the DP-means algorithm Kulis & Jordan (201I), with two key differences:
the counts of the cluster-center are discounted at each step, allowing our approach to deal with the non- stationarity of the data due to changes in the policy and the embedding function, effectively reducing the weight of stale cluster-centers in the memory,
when creating cluster-center, remove underpopulated one, so to keep the size of the memory constant.
The adaptations are necessary to accommodate the additional complexities of our setting, which follows streaming protocol (i.e. data must be explicitly consumed or stored as it arrives, and data that are not stored cannot be accessed again) and is non-stationary (i.e. data are not assumed to be identically distributed as time advances). The clustering algorithm resulting from these adaptations shown in Algorithm?? 2 RECODE implements such an algorithm to update the memory, and also calculates intrinsic reward for the observed embedding e, as described in Section 3
15

---

Published as a conference paper at ICLR 2024
Algorithm 2 A streaming clustering algorithm.
1: Parameters:
Number of clusters |
Number of nearest cluster centres k
Discounting of counts at each step
Distance threshold to propose the creation of new cluster x.
Probability of accepting the creation of new cluster !
2: State:
Threshold to create new cluster (i.e. average cluster distance) 0 Cluster centres 0
vi E 1... |M]
Cluster counts Q 0
1l e [M]
Indices of n-areestt neighhours of point e: (eigh((e)
3: Implementation:
4: for all received embedding E (co, C1, e..... do
5: Update average inter-cluster distance <- (1 r)d + 1. EieNeigh, [|m; c12
6: Discount all cluster-center counts Q +- 7 v1 El 1E.,...,IM
7: Find index of nearest cluster center m, = arg min ||m - c||2
8: if [|m; ell2 > y. and with probability then
9:
Sample index of cluster center to remove with probability P(j) 1/ej
Find index of nearest cluster center to mj: m; = arg min MiLy ||m m5l2
11:
Redistribute the counts of removed cluster center: + Cj + +
12:
Replace cluster with the new cluster at m, + e, Cj +-1 else
13:
c. Update nearest cluster center m, c. Hi + c.t1
15:
Update nearest cluster-center count C. <- Cx +
16:
end if
17:
end for
PV)
PV 1102
begmindcll
75
Figure 8: Effect of removal strategy on evolution of cluster centers and counts (with counts corre- sponding to the size of the marker). At each timestep we sample batch of 64 2D-embeddings from a square of side min t). After / = 100 the distribution remains stationary and we would like the distribution of cluster centers and counts be to become approximately uniform after enough time has passed. For a deterministic removal strategy which selects the clusters with the lowest counts, the cluster centers can remain skewed long after the distribution has stopped changing. For both probabilistic removal strategies, the cluster centers become approximately uniform, but only for the 1/e3 removal strategy we observe that both ceenter and counts become uniform. Note that we use discount of 7 0.9999.

---

Published as a conference paper at ICLR 2024
D.1 ADDRESSING FINITE- MEMORY LIMITATIONS.
We first address the modifications introduced to deal with the memory limitations of the streaming setting: 1) each datum (embedding et in our notation) is incorporated into cluster distribution approximation once, then discarded; 2) the total number of clusters is stochastically projected down onto an upper limit on the number of clusters (otherwise they would grow without bound- albeit progressively more slowly). Both modifications allow our method to maintain constant space complexity in the face of an infinite stream of data.
The step-wise justification of the Algorithm is relative straightforward. At step t, for embedding et, we show that the following objective minimised:
min IEl...., [M]
|| -
eel1
s.t.
Im - eel1 - S Rd
Working backwards: updating the cluster center reduces the objective directly and will not violate the constraint (unless it was already in violation; this excluded in the precondition of this branch). This accounts for the "else" branch. The "if" branch introduces a new cluster center precisely et, thus equation i minimised completely: it is zero for this branch. Finally, selecting the index of the nearest cluster center directly minimises placement of the branch according to equation 6 ignoring the constraint (which is latest ensured by the "if/else").. Note that the hard constraint of equation takes the place of the soft cluster penalty of DP-means (Kulis Jordan 2011).
The updates to the cluster centers. unlike k-means and DP-means, are done in an exponentially- weighted moving average of the embeddings, rather than as global optimisation step utilising all of the data. Consequently, and importantly, what happens to equation 66 evvallude for es, where 8 & t, of significant interest, as objectives for k-means and DP-means account for all data, rather than a single datum.
We tested the qualitative behavior of different removal strategies in Fig. This study suggested that a stochastic removal of a cluster with probability x c=2 was more stable and better tracked a non-stationary distribution. The intuition we got from these toy examples is also confirmed in ablation experiments on the Atari environment, as shown in Fig. 9 where compare RECODE runs with different removal rules.
monrezuma, revange
O tfall
arivete.eg eye
seleris
32302
500002
103000
1/W33
7507
61000
60500
5002
20000
0.0
1.5
3.0
4.3
6.5
25
0.5
1.5
3.0
0.5
6.0
2.5
0.3
1.5
3.0
4.5
64
7.5
0.0
1.5
3.0
4.5
64
25
obert
3.4
3.0
2k:
recope
550000
432000
2500
300000
2300
235000
0.2
1500
3.0
Kattbber of Erariss
1est
Numberr fratress
1ect
Nuitiber of Tratruns
1eft
Figure 9: Effect of removal strategy on performance. All choices of removal strategy considered result in viable algorithm, but there some environments (most notably Pitfall!) where the chosen strategy of 1 fc² appears to be more robust.
D.2 DEALING WITH NON-STATIONARY EMBEDDING DISTRIBUTIONS.
We now turn to the question of how to deal with the non-stationarity of the embedding distribution. We introduced the following modifications deal with the non-stationarity of our data stream:
1. the cluster count decays,
2. two clusters be merged to accommodate new one,
17

---

Published as a conference paper at ICLR 2024
3. the use of an exponentially weighted moving average update of cluster centers.
In k-means, all of the data are retained. This makes k-means costly: each step of fitting the entire data set is examined to update the cluster assignments and update the cluster means. Instead, we take distributional approximation to the data associated with each cluster, and when re-adjusting cluster assignments according to equation 6 we do so in terms of this distributional approximation.
In particular, each cluster is approximated by Gaussian distribution with precision and whose mean is unknown but with prior zero and precision 1. Specifically:
A4t ~ -~(0,c
eilp ~ (N4,
where N M(H 7) denotes Gaussian (or normal) distribution with mean μ and precision 7 (precision is the inverse variance). Since the prior on lr is conjugate to the likelihood on e₁, we know that the posterior on my will have the form N (pt, q). Updating this posterior with a single embedding e has the form:
C1
M
+ VI
C1 + c/.
C - 4-4+1
This is precisely the update Algorithm
Note that in this model, the counts are also the precision parameters of the distribution, representing the inverse spread (or the concentration) of each cluster. At each step of Algorithm 2 these counts are decayed. Effectively, this causes the variance of the distribution representing each cluster to spread out: thus at each time step, each cluster becomes less concentrated and more uncertain about which data points belong to it. The hyperparameter "y captures the rate of diffusion of all clusters in this manner. This uncertainty increase applied at each step acts as a "forgetting" mechanism that helps the algorithm to deal with changing data distribution.
Cluster re- re-sampling. as already justified for e above in terms of equation ensures that the number of clusters is bounded by [M]. There two details to examine: what merged, and how it is merged. As Cj + O, the probability assigned by the Gaussian likelihood of cluster j to any new datum approaches zero also, thus the cluster with the lowest counts likely to have the least impact on future density estimates (as it is most diffuse). When Cj >> 0, however, it is not clear which cluster should be removed. Therefore, we stochastically select which cluster remove with probability inversely proportional to the square of the counts (using the square of the counts emphasizes small differences in counts more than 1/e,). The cluster could potentially be removed completely, but we instead choose to re- re-assign its counts to the nearest cluster as we experimentally found this strategy to be less sensitive to the choice of hyperparameters.
To help build some intuition about the effects of the discount factor, we illustrate its effects on toy example with a non-stationary embedding distribution in Fig. TO We find that tuning the discount ? allows to smoothly interpolate between short-term and long-term memory.
Figure 10: Non-stationary density estimation using RECODE on a toy example. For step O,..., 100, we sample a batch of 64 -D-eemddddgggs uniformly from the square of side Vt. The support of the embedding distribution therefore expands over time to simulate non -ta-statoonary distribution similar to the distribution of states visited by an RL agent over course of exploration. We plot the atoms learned by RECODE with size proportional to their count. We find that for a small enough discount, RECODE exhibits a short-term memory, accurately approximating the distribution of the final distribution. As we increase the discount, RECODE exhibits a longer-ten terger--ermmnne memory, approximating the historical density of states, as can be seen by the concentration of probability mass in the bettom-left corner.
18

---

Published as a conference paper at ICLR 2024
To confirm the practical necessity of cluster discount, perform additional ablations the Atari environment. If we don't 't train the representation during the exploration, but start with pretrained one, we see that RECODE However, as shown Fig- 11 as soon as we also train the representation at the same time, the ability to forget old observations allows compensate distribution-shiftt and achieves quite better performance.
perform reasonably well also without using discount (see Fig. [2]
manreruma_
tial
privateeerv
120003
102000
30002
53300
30200
52300
1001
600800
90200
2E000
1:8
108
108
Ice
A06
Scaqquest
obert
RECODE Discount
3500
Haman
3.5
350002
2300
2.4
100000
332000
5C000
100000
3.0
6.5
2.5
3.0
4.5
2.5
Kumber
160
Figure 11: Effect of discount on performance. As embeddings evolve throughout the training it may happen that older clusters stop being meaningful under the current representation. Deactivating the discount (i.e. 1) results in significant degradation in performance, especially in hard-exploration settings like Montezuma's 's Revenge and Pitfall!.
Monitezuma
RECODE AP No Discount Pretrained
12000
10000
5000
Eploode
6030
4000
0.0
3.3
3.0
4.5
6.0
7.5
Number of Freemee
168
Figure 12: Pretrained vs concurrently-traine embeddings and sensitivity to discount. We take snapshot of the embeddings at 400M frames and then the agent is trained again from scratch with these frozen embeddings. We find that we can achieve similar performance even with a = 1 (i.e. no discount). Interestingly, also observe that the agent achieve much higher scores than those the original agent had achieved at the time the snapshot was taken.
E ANALYZING EXPLORATION WITH RECODE
In this section, we present a simple example to show a simple example to illustrate how the exploration process unfolds for RECODE. We use a variant of the Random Disco Maze: grid-world environment proposed in Badia al. 2020b to show the the importance of estimating the exploration bonus using a controllable state representation, depicted in Fig. 13 (left). The agent starts each episode in fixed position of fully observable maze of size 21x21. The agent take four actions (left, right, up, down). The episode ends if the agent steps into a wall, reaches the goal state or reaches a maximum of 500 steps.
Crucially the environment presents random variation in the color of each wall fragment at every time step. Specifically, the color of each wall fragment randomly and independently selected from set
19

---

Published as a conference paper at ICLR 2024
of five possible colors. This introduces great deal of irrelevant variability into the system, which presents a serious challenge to exploration bonus methods based on novelty. The reason for this is that the agent will never the same exact state twice, the colors of the wall fragments will always be different each time step.
We RECODE for 100 million steps. The agent is able to find the goal after collecting around 50 to 60 million steps. Fig. [3 (right) shows how the distribution of clusters changes as the agent explore this environment. As the agent always starts in the same position (bottom-left corner of the maze), the distribution is heavily skewed towards over-representing this points, We can see that as time progresses the cluster centers uniformly cover all the maze.
geal
Figure 13: (Left) Random Disco Maze (Right) Evolution of the distribution of the clusters learned by RECODE over time, see text for details.
We investigate how far back the memory of RECODE goes in Montezuma's Revenge. The results are shown in Fig. 14 We find that the distribution of the age of the clusters learned by RECODE (i.e. how many steps ago each atom has been inserted in the memory) exhibits mode around 10 actor steps, which corresponds hundreds of episodes, with significant number of clusters times older than that. We remind that NGU's short-term non-parametric novelty estimated at most one episode (red line in the Figure).
Loreeel lipoode Lengh RECOCE
32"
Couft
107
10
159
Steps
Figure 14: Age distribution of the clusters learned by RECODE on Montezuma's Revenge. RECODE's memory horizon spans much more than single episode. We set 0.999 in the experiments of Fig. 4 We indicate red the average length of an episode, showing that in this setting, RECODE's memory reaches back thousands of episodes.
20

---

Published as a conference paper at ICLR 2024
Table 1: Atari Hyper-parameters.
Parameter
Value
RECODE memory size
5 x 10*
RECODE discount Y
0.999
RECODE insertion probability
0.05
RECODE relative tolerance R
0.2
RECODE reward constant
0.01
RECODE decay rate т
0.9999
RECODE neighbors l
20
IM Reward Scale BIM
1.0
Discount
0.9997
Min Discount
0.97
Replay Period
80
Trace Length
160
Replay Ratio
6.0
Replay Capacity
2 x 10t trajectories
Batch Size
RL Adam Learning Rate
3 x 10-4
Emedding Adam Learning Rate
x 10
RL Weight Decay
0.05
Embedding Weight Decay
0.05
RL Torso initial stride
4
RL Torso num blocks
(2, 3,
RL Torso num channels
(64, 128, 128, 64)
RL Torso strides
F HYPER-PARAMETERS AND COMPUTATIONAL RESOURCES
We implemented RECODE and all the baseline agents and novelty mechanisms in distributed setting (see App.Gr and and H). We report here the total computational infrastructures used by each distributed agent, (including multiple actors, learner and RECODE memory mechanism where applicable). One seed for Atari experiments (e.g., for SEMEERECODEEAA and MEME-NGU-A execute using multiple servers with total of 64 CPUs, ITB RAM, and 5 TPUv4, One seed for DM-HARD-8 experiment (e.g., for NEME-RECODE--ASS and MEME-NGU-CASM took 90h to execute using multiple servers with total of 512 CPUs, ITB RAM, and TPUv4.
took 24h to
We also report here the precise hyper-paramete values used in our experiment, Table | for Atari and Table 2 for DM-HARD-8 We omit hypers which do not differ from the base MEME agent Kapturowski al. 2022).
We emphasize that the relevant hyperparameters for RECODE, i.c. those for which the algorithm is sensitive changes, are only
RECODE memory size
RECODE discount
See the previous Appendix D and the main text for the discussion of their interpretation. The other parameters are particular choices, for which the algorithm proved to be robust in many different environments, and we did not need to retune them. We permutations of r E 0.05, 0.2 and memory size E [5 x 10 2 x 10 in both Atari and D and found that performance in most environments had little sensitivity to these choices.
additional experiments sweeping across all
21

---

Published as a conference paper at ICLR 2024
Table 2: DM-HARD-8 Hyper-parameters.
Parameter
Value
RECODE memory size
2 10
RECODE discount Y
0.997
RECODE insertion probability 7
0.2
RECODE relative tolerance
0.2
RECODE reward constant
0.01
RECODE decay rale т
0.9999
RECODE neighbors k
20
IM Reward Scale 9IM
0.1
Max Discount
0.997
Min Discount
0.97
Replay Period
Trace Length
80
Replay Ratio
2.0
Replay Capacity
5000 trajectories
Batch Size
128
RL Adam Learning Rate
1 x 10-1
Embedding Adam Learning Rate
3 x 10-
RI Weight Decay
0.1
Embedding Weight Decay
0.1
RL Torso initial stride
2
RL Torso num blocks
(2, 4, 12, 6)
RL Torso num channels
(64, 128, 128, 64)
RL Torso strides
(1, 2,,
Table 3: CASM Hyper-parameters.
Parameter
Value
Transformer Type
GatedTransffrmerxI
State Mask Rate
0.8
Num Masks Per Trajectory
Action Embedding Size
32
Num Layers
Attention Size
128
Num Attention Heads
MLP Hidden Sizes
(512.)
Predictor Hidden Sizes
(128.)
G ARCHITECTURE OF A DISTRIBUTED AGENT USING RECODE
We now detail how RECODE can be efficiently integrated in typical distributed RL agent (Espeholt et al. 20188 Kapturowski et al. 2018) that comprises several processes that run in parallel and interact with each other, allowing for large-scale experiments. Classically, a Learner performs gradient steps to train policy and an embedding (representation) function fo. forwarding the parameters 0 to an Inference Worker. A collection independent Actors query the inference worker for actions that they execute in the environment and send the resulting transitions to the Learner, optionally through a (prioritized) Replay (Mnih et al. 2015 Schaul et al. (015). When using RECODE, the Actors additionally communicate with shared Memory implementing Algorithm [ at each step t, they query from the Inference Server an embedding fe(hs) of their history and send it to the shared Memory which returns intrinsic reward rı that is then added to the extrinsic reward to train the policy in the Learner process. A diagram giving an overview the typical architecture of a distributed agent using RECODE given in Figure 15
22

---

Published as a conference paper at ICLR 2024
Replay Queue
Transbonn
Transitions
Embediding A00d
Learner
Actors
Memory
Pasmmttes
Hozorice
Ennosskiie ap (rhh..
Inference
Worker
Figure 15: Overview of the architecture of a distributed agent using RECODE.
H AGENT TAXONOMY
All methods evaluated in the experiments in the main paper and the extended experiments in App. are composed by three main components.
A base agent that oversees the overall RL learning process (e.g., executing actions and collection observations, computing adjusted returns, updating the po.icy, We focus MEME (Kapturowski et al. 2022), recent improvement over Agent57 (Badia et al. 2020a) that achieves much greater sample efficiency and is the current state-of-the-art Atari, and a VMPO- agent (Guo et al.) 2022) that is the current state-of-the-ars on DM-HARD-8.
A representation learning mechanism generate observation embeddings which fed to the intrinsic reward generator. We consider both Action Prediction (AP) and CASM embeddings. Note that some intrinsic reward modules cannot make effective of the representation learning module (e.g., RND), while others merge both second and third module in a single approach (e.g., BYOL-Explore)
An algorithm to generate intrinsic rewards. In addition to RECODE, we also consider the recent BYOL-Explore Guo al. 2022), NGU Badia al. (2020b) and NGU's building blocks, RND Burda et al. (2019) and Episodic Memory (EMM) Pritzel et al. (2017).
For example, in more detailed taxonomy the original MEME agent described in Kapturowski et al. (2022) is denoted as the MEME-NGU-AN baseline, and compared against our novel JEME-RECODE-AR agent where the only modifications is the changed exploration reward. Table reports more details on all combinations available present in our experiments.
Table Taxonomy of agents used in the experiments.
Agent name
Base agent
Intrinsic reward
Representation learning
MEME-HGU-AP
Kapturowski al. 2022
MEME
NGU
AP
MEME-RID
(ablation)
MEME
RND
N/A
MEME-EMM- -MP--E
(ablation)
MEME
ENM
DOME-RNDDnAAP
(ablation)
MEME
RND
(b)
MEME-RECODE-AP
(this paper)
MEME
RECODE
HEME-RECODE-CACSS
(this paper) (ablation)
MENE
RECODE
CASM
SEME-HGU-CASE
MEME
NGU
CASM
MMPO-BYOL-Explos
Guo et al. 2022
VMPO
BYOL-Explore
WXOL-Explore (c)
(a) As in the original paper RND takes as input raw observations.
(b) To test RND ability to cope with non-stationary representations, we truin an AP encoder concurrently with the policy and use to create embeddings of observations that are in RND (i.c., running RND on top of AP). (c) The BYOL-Explore mechanism internally trains neural network predict the dynamical evolution of the observations. This provides the agent with both reward/novelty signal (prediction error) as well embedded representation of the observations (that can be extracted from the last few layers of the network).
23

---

Published as a conference paper at ICLR 2024
I EXPLORATION IN THE DM-HARD-8 ENVIRONMENT
The lireesdameniional tasks in DM-HARD-8 can have extremely large state space to explore. Consider for example the Baseball task, as shown in Fig. I6 the agent needs to look at the scene, find the bat, pick it up, throw the ball down, pick up the ball and be able to get the apple.
1st frams
for bii
Pick bat
Throw ball down
Pick
Place
Figure 16: First-person-view snapshots of an agent solving the DM-HARD-8 Baseball task. Images are ordered chronologically from left to right and top to bottom. Each image depicts specific stage of the task. The agent must interact with specific objects the environment in order solve the task.
J AGGREGATED RESULTS
In this section, show the aggregated results over all different environments, both for the Atari suite and for DM- To ensure that no single environment dominates due to larger reward scales we the Human Normalized Score (Mnih al., 2015) each environment, and then cap scores above 100% prior to averaging.
As Fig. 17 shows, (Leff), the uncapped simply an artifact the high variance present in **bert. This variance arises due to a bug in Q 0*bert, which allows for much larger scores to be obtained if exploited.
swing significantly time, which in this case is
Mean HNS over Atari hard exploration games
Capped exploration games
HNS
600
Capped
400
200
NEME-NGU-M
0.0
1.3
Number of frames
3.0
4.5
6.0
7.5
leB
0.0
1.5
Number Trames
5.0
4.5
R.D
7.5
Figure 17: Aggregated results Atari. (Left): Mean Hlmann-Normaliza Scores of MEME-RECODE-AN AP compared to MEME-NGU-AP on Atari games. (Right): Capped -ooman--d Scores.
In Fig 18
we present our main results aggregated over all environments in each DM-HARD-8 task suite. Table summarizes the results and compares the performance of the RECODE novelty reward mechanism with that of NGU. We emphasize how effective our approach is when applied to three- dimensional environments like DM-HARD-8, if compared to alternatives like BYOL-Explore.
24

---

Published as a conference paper at ICLR 2024
Mean Capped HNS on DM-HARD-8
80
70
Capped HNS
20
30
MEME-RECCDE-CASM VMPO BYOL Explore
20
Number of frames
1e9
Figure 18: Aggregated results on DM-HARD-8. Mean Capped Human Normal-nr Scores of MEME- RECODE-CASM are compared to IMPO-BYOL-Expl-1
Table 5: Aggregated results over DM-HARD-8 environment tasks.
Game
Human
MEME-NGU-AP
MEME-NGU-CASM
MEME-RECODE-AP
MEME-RECODE-CASH
Baseball
7.90
0.00 0.00
0,00 + 0.00
10.00
0.00
Drawbridge
12.30
0.00 0.00
4.15 + 6.62
12.41 + 0.07
12.86 0.13
Navigate Cuhes
7.80
3,33 + 5.33
9.76 + 0.25
10.00 + 0.00
10.00 + 0.00
Push Blocks
8.40
0.00 -: 0.00
0.24 0.24
1.07 + 0.62
4.06 +: 2.14
Remember Sensor
7.60
0.00 1
0.00 0.00
0.00 + 0.00
0.00 = 0.00
Throw Across
5.70
0.00
0.00
0.00 0.00
9.72 0.31
Wall Sensor
9.10
0.00
10.00 0.00
10.00 0.00
Wall Sensor Stack
8.60
0.00
0.00
0.00 0.00
0.00
Table Aggregated results over Atari environment tasks.
Game
Human
MEME-NGU-AP
MEME-RECODE-AP
p-value
Final Performance
Montezuma's Revenge
4753.3
10715.2 + 4967.4
11591.94 + 1112.67
0.1970
Pitfall!
6463.7
44947.6 13020.0
77737.38 + 14669.51
0.0043
Private Eye
69571.3
100796.2 2.5
100794.7 +: 2.0
0.3496
Solaris
12326.7
19810.1 + 5060.0
26152.5 4503.0
0.1970
Seaquest
42054.7
782330.1 159871.7
793697.7 + 184030.5
0.4091
Hero
30826.4
187244.3 + 27855.2
141638.4 + 60803.2
0.0465
QeBett
13455
57751.0 26942.8
182638.3
98698.3
0.0465
Venture
1187.5
2552.2
2502.4 + 78.7
0.3496
AUC
Montezuma's Revenge
6742.1 2144.8
7498.1 + 130.8
0.1970
Pitfall!
25784.5 12320.7
12320.7
26775.0 6949.1
0.3496
Private Eye
86109.6 + 3649,4
89980.2 + 1377.6
0.1548
Solaris
15338.7 2569.7
22454.0 600.4
0.0011
Seaquest
631153.4 67020.8
730154.4 + 86270.1
0.0325
Hero
89115.6 - 8964.1
71842.1 = 16767.4
0.0898
QeBt
148095.1 + 67841.9
166216.5 + 13799.1
0.0898
Venture
2322.5 + 27.3
2253.8 + 61.6
0.1201
Mean HNS
543.6 643.66477.92 (177.99660088
715.5 (15.5(606.1885)) 660.1.833.3
0.0185
Median HNS
329. 4 (2244.9.430.9
357.9 (77.9/270.1147..2
0.3551
Mean Capped HNS
95.7 95.7(89..,100.0 1.100.0)
100
0.0419
Table 7: Atari final performance and AUC. For individual games
one-sided Mann- WhinnWy test for difference in mean between RECODE and NGU and report the corresponding p-values. For aggregate statistics (Mean and Median HNS) we compute p-values using bootstrap estimate.
25

---

Published as a conference paper at ICLR 2024
K MULTITASK EXPERIMENTS
We also implemented RECODE in VMPO-based agent similar the one used with BYOL- Explore (Guo et al. 2022), and compared our performance with B BYOL-Explore in the multi-task setting. This experiment serves two different purpses.. First, this demonstrates the generality of our exploration bonus, that is shown to be useful in widely different RL agents, be they value-based or policy- Second, we can do a direct comparison with the state of the B -xplore agent in the multi-task settings. However, we note that the representation learning technique used in this experiment, 1-step Action Prediction, based on feed-forward embedding that discards past history, and may therefore not be the best fit for exploration in Partially Observable MDPs (POMDPs). Still, Fig. [9 shows that RECODE's performance competitive with that of BYOL-Explore, with only one level missing match its performance. Improving this performance using better-suited representations, such as CASM, is left for future work.
Drawbridge
Navigati
Bocks
10.
7.5
E30066
2.5
1e9
1e9
Beraerrbr Senoot
Throw Across
Wall Senoar
Wall Sansor Stack
INFFORED0DE-N AP
WMFodro, -byltre mummnn
530dt 2009
Number of fromess
1ee
Number of frames
1e5
Namber fraines
1e5
Number of finess
1e9
Figure 19: Performance of RECODE compared to BYOL-Explore on the multi-task version of DM-HARD-8. Our RECODE implementation in this experiments based on VMPO, using a con- tinuous action set.
L ADDITIONAL ABLATION STUDIES
L.1 MEMORY SIZE AND INSERTION PROBABILITY.
We report here additional results on the performance of RECODE (in particular MEME-REE COME-RECODE-AR AP agent) on Atari for different memory sizes (2 . 105,5 5 102) and E (0.2, 0.05).
Hilel
100033
700000
1833
23000
3000
38000
73000
1.5
1.6
02211
Venture
08:03
DCY Min- 200, el
nor 200 0B-2
253000
net 4th $0,
250
40000
Far Ait. popepooddipooo)
15990
9.4
1500
23000
32
20
4.5
15
00
1.5
56
4.5
5.9
15
4.5
68
7.5
1.5
30
4.5
63
7.5
Nander flames
Marbe
Figure 20: Ablation study on and memory size for RECODE. The combination 7) = 0..5 and memory size 50k is the one reported in the main paper.
26

---

Published as a conference paper at ICLR 2024
We see that for all combinations RECODE achieves a robust performance most environments, never failing to achieve super-human performance.
L.2 CASM MASKING
Here we analyze the performance of our technique when removing the masking strategy in CASM (see main text). For each element of the sequence, instead of providing either the embedded observation or the action, always provide both, making the classifier upstream task too simple. Masking allows to provide extra context (with respect to action-prec preeiccoreiiiion while keeping the prediction task hard enough to require the encoding of high-level features in the representation.
Mean Capped on DM-HARD-8
Prediction loss on DM-HARD-S
80
3.
REME-RECODE-CAMM
Log of pred cneiicion loss
MEME RECODE-AP
Capped HNSS
MEME RECODE CASnorM
60
1.5
20
RECOER
MEME-RECODE-A
MEME RECODE-CASnnMM
0
Number of frames
1e9
Number of frames
Figure 21: CASM masking ablation. (Left): CASnotM is CASM without any masking applied to the trajectory. Even by itself, the extra context provides a clear advantage over AP, but the masking strategy is essential to solve Push Block. (Right): The loss for CASnotM is much lower than for both AP and CASM suggesting that the additional context without any masking makes the prediction task easier (but leading to a less robust representation).
L.3 RND ON TOP OF ACTION PREDICTION EMBEDDINGS
We adapt RND to leverage trained action To that effect, we use a randomly initialized Multi-Layer Perceptron (MLP) to perform a random projection of the embedding, and The reconstruction error provides an intrinsic reward for exploration, which we normalize by a running estimate its standard deviation in Burda al. 2019). We find that the resulting agent unable to solve some of the hardest exploration games such as Montezuma s Revenge or Pitfalll !. The results of this ablation is shown in Fig. 22 Experiments with pre-trained embeddings do seem indicate that RNDonAP can obtain stronger performance in this setting, but the inability to concurrentyy train the embeddings greatly limits the general applicability of the method.
action-predictinn prediction embeddings. which refer to RNDonAP.
second, trained MLP, reconstruct this random projection.
pttail
30003
20009
2000
10000
5033
3003
40003
2503
20000
30003
1e0
1et0
q
M8.8 NCB-P
MEVE REDDDE AP
430003
2549
530009
MEVE DND
0.5
15000
CHM6-F
430003
50000
00
569
Number of frorrs
1cB
Mumber of ftames
168
Mamberr flames
168
Mumber of franes
168
Figure 22: Performance of RECODE compared MEME and its ablations on 8 hard exploration Atari games. We find that this approach does not allow to solve some of the hardest games such as Montezuma's 's Revenge Pitfall!
27

---

Published as a conference paper at ICLR 2024
One possible explanation of this failure is the fact that large RND error be caused by either the observation new state, or drift in the representation of an already observed one. The failure of RND to disentangle these two effects results in poor exploration.
L.4
RECODE VS NGU COMPARISON ON PRE-TRAINED EMBEDDINGS.
Mortezumm
10082
MEME-REDIDE-AP
NEME-EC0DE-47
Pretriint 9-00M/ HHB
Provcheed GADEM) NEU Hares
Hummn
12000
[ploste 2014
13000
7330
66
Number frames
1eB
Number frames
Tel
Figure 23: Performance with pretrained embeddings used with or NGU.
In this ablation we test how much the performance of NGU can be improved when using pre-trained embeddings. As one of the main limitations of the RND component in NGU is the low compatibility with concurrently-traine embeddings, we expect that when using high quality pre-trained and fixed embeddings RND (and thus NGU) can perform much better.
We evaluate this in the two most representative games from the Atari suite, Montezuma's revenge and Pitfal!!. For each game, we first run RECODE with concurrently-trained embeddings, and then take snapshot of the embeddings at 400M frames. The agent is then trained again from scratch with these frozen embeddings using either RECODE (with further concurrent training) or NGU (further training only for the EMM part). In Figure 23 we report results for the original RECODE run, the new runs with pre-trained embeddings as well Montezuma's revenge, where there are less visual confounding factors and policy learned directly in pixel space more effective, pre-training brings only small improvements for both RECODE and NGU. However in Pitfall!, where it is important to have good representation to filter out uncontrollable elements, agents that leverage pre-trained embeddings can achieve much higher scores than those the original agent had achieved at the time the snapshot taken.
non-pretrained NGU run for reference. We find that on
L.5
CASM IN ATARI
As CASM was specifically designed to aid representation learning in partially observable and 3D environments, it might be expected to be less beneficial in environments such as Atari which have more limited degree of partial observability. Indeed, performance is quite similar between MEME- RECODE- CASM and MEME-N CEMMERECCDDE-AE across most games, with the notable exception of Hero, in which CASM yields significant performance boost.
28

---

Published as a conference paper at ICLR 2024
20200
montesuma revenge
pitaal
private
selaris
15300
50500
60003
50000
0.3
1.5
3.0
4.5
8.0
7.5
1.5
3.0
45
7.3
8.0
1.5
50
HENE ADCOD:-F
520000 9000
0.6
Human
0.4
200000
0.2
0.0
Number trimees
160
Number of framess 166
Number trames 1e0
Number of thammss 1s8
Figure 24: Comparison of CASM with AP on Atari. All CASM hypers are identical to those used in DM-HARD-8 except for the state mask rate (set in these experiments, as we observed that high values exhibited much higher variance between seeds on Pitfall!)
L.6 RECODE TOP BYOL EMBEDDINGS
To further show RECODE robustness to change of representation, in Fig. 25
we compare the per- formance of the RECODE embedding on top an prrdit-predittion representation with respect to BYOL representation (as in Guo et al. (2022)).
monezuma revenge
o tial
Orivate eye
solenis
130000
105000
2503
553000
2000
10000
42001
0.0
1.5
3.0
4.5
6.6
7.5
0.5
3.0
2.5
6.0
7.5
0.3
1.5
3.0
4.5
64
7.5
0.0
1.5
3.0
1.5
62
25
obert
232000
RDCODC-AP Recote Syll
2500
2250
302200
832000
2300
330000
1190
300002
233000
0.2
3.0
Number franncs
160
Number of frannes
160
Sumber framnes
160
Numrerr of frannes 160
Figure 25: Comparison of RECODE with AP vs BYOL embeddings. We observe that RECODE is able to leverage BYOL embeddings to achieve superhuman performance Montezuma's Revenge and achieve positive scores on Pitfall!, though significantly underperforming compared to AP embeddings. For BYOL, sweeped over embedding sizes of {32, 128, 512) and report the best performing size, 32.
29


Reviews:
[{"review_id": "WCg9I2iOcZ", "reviewer": "Reviewer_tYtD", "summary": "This paper proposes a novelty-based intrinsic reward (RECODE) to facilitate exploration in sparse-reward environments. The proposed approach derives intrinsic rewards using approximate state visitation counts in a suitably constructed embedding space. \n\nRECODE builds on the previously proposed Never Give Up (NGU) intrinsic reward in two ways.\n\nFirst, unlike in NGU, which uses one-step inverse dynamics to learn an observation embedding, this paper proposes Coupled Action-State Masking (CASM) to capture controllable features across multiple steps.\n\nSecond, the proposed approach maintains and derives intrinsic rewards from a global memory of embeddings updated based on a clustering principle. Whereas NGU derived intrinsic rewards using the contents of an episodic buffer and a global bonus from Random Network Distillation (RND).\n\nExperiments show that the proposed approach to providing intrinsic rewards outperforms baselines in challenging exploration settings. Further analysis shows that the proposed approach is more robust to observation noise than NGU and RND.", "questions": "Q. In many procedurally generated environments, episodic resets to the memory (as in NGU) could be preferable. Consider a scenario where blue circles are actually novel in the current episode (and should be sought) but have been seen in previous episodes in other contexts. Of course, some notion of global novelty would also typically be needed. It would seem that something like NGU would again be preferable to RECODE in many of these settings. I am curious to know the authors’ thoughts regarding this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.**  The problem of exploration in sparse-reward settings and large observation spaces is of significant interest to the research community.\n\n**S2.** The proposed approach simplifies NGU in a crucial way. NGU needed to combine an episodic bonus and a global bonus from RND. RECODE removes the need for a separate episodic novelty component. However, this simplification comes at the cost of mechanisms to manage the memory (see W1).\n\n**S3.** In terms of the empirical evaluation, the environments considered are indeed challenging exploration problems. Further, the proposed approach provides empirical benefits, especially on the DM-HARD-8 problems.\n \n**S4.** The paper is well-written and easy to follow.", "weaknesses": "**W1.** A crucial weakness is that the proposed approach introduces additional complexity in memory management compared to NGU, which had a more straightforward episodic reset for memory. Here, there are hyperparameters for atoms of memory size, a discounting of counts and additional heuristics to update/add/remove atoms of memory. Some of the additional complexity and increase in hyperparameters might make it hard to apply this approach to new environments.\n\n**W2.** Further ablations are needed to clarify certain issues.\n\nIs CASM only beneficial over one-step action prediction (AP) when there is high aliasing due to partial observability (like DM-HARD-8)? Are results for MEME-RECODE/NGU-CASM available for the Atari environments? I may have missed them in the appendix.\n\nIn Appendix L, results are presented for RND on AP (one-step action prediction), where a random net is applied to features extracted by AP. As the random network is typically seen as a feature extractor itself, wouldn’t it be more natural to obtain RND-like intrinsic rewards from a predictor of the AP feature of a state (rather than a random embedding of the AP embedding)?\n\nSimilarly, it would be helpful to evaluate if RECODE is better than NGU when NGU uses RND on AP (or an RND-like bonus with AP features directly) for its global bonus.\n\n**W3.** The paper would benefit from connections to existing work. For instance, the recently proposed MIMEx [1] also uses masked transformers to derive trajectory level intrinsic rewards. \n\nPrevious works have studied incorporating information over longer trajectories to derive better intrinsic rewards under partial observability, [4] uses general value functions based on RND, other approaches use successor features which naturally incorporate multi-step information [2,3].\n\n**W4.** In the abstract and the conclusion, the authors claim that the proposed approach sets a new state-of-the-art on Atari’s hard exploration dataset and DM-HARD-8. While this claim is reasonable for DM-HARD-8, I am not sure it applies to the Atari experiments, as RECODE appears to be worse than NGU in hero and venture.\n\nI remain open to increasing my score, should the weaknesses and questions be adequately addressed/clarified.\n\n\n—------------------—------------------—------------------—------------------—------------------\n\n### References\n\n[1] Lin, T., & Jabri, A. (2023). MIMEx: Intrinsic Rewards from Masked Input Modeling. arXiv preprint arXiv:2305.08932\n\n[2] Machado, M. C., Bellemare, M. G., & Bowling, M. (2020). Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence\n\n[3] Janz, D., Hron, J., Mazur, P., Hofmann, K., Hernández-Lobato, J. M., & Tschiatschek, S. (2019). Successor uncertainties: exploration and uncertainty in temporal difference learning. Advances in Neural Information Processing Systems\n\n[4] Ramesh, A., Kirsch, L., van Steenkiste, S., & Schmidhuber, J. (2022). Exploring through random curiosity with general value functions. Advances in Neural Information Processing Systems\n\n------------------------------------------------------------------------------------------------------\n\n**EDIT (Post-rebuttal):** \n\nThanks for your detailed response to the weaknesses and for sharing your thoughts regarding resets in different procedurally generated environments. The response largely addresses my concerns, and I have updated my score accordingly.\n\nThe promised experiments/ablations and clarification about RND-on-AP alleviate my concerns about W2. Incorporating the comments shared in the reply to W3 would help better contextualize the contributions of RECODE. Modifying the wording and including the details shared here will avoid misinterpreting claims (W4). \n\nComing back to W1, your response helps me better understand the \"complexity\" trade-off with respect to NGU. I appreciate the point regarding long episodes. Removing RND is a significant plus, especially if it is hard to tune. The new sensitivity analysis in the appendix supports the paper (it would be nice to see more values considered in the next version). *Ideally*, this claim should also be backed through sensitivity analysis with NGU/RND.\n\nA minor note I missed in the original review is a slight inconsistency between the Background and General Notation sections. For example, policies map observations to distribution over actions in Background, but in General Notation, the policy operates on histories. \n\nAnother minor issue is that the caption of Figure 6 uses top/bottom instead of left/right.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKC4gEeVKO", "reviewer": "Reviewer_BYvq", "summary": "The document presents a novel approach known as RECODE for enhancing exploration in the realm of Reinforcement Learning. This method combines elements from both parametric and non-parametric techniques in this domain.\n\nInitially, the paper delves into the existing literature of novelty-driven exploration. Parametric strategies involve assessing novelty by utilizing trained representation models on states and actions. However, these methods are prone to issues like catastrophic forgetting or predicting novelty where it is irrelevant with respect to \"controllable\" features. On the other hand, non-parametric techniques maintain a repository of historical embeddings. They employ approaches such as state counting to drive novelty-based exploration, or approximate state counting in continuous domains.\n\nIn contrast to traditional methods, the authors introduce RECODE, which is a non-parametric approach relying on a historical embedding repository. Instead of simply erasing entries from this history, RECODE dynamically adjusts embeddings and counters by employing a system of discounted sums.\n\nThe paper also offers a series of experiments showcasing RECODE's superior performance when compared to baseline methods. Remarkably, RECODE stands out as the first Reinforcement Learning algorithm to tackle the game \"Pitfall!\" and surpass the human baseline in \"Push Blocks\" from DM-HARD-8.", "questions": "Would you kindly answer the following questions?\n* Please clarify how the experimental figures support the conclusion that the method is indeed superior to alternatives.\n* I believe that Figure 1 would benefit from a more explanatory caption. There is no definition for HNS, Capped HNS and the number of frames in the caption.\n* Could you explain how your method is better than the alternatives when it comes to design space complexity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important problem within the realm of Reinforcement Learning by introducing an algorithm centered on novelty-driven exploration. As a solution, RECODE effectively mitigates significant shortcomings observed in prior visitation count techniques like NGU. It strikes a balance between short-term and long-term novelty, avoiding an excessive bias toward short-term novelty and integrating long-term memory into the novelty determination process, despite having finite memory constraints.\n\nIn the experiments, the RECODE approach completes the “Pitfall!” benchmark for the first time, and it also attains performance levels exceeding human capabilities on the \"Push Blocks\" challenge from DM-HARD-8.", "weaknesses": "The rationale behind using RECODE as opposed to NGU is simplicity: RECODE uses one singular mechanism, while NGU uses two. However, it is not apparent to me that overall RECODE is simpler in terms of hyperparameter count / design space. Is there a rationale behind the optimal choice of hyperparmaters?\n\nIn addition, the experiment figures do not seem to support an overwhelming positive conclusion in favor of RECODE, besides for the headline successes on “Pitfall!” and DM-HARD-8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sRTQQI0MGr", "reviewer": "Reviewer_CuGy", "summary": "This work presents a framework that integrates sophisticated representation learning with long-term memory to improve novelty-seeking behaviours in deep reinforcement learning. The framework consists of two key components: RECODE, a memory buffer that approximates visitation counts in a learned embedding space; and CASM, a transformer model that learns the embedding space by jointly addressing masked sequence modelling and inverse dynamics prediction. \n\nThe paper explores and implements various components to tackle challenges inherent in deep RL, such as non-stationary distributions, online learning of representations, and sparse rewards. Experimental results on challenging benchmark tasks demonstrate that RECODE (the memory buffer) enhances performance regardless of the representation learning method, and CASM improves performance independently of the memory-based approach. The algorithm is compared with NGU and Byol-Explore, two recent and popular deep exploration algorithms. \n\nOverall, the paper presents a sophisticated algorithm that achieves great performance in hard-exploration tasks, and its implementation details are well-justified and clearly presented.", "questions": "The computational resource requirements for training RECODE remain unclear. The paper indicates a substantial number of environment interactions, around 7.5e8 steps, for various environments, comparable to previous studies. However, essential details such as memory usage, CPU requirements, and GPU memory are not provided. This lack of information poses a significant concern, as it could either limit the framework's practical applicability or potentially underscore its uniqueness as a noteworthy contribution.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper builds upon prior work in the fields of exploration in deep reinforcement learning and representation learning. Concerning RL-related work, it extends existing non-parametric novelty estimation methods that maintain a history of observations in a memory buffer and use KNN to compute similarity. However, this work pushes the boundaries of previous memory-based algorithms by operating the memory buffer in a much more compact embedding space. Additionally, the paper presents several implementation details to ensure an accurate estimation of true visitation counts, addressing challenges related to non-stationary data and representation distributions. For representation learning, the paper adopts the widely recognized transformer architecture for sequence modelling and employs masked training to learn compact representations in reinforcement learning. Furthermore, the study incorporates the established inverse dynamics prediction objective but frames it within the more condensed embedding space. This work effectively combines and adapts robust foundations from multiple fields into a high-quality and well-justified framework for reinforcement learning, resulting in outstanding practical performance.\n\nThe paper maintains clarity, offering comprehensive descriptions of the background theory and the architecture of the presented framework. It includes numerous helpful figures and algorithm pseudo-code, enhancing the reader's understanding of the method. Notably, the paper meticulously addresses significant challenges related to long-horizon exploration in deep RL, each thoroughly examined with dedicated experiments and figures that surpass typical task-return analyses.\n\nIn summary, this work makes a significant contribution to the community. Specifically, it introduces an algorithm that harnesses potent representation learning and precise novelty estimation to solve challenging long-horizon exploration problems effectively.", "weaknesses": "The presented algorithm demonstrates impressive performance in well-established hard-exploration benchmarks. However, it lacks evaluation in an open-ended (yet equally challenging in terms of exploration) environment, such as Minecraft, which would combine the complexities of long-horizon tasks from Atari with rich observations from environments like DM-Hard-8, as utilized in the paper.\n\nWhile the authors assert the robustness of the presented algorithm to different RL algorithms for policy learning, the RL training solely relies on MEME, a powerful RL algorithm with substantial learning capacity. While this choice appears suitable, the paper lacks an analysis showcasing the performance of more commonly used algorithms like PPO or DQN when coupled with RECODE. Including such comparative results would significantly support the paper's claims regarding RECODE's versatility in enhancing long-horizon exploration across various RL algorithms. VMPO is also used for the DM-Hard-8 suite, but I believe this doesn't provide evidence that RECODE would still work well with more popular RL algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)