BYOL-Explore: Exploration by Bootstrapped Prediction
Authors: Zhaohan Daniel Guo
Shantanu Thakoor
Miruna Pislar
Bernardo Avila Pires
Florent Altché
Corentin Tallec
Alaa Saade
Daniele Calandriello
Jean-Bastien Grill
Yunhao Tang
Michal Valko
Remi Munos
Mohammad Gheshlaghi Azar
Bilal Piot
Venue: NeurIPS 2022 Accept (2022)
Keywords: Exploration
Deep Reinforcement Learning
Representation Learning
Self-Supervised Learning

Abstract: We present BYOL-Explore, a conceptually simple yet general approach for curiosity-driven exploration in visually complex environments. BYOL-Explore learns the world representation, the world dynamics and the exploration policy all-together by optimizing a single prediction loss in the latent space with no additional auxiliary objective. We show that BYOL-Explore is effective in DM-HARD-8, a challenging partially-observable continuous-action hard-exploration benchmark with visually rich 3-D environment. On this benchmark, we solve the majority of the tasks purely through augmenting the extrinsic reward with BYOL-Explore intrinsic reward, whereas prior work could only get off the ground with human demonstrations. As further evidence of the generality of BYOL-Explore, we show that it achieves superhuman performance on the ten hardest exploration games in Atari while having a much simpler design than other competitive agents. 

BYOL-Explore: Exploration by Bootstrapped Prediction
Zhaohan Daniel Guo" DeepMind monielguo@deepminndd
Shantanu Thakoor* DeepMind
Miruna Pislar"
DeepMind
Bernardo Avila Pires* DeepMind
Florent Altché*
Corentin Tallec*
Alaa Saade Deep.Mind
DeepMind
DeepMind
Daniele Calandriello DeepMind
Jean-Bastien Grill DeepMind
Yunhao Tang DeepMind
Michal Valko
Rémi Munos DeepMind
DeepMind
Mohammad Gheshlaghi Azar" DeepMind mazar@decpmind.comm
Bilal Piot DeepMind piot @ deepmind.com
Abstract
We present BYOL-Explore, a conceptually simple yet general approach for curiosity-drivee exploration in visually-complex environments. BYOL-Explore learns a world representation, the world dynamics, and an exploration policy all- together by optimizing single prediction loss in the latent space with no additional auxiliary objective. We show that BYOL-Explore effective in DM-HARD-8, a challenging partially-observable continuous-actin with visually-rich 3-D environments. On this benchmark, we solve the majority the tasks purely through augmenting the extrinsic reward with GYOL-Explord ore'''r's intrinsic reward, whereas prior work could only get off the ground with human demonstrations. As further evidence of the generality of BYOL-Explore, that achieves superhuman performance on the ten hardest exploration games in Atari while having much simpler design than other competitive agents.
hard-exploration benchmark
show
Introduction
Exploration is essential to reinforcement learning (RL) [67]. especially when extrinsic rewards sparse or hard to reach. In rich environments, the variety of meaningful directions of exploration visit everything. Thus, the question becomes: how can an agent determine
makes impractical which parts of the environment are interesting to explore? One promising paradigm to address this challenge is curiosity-din driven exploration. consists of (i) learning a predictive model of some information about the world, called world model, and (ii) using discrepancies between predictions of the world model and real experience to build intrinsic rewards (59 66 60
34
5T
52 2. An
RL agent optimizing these intrinsic rewards drives itself towards states where the world model is incorrect or imperfect, generating new trajectories on which the world model can be improved. In other words, the properties of the world model influence the quality of the exploration policy, which turn gathers new data shape the world model itself. Thus, can be important not learning
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the world model and learning the exploratory policy as two separate problems, but instead altogether as a single joint problem to solve.
In this paper, we present BYOL-Explore, a curiosity- exploration algorithm whose appeal resides in its conceptual simplicity, generality, and high performance. Bxpl-Explore learns world model with self-supervvsed prediction loss, and uses the same loss to train a curiosity-driven policy, thus using a single learning objective to solve both the problem of building the world model's representation and the curiosity-driven policy. Our approach builds upon Bootstrap Your Own Latent (BYOL), a latent-pree preeiccve self- supervieer method which predicts an older copy of its own latent representation. This bootstrapping mechanism has already been successfully applied in computer vision 20 [56, graph representation learning [71), and representation learning RL [24] 62]. However, the latter works focus primarily on using the world-model for representation learning in RL whereas BYOL-Explore takes this one step further, and not only learns a versatile world model but also uses the world model's loss to drive exploration.
We evaluate BYOL- Exploe on DM-HARD-8 [22], a suite of 8 complex first-person-view 3-D tasks with sparse rewards. These tasks demand efficient exploration since in order to reach the final goal and obtain the reward they require completing a sequence of precise, orderly interactions with the physical objects in the environment, unlikely to happen under a vanilla random exploration strategy (see Fig. 2 and the videos in supplementary materials). To show the generality of our method we also evaluate BYOl-x on the ten hardest exploration Atari games [5]. In all these domains, BYOL-Explrre outperforms other prominent curiosity-drine driven exploration methods, such as Random Network Distillation (RND) [ and Intrinsic Curiosity Module (ICM) [51]. In DM-HARD-8, BYOL-Explore achieves human-level performance in the majority of the tasks using only the extrinsic reward augmented with BYOL-Explore ore's 's intrinsic reward, whereas previously significant progress required human demonstrations [22]. Remarkably, BYOL-Explors achieves this performance using only single world model and a single policy network concurrently trained across all tasks. Finally, as further evidence of its generality, BYOL-Explore achieves superhuman performance in the ten hardest exploration Atari games [5] while having a simpler design than other competitive agents,
such as Agent57 and Go- Expplore [4 15
2
Related Work
There is a large body of research in building world models either for planning 666
63
27 26
61
representation learning 62 24 [9] curiosity-driven exploration [59] 68
60
34
51
52
2
63
21 653. Most works consider world models that predict the entire observations [58]
48
[6]
19
which necessitates a loss in pixel space when observations are visually complex images. Some works have considered predicting latent representations, whether they are random projections 7 8
or learned representations from a separate model, such as an inverse dynamics model [51] or an
auto-encoder [25] (7] Finally, some RL works [6]] have focused on predicting quantities such as the extrinsic reward, the seliinnne policy, and the value function build a world model.
Our BYOL-Exp1l Er0l-exploss's world model operates in latent space and uses the same loss both for repre- sentation and intrinsic reward, simplifying and unifying representation learning and exploration. XYOL-Explore 's world model is derived from recent self-supervised representation learning meth- ods 120 [5666
55 71 and is similar to the ones in self-supervised RL [62 (24.. These previous works
focused on the benefit of shaping representations for policy learning and have not looked into explo- ration. We build on this previous work to show that we can take the impact of good representation technique further and use it to drive exploration.
While our approach belongs to the curiosity-drin -riiven exploration paradigm (50 42 49 59 34 5T
5 68
60
52 2, 63), other exploration paradigms have also been proposed. The maximum entropy paradigms try to steer the agent to a desired distribution of states (or state-action pairs) that maximizes The goal-conditioned paradigm has the agent set its own
the entropy of visited states [29 69 70 23
goal to drive exploration 57
28 15 54 80 53). The reward-free exploration
*Contrary to Agent57, BYOL- Explore neither requires episodic memory nor using an additional bandit mechanism to mix long- term and short-term rewards. As opposed to Go- -xxploree we do not have to explicitly keep in memory set of diverse statestates to visit, which requires setting additional hyper-parameter that are emnironment-dependdan.
2

---

paradigm consists of training an agent to explore the environment such that it would be able to produce near-optimal policy for any possible reward function 37 39
8
79 81
3
Method
Our agent has three components: self-supervised latent-predictive world-model called BYOL-Explore, generic reward normalization and prioritization scheme, and an off-the-shelf RL agent that can optionally share its own representation with BYOL-ES -Explore 's world model.
3.1 Background and Notation
We consider a discrete-time interaction process 44
36 13 between an agent and its environment where, at each time step N, the agent receives an observation o, E O and generates an action a E A. We consider an environment with stochastic dynamics H. x A + Ae that maps history of past Dbservations-actiion and a current action probability distribution over future observations. More precisely, the space of past observations-action is H. = ULEN Hi where Ho O and Vt E N', Hc+1 H4 x We consider policies π H - AA that maps history of past observations-action to probability distribution over actions. Finally, an extrinsic reward function re A + maps a history past observations-actions to real number.
35
3.2 Latent-Predictive World Model
BYOL-Expp -Yploex world model is a multi-step predictive world model operating at the latent level. It is inspired by the self-supervised learning method BYOL computer vision and adapted to interactive environments (see Section 3.1 Similar to BYOL, EYOI-Explore model trains an online network using targets generated by an exponential moving average (EMA) target network. However, BYOL obtains its targets by applying different augmentations to the same observation as the online repre- sentation, whereas BYOL-Explors model gets its targets from future observations processed by an EMA of the online network, with no hand-crafted augmentation. Also SYOL-Explore model, uses a recurrent neural network (RNN) 33 [ to build the agent state, i.e., the state RNN, from history of observations, whereas the original BYOL only uses a feed-forward network for encoding the observations. In the remainder of this section, we will explain: (i) how the online network builds future predictions, (ii) how targets for predictions obtained through target network, (iii) the loss used to train the online network, and (iv) how we compute the uncertainties of the world model.
O++1
2
O
Targe Prente
Target fa
fo
fr
K
go(br,))
folocta) (l
1
GEVoL-Exploze (0, (0,
K
(bo
by
Go
96
96
0t-1
be.1
bt.2
br,
Cheeet wee
b
Opwnviose
b
Opanioso
b.
Openr hoom
h
hg
h
h
0:
Croader
Dr
0+++1
a++K-1
Figure 1: EYOL-Explore's Neural Architecture (see main for details).
We write Ay the set of probability distributions V.

---

(i) Future Predictions. The online network is composed of an encoder fo that transforms an observation Or into an observaatoon-reprseentttoo fo(o;) RN,, where N E N is the embedding fo(or) is then fed alongside the previous action 4,-1 to RNN cell hg that is referred as the close-loop RNN cell. It computes a representation b, E RRM of the history ha E HH seen so far as b, hg (b 11-1, fo(o))) where M N is the size of the history-represennation.. Then, the history-eppresentttinn b, is used to initialize an open-loop RNN cell hg that outputs open-loop representations (br.x E RR M ))= as bt, = (g (14k-1) where br.0 = b, and K is the open-loop horizon. The role of the open-loop RNN cell is to simulate future historryresssssenttiionn while observing only the future actions. Finally, the open-loop representation b is fed to predictor de to output the open-loop prediction (o(bt,k) R- at time I. k that plays the role of our future prediction at time + k.
size. The
(ii) Targets and Target Network. The target network is an observation encoder fo whose param- eters are an EMA of the online network's 's parameters 0. It outputs targets fø(ot+*) E IR that are used train the online network. After each training step, the target network's 's weights are updated via an EMA update Φ + oz + (1 - a) where a is the target network EMA parameter. A sketch of the neural architecture is provided in Fig. with more details in App. A
(iit) Online Network Loss Function. Suppose our RL agent collected a batch of trajectories B-1
(oo,, j where T E N is the trajectory length and B E No is the batch size. Then, the loss LEvoL-Explora (0) minimize is defined the average cosine distance between the open-loop future predictions go(b),c) (b),,) and their respective targets folo)+)) at time L ++ k:
g0((),,)
folo(+1)
2
Levot-Explora (k (. =
[190
(1
B-1T-2
K())
CEVOL CeyoL-Expiore(()
Σ
Σ
CovoL (0,
BT -
K(t)
1=0
where (1) = min(K, - the stop-gradient operator.
- 1) is the valid open-loop horizon for a trajectory of length T and sg
(iv) World Model Uncertainties The uncertainty associated to the transition (di, o(41) is the
sum of the corresponding prediction losses:
= Σ D+9
Lavot-Explare( (0, p)
where 0 < D K S 2. This accumulates all the losses corresponding to the world-model uncertainties relative to the observation o/-1- Thus, timestep receives intrinsic reward based on how difficult its observation to predict from past partial histories.
Intuition on why BYOL-Explore learns meaningful representation. The intuition behind BYOL-Explore is similar in spirit to the one behind BYOL. In early training, the target network is initialized randomly, and so BYOL-Explore's online network and the closed-loop RNN are trained to predict random features of the future. This encourages the online observation representation to capture information that useful to predict the future. This information is then distilled into the target observation encoder network through the EMA slow copy mechanism. In turn, these features become targets for the online network and predicting them can further improve the quality of the online representation. For further theoretical and empirical insights why the bootstrap latent methods learn non-trivial representations see, e.g., 72 76
3.3 Reward Normalization and Prioritization Scheme
Reward Normalization. We use the world model uncertainties e/¹ to generate an intrinsic reward. To counter the non-stationarity of the uncertainties during training, we adopt the same reward normalization scheme as RND ( and divide the raw rewards ((e)) (()))T=)) by an EMA estimate of their standard deviation o,. The normalized rewards are e/ o, Details are provided in App. A.3

---

Reward Prioritization. In addition to normalizing the rewards, we can optionally prioritize them by optimizing only the rewards with highest uncertainties and nullifying rewards with the lowest uncertainties. Because of the transient nature of the intrinsic rewards, this allows the agent to focus first on parts of the environment where the model is not accurate. Later on, if the previously nullified rewards remain, they will naturally hecome the ones with highest uncertainties and be optimized. This mechanism allows the agent to optimize only the source of high uncertainties and not optimize all sources of uncertainties once. To do so, let us denote by Me/or, the adjusted EMA mean relative to ) j== We use pag/a as a clipping threshold
the successive batch of normalized rewards separating high and low-uncertainty rewards. Then, the clipped and normalized reward that plays the role of intrinsic reward is: r/, max(E? (&f /o, Hefo, 0).
3.4 Generic RL Algorithm and Representation Sharing
BYOL- -xxlore can be used in conjunction with any RL algorithm for training the policy. In addition to providing an intrinsic reward, BYOL Exppoor can further be used to shape the representation learnt by the RL agent by directly sharing some components of the BYOL-Explore world model with the RL model. For instance, consider recurrent agent composed of an encoder fa. an RNN cell hgg. policy head T.. and a value head Uy that are shaped by an RL loss. Then, we can share the weights 0 of the BYOL-Explore world model and the weights & of the RL model at the level of the encoder and the RNN cell: fo fe and hg = h& and let the joint representation be trained via both the RL loss and BYOL-Explore In our experiments, we will show results for both the shared and unshared settings. Architectural details are provided in Appendix A
Experiments
We evaluate the algorithms on benchmark task-suites known to contain hard exploration challenges. These benchmarks have different properties in terms of the complexity of the observations, partial observability, and procedural generation, allowing us to test the generality of our approach.
Atari Learning Environment (6) This is widely used RL benchmark, comprising approximately 50 Atari games. These are 2-D, -ubly-observable, (fairly) deterministic environments for most of the games but have a very long optimization horizon (episodes last for an average of 10000 steps) and complex observations (preprocessed greyscale images which are 84 x 84 byte arrays). We select the 10 hardest exploration games [5] to conduct our experiments: Alien, Freeway, Gravitar Hero, Montezuma' 's Revenge, Pitfall, Private Eye, Qbert, Solaris and Venture.
Hard-Eight Suite (22]. This benchmark comprises 8 hard exploration tasks, originally built to emphasize the difficulties encountered by an RL agent when learning from sparse rewards in procedurally-generaa
3-D world with partial observability, continuous control, and highly variable initial conditions. Each task requires the agent to interact with specific objects in its environment in order to reach a large apple that provides reward (see Fig. ( Being procedurally-geenratee properties such as object shapes, colors, and positions are different every episode. We provide videos in the supplementary materials to ground the difficulty of these tasks. Note that the current best RL agents that solve these tasks require a small (but non-zero) amount of human expert demonstrations. Without demonstrations or reward shaping, state-of-the-ar deep RL algorithms, such as R2D2 [38), do not get positive reward signal on any of the tasks. our case, we train a single RL agent and a single world model to tackle the 8 tasks all-together, making for challenging multi-task setting.
4.1 Experimental Setup
At a high level, BYOL- Exxplore has 4 main hyper- parameters: the target network EMA parameter a, the open-loop horizon K, choosing to clip rewards and to share the BYOL-Explore representation with the RL network. To better understand what part of BYOL- -xx-lxro is essential to perform well, we run 4 ablations. Each ablation corresponds to BYOL-Explore where only one has been changed. The 4 ablations are namely Fixed-targets where the target network EMA parameter set to a = 1, Horizon= where the horizon set to K = 1, No clipping where we do not use clipping for the intrinsic rewards and No sharing where we trained separately the RL network and the
5

---

1F Trama
Look Frr
Pick
Throw ball doun
Enter chaimbor
Pick apple (final goal)
Figure 2: -i-e snapshots of the human player solving Baseball task. They are ordered
chronologically from left to right and top to bottom. Each image depicts specific stage of the task.
BYOL-Explore 's world model. addition to BYOL-Explore, we also run as prominent baselines RND, ICM (see App. B for details), and pure RL which is an RL agent only using extrinsic rewards.
Finally, we run experiments on two different evaluation regimes. The first regime uses mixed reward function r = Te,r + Ari, which is linear combination of the normalized extrinsic rewards re,s and intrinsic rewards computed by the agent ri,t with mixing parameter X. This may be the most important regime for practitioner as we can see if our intrinsic rewards help improve performance, with respect the extrinsic rewards, compared to the pure RL agent. The second regime is fully self-supervised where only the intrinsic reward Ti, is optimized. This regime gives sense of how pure exploration methods perform complex environments.
Choice of RL algorithm. We use VMPO 64 as our RL algorithm. VMPO is an efficient on-policy optimization method that has achieved strong results across both discrete and continuous control tasks, and is thus applicable to all of the domains we consider. Further details regarding the RL algorithm setup and hyperparameters are provided in Appendix C
Performance Metrics. We evaluate performance in terms of the agent score at a number of observations/frames I, Agent (4), measured by undiscounted episode return. The number of frames corresponds to all the frames generated by all the actors by interacting with the environment, even the skipped ones. Frames/observations can be skipped if there an action repeat which the case in Atari where the action repeat is of 4.
through training Agent max; Agent (t), done in [8] 3 We define, for each game, the Human Normalized Score (HNS) at number of frame t: HNS (t)
We define the highest agent
Bunaaaapree Randomazore as well the HNS over the whole training: HMS = max, HNSS(t). A HNS higher than means superhuman performance specific task. We similarly define the CHNS Score as HNS clipped between and 1.
Agent (a)dRoodot
4.2 Atari Results
In these experiments, we set the target EMA rate 0.99 and open-loop horizon K 8. We use À 0.1 to combine the intrinsic and extrinsic rewards. We follow the classical 30 random no-ops evaluation regime [46] 73). and average performance over 10 episodes and over seeds. This evaluation regime does not use sticky actions 43
Fig. 3 (left) shows that BYOL PVOL-Explore is almost superhuman on the 10- (0-hardest exploration games and outperforms the different baselines of RND, ICM, and pure RL. 3 (right) compares BYOL-Explore against its ablations to gain finer insights into our method. The No clipping ablation performs comparably, showing that the prioritization of intrinsie rewards is not necessary on Atari tasks. Similarly, the Horizon=/ / ablation performs slightly better, indicating that simply predicting one-step latents is sufficient explore efficiently on the fully-observable Atari tasks. The Fixed Targets ablation performs much worse, showing that our approach of predicting learned targets (rather than

---

80
80
Mean CHNS in %
Mean CHNS in
60
60
BYOL- EYOo-Explore
BYOL Explore
clipping Fixed targets
40
BYOL-Explore (big)
40
RND
sharing
20
ICM
20
Horizon-1
1.5
Number of frames
1e10
Number of frames
1e10
Figure 3: Mean CHNS(t) score across the tasks in Atari. Left: BYOL ExpL-explore and the baselines in the mixed regime for Atari. Right: BYOL-Explore and its ablations in the mixed regime.
fixed random projections) is vital for good performance. is also worth noting that all the ablations except Fixed Targets outperform all of our baselines, demonstrating the robustness of our approach.
Finally, because the Horizon= ablation was close to superhuman on Atari, we run the same configuration but double the length of the sequences which we train from 64 to 128 (also doubling memory requirements while learning). With this small adjustment, this agent (BYOL-Explore (big)) becomes superhuman all of the 10-hardest exploration games.
Purely intrinsic exploration. We test how BYOL-Explore behaves when only given intrin- sic rewards without any extrinsic signal on the well-known Montezuma's Revenge game. We measure exploratory behavior in terms of the number of different rooms of the dungeon the agent able to explore over its lifetime. Note that accessing later rooms requires navigating complex dynamics such as collecting keys to open doors, avoiding enemies, and carefully traversing rooms filled with traps such as
Montezuma's Revenge
20
Number rooms visited
15
timed lasers. Figure 4 shows how much room coverage achieved during training when no extrinsic reward used, showing that BYOL-Explore explores further than the best result reported by RND [8]. Importantly, we the episodic setting for intrinsic rewards whereas the published RND results considers the non-episodic setting for intrinsic rewards facilitating exploration as the agent is less risk-averse. Therefore, our setting could be considered even more challenging. Our agent explores more than 20 rooms on average versus 17 with best published RND results. As expected in the episodic setting, our RND re-implementation visits even fewer rooms. However, we can reproduce the published RND results in the episodic setting when using recurrent policies.
BYOL-Explore
ICM
Best RND (paper)
1.2
Number of frames
1e10
Figure 4: Number of rooms visited in Montezuma 's Revenge during training in the self-supervised regime over 3 seeds.
Further results. More fine-graines results are reported in App D.I We report, in Fig and in Fig the agent scores learning curves for each game. Tab, and Tab. 2 have agent score at the end of training. Finally, Tab. 3 and Tab, show the mean CHNS and different statistics (mean and percentiles) of the HNS across the selected games. An interesting finding from examining the HNS is that clipping and longer-horizee such as Montezuma' s Revenge or Hero. BYOL-ES Explore has a median HNS of 331.98 compared to No-clipping ablation and the Horizom= which have a median HNS of only 181.39 and 199.80 respectively. Therefore, while clipping is not necessary to get to human-level performance, it is still crucial achieve top performance. We also provide further results regarding the pure exploration setting all 10 games in App. D.2
predictions are critical for very high scores on some games
7

---

4.3 Atari's results in the Presence of Stochastic Distractors
Stochastic distractors or noise are well known to break intrinsic curiosity methods based on prediction enror of future frames. This is because the prediction squared-error loss of a future frame has an irreducible component which corresponds to the variance of the future frame distribution [51] [2]. However, because BYOL-Explore is not prediction error method at the frame-level but at the latent level, we can hope that some noise present the frame can be removed from the latent embedding. More specifically, we hypothesize that BY0L-Explore removes noisy features of the frame that are not useful to minimize the BYOL-Explore loss in order to better minimize this loss. Those features that not useful for future predictions. On the other hand, since RND uses random network to build it targets, RND by construction cannot actively remove noisy features from its targets. To better show this, slightly changed the Atari environment to generate noisy frames. More precisely, the new observation is made of two parts of the same size (84 x 84), on the right side we have noise and on the left we have the original Atari image. Each time the no-op action chosen by the agent, a new noise is sampled. The noise composed of 14 x 14 blocks of 6 x 6 pixels. Each block of pixel is assigned uniform random value between and 255 if provide a frame of this modified Atari environment in Fig. (left..
noise is sampled. We
Montezuma's Revenge
Episide Soore
8030
6030
4030
BVOL-Eaplone
BVOL-taplone (kotey.
AND (Hott))
Number
1e10
Figure 5: Experiments on noisy Atari. Left:
Noisy Atari environment. Right: Learning curves in terms of agent score for BYOL-Explore and RND for noisy and normal Atari on Montezuma's Revenge, averaged over 10 episodes and 3 seeds.
We train BYOL-Explore and RND on this noisy and normal version of Atari on the game Montezuma's s revenge (we do not include comparison with ICM because it was already not performing well in the normal version). To get better results for BYOL-Explore, we increase the predictor to have hidden layers of 512 instead of hidden layer of 256. The results reported in Fig. (right). We observe that BYOL-Explo ore is perfectly able to deal with that type of controllable- noise whereas RND is not and completely flat-lined in the noisy environment because the agent is attracted to the noise and keeps repeating the no-op action.
4.4 DM-HARD-8 Results
In these experiments, we set the target EMA rate a = 0.99 and open-loop horizon K = 10. We use À 0.01 to combine the intrinsic and extrinsic rewards. In contrast to prior work (22). we perform experiments in the more challenging multi-task regime, training a single agent to solve all eight tasks. At the beginning of each episode, a task is drawn uniformly at random from the suite.
In Fig. 6 (left) we report the mean CHNS(t) across the tasks, averaged over seeds. We see that BYOL - Expplore outperforms the baselines of RND, ICM, and pure RL by a large margin. Fig. (right) compares the performance of BYOL-Explore to its various ablations. Note that the No-clipping ablation performs similarly to BYOL-Explore in terms of CHNS. However, unlike the fully-observable Atari tasks, the /orizon=/ ablation learns considerably slower and achieves lower final performance (see also our extended ablations the horizon length in Fig. 15 in App. D.3) We note once again that
8

---

the BYOL-Explore bootstrapping mechanism for learning representations is essential, as confirmed by the poor performance of the Fixed targets ablation. Due computational limitations, we did not run the No Sharing ablation, as using separate networks requires twice the memory.
BYOL-Explore
BYOL-Explore
RND
clipping
ICM
Harlzon=1
Episode score
RL
Episode score
Fixed targets
4
2
0
2
8
2
Number of frames
1e9
Number of frames
1e9
Figure 6: Mean (H) score across the tasks in the DM-HARD-S suite. Left: BYOL-Explore against baselines: ICM, RND and Pure RL. Right: -YOl-Explore against various ablations.
We now analyze our method more closely by examining per-task performance. The full learning curves for each task be found in Fig. 7 for EYOL-Explore and the main baselines and in Appendix D.3 (see Fig. 14) for the various ablations. First, we take note that other curiosity-driven methods (ICM and RND) cannot get any positive
the majority of the OM-HARD-8 tasks, with additional hyperparameter tuning and reward prioritizing (see Fig. 17 and Fig. [8 in App. D.3).
In contrast, we see that BYOL-Explore achieves strong performance on five out of the eight hard exploration tasks. Importantly, BYOL-Explore achieves this without human demonstrations, which was not the case in prior work [2 BYOL-Explor even surpasses humans on 4 tasks, namely Navigate cubes, Thros-across, Baseball, and Wall Sensors (see Tab. 2 in App. [D.3] for details). Most impressively, Exp0--Explord can solve Throw-across, which is challenging task even for a skilful human player and was not solvable in prior work without collecting additional successful human demonstrations
22
Interestingly, note that on the Navigate Cubes task, both RND and the Fixed-targets ablation achieve maximum performance alongside BYOL-Explore. We argue that this is because the prediction of random projections (either at the same step as done by RND or multi-step as done by BYOL-Explores leads to the policy learned performing spatial, navigational exploration required to explore well on the Navigate Cubes task. In contrast, the other tasks require exploratory behavior involving interaction with objects and the use of tools, where both RND and the Fixed-targett ablation fail. Finally, we observe that two games, namely Remember Sensor and Push Blocks, particularly challenging, where all of considered methods perform poorly. We hypothesize that this due to the larger variety of procedurally generated objects spawned in these levels, and the need remember previous cues in the environment leading hard credit assignment problem.
this is the kind of behavior
Purely intrinsic exploration. Each of the DM-HARD-8 tasks has complex dynamics and object interactions, making difficult assess qualitatively the behavior of purely intrinsically motivated exploration. Nevertheless, for completeness, we provide results of BYOL-Explore trained only with intrinsic rewards in App. D.3 showing that it does achieve some positive signal on the Drawbridge and Wall Sensor tasks (see Fig. 19
5 Conclusion
We showed that FYOL-Explore is simple auriosity-driver exploration method that achieves excel- lent performance on hard exploration tasks with fairly deterministic dynamics. BYOL-Explore is a multi-step prediction error method at the latent level that relies recent advances in self-supervised learning to train representation as well as world-model without any additional loss. Atari, BYOL-Explos achieves superhuman performance on the 10-hardest exploration games while being

---

Drawbridge
Remember Sensor
Navigate Cubes
Wall Sensor
0
10
stro--exlome
10
10
Epicode score
R84D
ICM
R..
0
2
2
0
2
2
4
0
1eg
1e9
1eg
1e9
Wall Sensor Stack
Push Blocks
Throw Across
Baseball
BYOL-Explore
Episode szore
RND
ICM
Number of framess 1eg
Number of frames
1e9
Number of framess 1e9
Number of frames 1e9
Figure 7: Agent's score for each task in the DM-HARD-8 suite for BYOL-Explore against baselines.
Shaded areas correspond to the minimum and maximum values across three seeds.
of much simpler design than other superhuman agents. Moreover, BYOL-Explors substantially outperforms previous exploration methods DM-HARD-8 navigation and manipulation tasks in 3-D, multi-task, partially-observahle and procedurally--eneraae ality of our algorithm to handle either 2-D or 3-D, single multi-task, fully partially-observable environments. While one main limitation of our algorithm is that it is not designed to fully handle stochasticity in environments (e.g. sticky actions), we show that BYOL-Explore is robust some simple kinds of controllable noise ("TV-noise") since operating in latent space allows the latent representation filter out.
environment. This shows the gener-
In the future, we would like to improve performance HMM-HAR-8 and to demonstrate the generality of method by extending it other domains. In DM-HARD-8, we believe can improve performance by scaling up the world model and finding better ways to trade off exploration and exploitation. Beyond DM-HARD-8, there opportunities tackle further challenges, most notably highly-stochastic and procedurally-generuae do so, we are investigating different mechanisms to adapt prediction-based methods stochastic environments in order to only use the epistemic uncertainty as an intrinsic reward and discard the aleatoric uncertainty.
environment dynamics such as NetHack [40]. To
Acknowledgments and Disclosure of Funding
We would like to thank Abbas Abdolmaleki, Arunkumar Byravan, Adrià Puidomenech Badia, Tim Harley, Steven Kapturowski, Thomas Keck, Jean-Baptiste Lespiau, Kat McKinney, Kyriacos Nikiforou, Georg Ostrovski, Razvan Pascanu, Doina Precup, Satinder Singh, Hubert Soyer, Pablo Sprechmann, and Karl Tuyls for their support and advice in developing and publishing this work.
10

---

References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pages 5048-5058, 2017.
[2] Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo Avila Pires, Bastinnttien Grill, Florent Altché, and Rémi Munos. World discovery models. arXiv preprint
2019.
[3] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[4] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
[5] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pages 1471-1479, 2016.
[6] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-2799 2013.
[7] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint 188i 04355, 2018.
[8] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Seventh International Conference on Learning Representations, pages 1-17, 2019.
[9] Xiaoyu Chen, Jiachen Hu, Lin F Yang, and Liwei Wang. Near-optimal reward-free exploration for linear mixture mdps with plug-in solver. arXiv preprint arXiv:2110.03244, 2021.
[10] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decooder approaches. arXiv preprint arXiv:1409.1 1259,, 2014.
[11] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.
[12] Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre- Yves Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages 1331-1340. PMLR, 2019.
[13] Mayank Daswani, Peter Sunehag, and Marcus Hutter. ment learning. Asian Conference on Machine Learning, pages 213-228. PMLR, 2013. Q-learning for history-based reinforce-
[14] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv:190) 10995, 2019.
[15] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-588) 2021.
[16] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical in- teraction through video prediction. Advances in neural information processing systems, 29, 2016.
[17] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International Conference on Machine Learning, pages 1515-1528, 2018
11

---

[18] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, al. Noisy networks for exploration. arXiv preprint arXiv:/700. 10295, 2017.
[19] Karol Gregor, Danilo Jimenez Rezende. Frederie Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping belief states with generative environment models for rl. Advances in Neural Information Processing Systems, 32, 2019.
[20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-21 212244, 2020.
[21] Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv preprint
2021.
[22] Caglar Gulcehre, Tom Le Paine, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making efficient use of demonstrations to solve hard exploration problems. In International conference on learning representations, 2019.
[23] Zhaohan Daniel Guo, Mohammad Gheshlagi Azar, Alaa Saade, Shantanu Thakoor, Bilal Piot, Bernardo Avila Pires, Michal Valko, Thomas Mesnard, Tor Lattimore, and Rémi Munos. Geometric entropic exploration. arXiv preprint arXiv:2101.0206 2021.
[24] Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Baster Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In International Conference Machine Learning, pages 3875-3886. PMLR, 2020.
[25] David Ha and Jürgen Schmidhuber. World models. arXiv preprini arXiv:1803.10122 2018. 2018.
[26] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprini arXiv: 1912.01022.03 2019.
[27] Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learming, pages 2555-2566 PMLR, 2019.
[28] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2020.
[29] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pages 2681-269) 2019.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[31] Matteo Hessel, Manuel Kroiss, Aidan Clark, lurii Kemaev, John Quan, Thomas Keek, Fabio Viola, and Hado van Hasselt. Podracer architectures for scalable reinforcement learning. arXiv preprint arXiv:2104.06272, 2021.
[32] Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 7803-3803, 2019.
[33] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 973):1735-7700 1997.
12

---

[34] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. Advances in Neural Information Processing Systems (NIPS), 2016.
[35] Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probubility. Springer Science & Business Media, 2004.
[36] Marcus Hutter et al. Feature reinforcement learning: Part I. unstructured MDPs. De Gruyter Open, 2009.
[37] Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinfrremeett learning. In International Conference on Machine Learning, pages 1870-4879. PMLR, 2020.
[38] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[39] Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Algorithmic Learning Theory, pages 865-891. PMLR, 2021.
[40] Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671 1-7844, 2020.
[41] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[42] Manuel Lopes, Tobias Lang, Mare Toussaint, and Pierre-Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems, pages 206-214, 2012.
[43] Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61: 61:523-562, 2018.
[44] R Andrew McCallum. Instance-based utile distinctions for reinforcement learning with hidden state. In Machine Learning Proceedings 1995, pages 387-395. Elsevier, 1995.
[45] Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilic Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning, pages 7599-7608. PMLR, 2021.
[46] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
[47] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31:9191-9200, 2018
[48] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. Advances in neural information processing systems, 28, 2015.
[49] Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
[50] Pierre-Yve Oudeyer, Frédéric Kaplan, and Véréna Hafner. Intrinsic Motivation for Autonomous Mental Development. IEFE Transactions on Evolutionary Computation, 11(2):265-22 2866 January 2007.
13

---

[51] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity -driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 16-17, 2017.
[52] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International Conference on Machine Learning, pages 5062-5071, PMLR, 2019.
[53] Silviu Pitís, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR. 2020.
[54] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering sulf-supee reinforcement learning. In International Conference on Machine Learning, pages 7783-7792. PMLR, 2020.
[55] Adria Recasens, Pauline Luc, Bean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Päträucean, Florent Altché, Michal Valko, et al. Broaden your views for self-supervised video learning. In Proceedings of the TEEF/CVF International Conference on Computer Vision, pages 1255-1265, 2021.
[56] Pierre H. Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, and Michal Valko. Byol works even without batch statistics. In NeurlPS 2020 Workshop on Self-Supervised Learning: Theory and Practice, 2020.
[57] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In International conference on machine learning, pages 1312-1320, 2015.
[58] Juergen Schmidhuber and Rudolf Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(01n02): 134, 1991.
[59] Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
[60] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):230-247, 2010.
[61] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-699 2020.
[62] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.005929, 2020.
[63] Ramanan Sekar, Oleh Rybkin, Kostas Danilidiis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
[64] H Francis Song, Abbas Abdolmalcki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv: 12238, 2019.
[65] Sumedh Sontakke, Arash Mchrjou, Laurent Itti, and Bernhard Schölkopf. Causal curios- ity: RI agents discovering self self-super experiments for causal representation learning. In International Conference on Machine Learning, pages 9848 -988-988. PMLR, 2021.
[66] Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International conference on artificial general intelli- gence, pages 4 41-51. Springer, 2011.
[67] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
14

---

[68] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753-2762, 2017.
[69] Jean Tarbouriech and Alessandro Lazaric. Active exploration in markov decision processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 974-982 2019.
[70] Jean Tarbouriech, Shubhanshu Shekhar, Matteo Pirotta, Mohammad Ghavamzadeh, and Alessan- dro Lazaric. Active model estimation in markov decision processes. In Conference on Uncer- luinty in Artificial Intelligence, pages 1019-1028. PMLR, 2020.
[71] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. Large- scale representation learning on graphs via bootstrapping. In International Conference on Learning Representations, 2022.
[72] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs, In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.
[73] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[74] Ruosong Wang, Simon S Du, Lin Yang, and Russ Salakhutdinov. On reward-free rein- forcement learning with linear function approximation. In Advances Neural Information Processing Systems, volume 33, pages 178 17826, 2020.
[75] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. In International Conference on Learning Representations, 2019.
[76] Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive supe-suue learning. arXiv preprint arXiv:2205.0622 2022.
[77] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[78] Andrea Zanette, Alessandro Lazaric, Mykel J Kochenderfer, and Emma Brunskill. Provably effi- cient reward-agnostic navigation with linear value iteration. In Advances in Neural Information Processing Systems, volume 33, pages 11756-1 11756-11766 11766, 2020.
[79] Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation. Advances in Neural Information Processing Systems, 34, 2021.
[80] Yunzhi Zhang, Picter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In Advances in Neural Information Processing Systems, pages 7648-7659, 2020.
[81] Zihan Zhang, Simon Du, and Xiangyang Ji. Near optimal reward-free reinforcement learning. In International Conference on Machine Learning, pages 12402-12412. PMLR, 2021.
[82] Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularizee multi- goali rein- forcement learning. In International Conference on Machine Learning, pages 7553 -5622, 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We including theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] We are not including theoretical results.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [No] The code proprietary
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We specify the main training details in the paper and we include full list of hyperparameters description the appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report error bars in learning curves of the agent score for every agent we run.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We include all the information regarding the compute in the appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use the ALE and DM- MM-HHRD-8 and we cite the creators.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] We did not use crowdsoureing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] We did not use crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/]] We did not crowdsourcing.
16


Reviews:
[{"review_id": "sd3hEu4Wxwq", "reviewer": "Reviewer_tiha", "summary": "This paper introduces BYOL-Explore, a novel exploration method for RL. The idea of BYOL-Explore is to train an encoder of observations based on BYOL and further utilize training loss as the intrinsic reward for exploration.\n\nThe architecture of BYOL-Explore consists of an observation encoder, a closed-loop RNN that embeds the history of interaction, and an open-loop RNN for future prediction. In training the observation embeddings, the authors adopt the learning scheme of BYOL and construct a targe encoder by a moving average of the trained encoders. The training of the encoder thus minimizes the distance between the RNN prediction and the target encoder of observations. \n\nExperiments suggest that BYOL-Explore outperforms the SOTA baselines RND and ICM. ", "questions": "$\\{\\bf{\\rm Question 1.}\\}$ The proposed RNN structure seems relatively large-scaled compared with most existing exploration strategies. How does BYOL-Explore compare with ICM and RND regarding the computation budget and training efficiency?\n\n$\\{\\bf{\\rm Question 2.}\\}$ Does the proposed intrinsic reward of BYOL-Explore capture the epistemic uncertainty of observations adequately? For instance, if a specific observation-action pair $(o, a)$ frequently occurs among the trajectories, we would hope that the intrinsic reward of visiting $(o, a)$ is small as it is of less novelty. Nevertheless, the proposed intrinsic reward sums up all the prediction errors involving the pair $(o, a)$ across the trajectories, which might be even larger than that of scarcely explored observation-action pairs. \n\n$\\{\\bf{\\rm Question 3.}\\}$ In addition, the proposed intrinsic reward seems to have a preference over actions that are distant from the initial observations. For instance, for a given trajectory ${x_1 = (o_1, a_1), x_2 = (o_2, a_2), \\ldots, x_n = (o_n, a_n)}$, the intrinsic reward of $x_1$ consists only the loss associated with the history representation $b_{1, 1}$. Nevertheless, since each closed-loop RNN is followed by the open-loop RNN cells that reconstruct the entire trajectory, the intrinsic reward of $x_4$ thus sums up three terms, namely, the loss associated with the history representations $b_{1, 3}$, $b_{2, 2}$, and $b_{3,1}$. Will such design leads to a higher intrinsic reward of $x_4$ than $x_2$? Is such a preference for ending steps desirable for exploration?\n", "limitations": "N.A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\{\\bf{\\rm Strength.}\\}$\n\n-The proposed BYOL-Explore calculates the intrinsic reward and thus is adaptable to existing RL algorithms.\n\n-The proposed curiosity is built on future multi-step predictions, thus utilizing the temporal information contained in observations. In addition, the intrinsic reward is defined by the training loss on latent representation. If the latent well captures the temporal information of observations, the proposed intrinsic reward can filter out the reconstruction loss of irrelevant observations (to the transition dynamics), which is desirable.\n\n$\\{\\bf{\\rm Weakness.}\\}$\n\n-The learning paradigm of BYOL-Explore falls in the common practice of fitting (coarse) transition models and then utilizing its training loss (or the disagreement among the ensemble of models) as the intrinsic reward/penalty for exploration/conservative RL. In that regard, the contribution of this paper is relatively marginal. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VG-4bjjPG8O", "reviewer": "Reviewer_YjyB", "summary": "The authors present a Bootstrap-your-own-latent inspired method for model learning to generate intrinsic rewards for exploration. The method consists of 2 parallel networks, one an exponential moving average of the other with goal of predicting k steps into the future conditioned on past actions, current state and k future actions. The predictions occur on the latent space with a cosine loss for training. The discrepancy between the predictions is used to generate rewards for the RL agent to induce exploration. The authors test their method on DM-Hard and hard exploration games in Atari and show considerable performance improvement with reference to curiosity baselines. \n", "questions": "How does it work in presence of distractors? Some experiments of the \"noisy TV\" would be nice to see. \n", "limitations": "The paper is principled and along the lines of previous works. The results are impressive and useful for the community. The evaluation is thorough in my opinion. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Method is principled. The intuition behind why it works is clear from previous works. \n2. Well-written and easy to follow: The authors describe the method well along with the setup. \n3. Extensive experimentation: The experimentation suite is large, with experiments in first-person ego-centric environments as well as hard exploration problems in third-person atari games. \n\nWeaknesses:\n1. Not a significant algorithmic development: The method proposed is not necessarily. While this is not a weakness on its own, it can be said that the contribution is not algorithmic development. \n2. Missing some related works: http://proceedings.mlr.press/v139/sontakke21a/sontakke21a.pdf and  https://arxiv.org/pdf/2109.08603.pdf among many many others. Please do a more thorough search and add relevant citations.  \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Eu7h2fVKxcG", "reviewer": "Reviewer_huAP", "summary": "This paper builds on the BYOL work from 2020 and explores using it in the context of curiosity-driven exploration. It learns a world model via an RNN + self-supervised loss mechanism. This mechanism, optionally with an external task loss, is used to train an RL policy in order to play Atari and DM-Hard-8. Besides presenting superhuman results on the ten hardest Atari games, they also present favorable exploration results on Montezuma's Revenge. Notably, they do not use any human demonstrations.\n\nThey compare to other works in the curiosity-driven space on the metric of mean CHNS (clipped human normalized score). Across all ablations except the one fixing the EMA update to not update, their reported method outperforms the baselines and is superhuman when allowed to train on longer sequences. \n\nThey show the exploration results on Montezuma's Revenge to test what happens in the purely intrinsic exploration setting. Because there are no external reward results given, we should not expect the model to beat the game but instead to explore the setting. Compared to the curiosity-driven baselines, it does that, finding 20 rooms compared against 17 in the published results.", "questions": "I don't have any major questions other than the one about the lambda sensitivity. It would be cool if you could keep going on the big model to see how it does in DM-Hard-8 as well, but I'd understand if the time/compute budget isn't there.", "limitations": "Yes, they understand that their model is quite a ways off from being realistically used yet. They talk in the conclusion about improving on DM-Hard-8 and then moving on to more difficult environments like NetHack.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is good. There doesn't appear to be anything world-changing in it, but it's a solid paper presenting a good idea and written well.\n\nOriginality:\n\nThe idea of forward prediction is not new, nether is BYOL, the surprisingly non-degenerative self-supervised model on which BYOL-Explore is built. Combining these concepts by using an RNN to do the prediction makes sense. This isn't original and I'm glad someone tried it.\n\nQuality:\n\nThe quality of this paper is high. I don't see any experiments that I need to have run, nor do I see any major faults with their reasoning. This is a good piece of work. That being said, I certainly would _like_ to see some other experiments for my own interest :).\n\nAn example is what happens as you scale up or down the lambda on the DM-8 experiments. It's quite low at 0.01 right now and so I'm left wondering if that's just for optimization reasons or if the model actually just doesn't like to solve tasks as that gets larger. If it was an order of magnitude lower, would it succeed a lot better? In other words, what's the sensitivity of this parameter as it's a crucial cog in the presented algorithm.\n\nClarity:\n\nThis paper is clear and well written. Thank you for that.\n\nSignificance:\n\nThe results aren't game changing, but they are of great interest to the self-supervised learning and curiosity-driven learning communities. BYOL made a large splash when it debuted and this extension of BYOL is notable for the community. Others are likely to use it in their baselines and perhaps build upon it.\n\nI'm still surprised that the BYOL models don't collapse to a single point, so more evidence of this mechanism proving fruitful in complex self-supervised methods is helpful for building my model of how this works.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9IxCBA2kXGC", "reviewer": "Reviewer_SjCF", "summary": "The paper presents a new way of providing intrinsic rewards for exploration, by learning a latent space and predicting its future latents. The intrinsic reward is then proportional to the error between the predicted and actual future latents, so that states that are harder for the world model to predict are rewarded more. The method achieves impressive results on two hard-exploration benchmarks: hard-exploration Atari games and DM-HARD-8. ", "questions": "See above, reiterating here:\n\n- What is your sample complexity?\n- What form of stochasticity did you use on Atari (sticky actions, no-ops, nothing…)?\n- Why do you think your RND performance appears lower than the RND paper?", "limitations": "One limitation that I would have liked discussed is the paper’s dependence on determinism or lack thereof. The RND paper tries to solve the following “noisy-TV problem” with sticky actions: suppose you are in room 1 of Montezuma’s and room 2 is immediately to your right, then depending on how the randomness of sticky actions work out, it may be that you only have a 50% chance of getting to room 2 during the next frame, and a 50% chance of staying in room 1. As a result, if you try to predict some feature of *future* frames for your exploration bonus, the exploration bonus may stay high for those stochastic states no matter how often they are explored, which is why RND tries to predict features of the *current* frame instead. Given that this work predicts features of future frames, it may be vulnerable to the noisy-TV problem stated above. The authors should either disclose this as a limitation or explain why they think their work  avoids this issue.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " Strengths:\n\nThe work shows impressive performance on Atari, and seemingly on DM-HARD-8 (note: I am very familiar with past Atari results so I can judge those at a glance. For DM-HARD-8 I am relying on the comparison included in the paper, hence the more qualified statement). I am particularly impressed by the performance on Pitfall, given in the appendix (IMO the authors should call that out more, as Pitfall is a harder game than Montezuma’s, and their performance improvement relative to, say, RND is larger on Pitfall than Montezuma’s).\n\nI agree with the authors’ claim that the method is simpler than Go-Explore and Agent57, so it seems to represent a genuine improvement for the field.\n\nWeaknesses:\n\nThe main weakness is that the paper fails to make standard declarations about how they ran the benchmarks, at least in the case of Atari (which again I am most familiar with). While I do think the work would be at least somewhat impressive no matter what these specifics happen to be, without them it is really hard to decide if the work is an incremental improvement or a game-changer in the field. \n\nSpecifically:\n\n- The number of sample frames is never stated. Instead, the x-axis of all plots is given in “Learner Steps” which is, frankly, a meaningless number. All Atari work I am aware of (including works that this work compares to like RND, Agent57, and Go-Explore) states the number of frames used in some form (note: it is standard practice to include skipped frames if using frame skipping, when updating, the authors should make sure they clarify their frame skipping strategy and whether it is included in their frame count). Without this number, it is hard to compare the method to, say, Agent57. Further, Figure 4 seems to indicate that the authors’ implementation of RND might perform substantially worse than that in the RND paper, but it is hard to tell if this is due to differences in sample complexity. If it is not, it would be interesting for the authors to briefly discuss why they think their RND implementation performs worse.\n- Likewise, it seems to have been standard prior to this work to give the sample complexity on DM-HARD-8 as well, so the authors should also report it.\n- No information as far as I can tell is given about how stochasticity is injected in the Atari results (no-ops, sticky actions, etc.). This has been a contentious point historically, with some evidence from Machado et al. [1] that sticky actions reduce average performance on Atari. Without knowing if they were used, it is hard to properly place the results. Further, RND (which is a primary comparison of this paper) was specifically designed to solve noisy-TV issues that arise due to sticky actions, so it would be useful to understand if this method is vulnerable to those issues (which I think it might be, see Limitations section).\n\nI recommend the authors consult Machado et al. [1] for a set of evaluation recommendations for Atari. While I personally don’t expect every Atari paper to follow this exact evaluation protocol (the recommended frame complexity is too low IMO, and while I believe sticky actions etc. should generally be followed, I am conscious of the fact that they aren’t universally adopted), I do think deviations from these recommendations are important to note explicitly, and they usually are in Atari work I am familiar with.\n\nOverall, my main request for this paper is for the frame complexity and the evaluation method for Atari (especially the way stochasticity is injected) to be clarified. The method is interesting enough and the performance is good enough that I will switch my rating to somewhere within the “accept” range when this information is provided (where exactly would depend in part on what this information is specifically: if the method uses, say, 10x fewer frames than Agent57, then that is a lot more impressive than if it uses around the same amount). However, I think it would do a disservice to the community to accept the paper in the current state, since these issues make it very hard to compare the results to competing methods.\n\n[1] Machado, Marlos C., et al. \"Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.\" 2017. *Journal of Artificial Intelligence Research*\n\n\n\nUPDATE: As promised, I am switching my rating to an \"Accept\" given that the information I requested was provided. My choice of going to \"Accept\" and not higher is primarily due to the information being in line with what I expected (frame complexity similar to Agent57, no sticky actions), which still qualifies these results as very solid and worthy of NeurIPS, but not absolutely groundbreaking.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept