Interesting Object, Curious Agent: Learning Task-Agnostic Exploration
Authors: Simone Parisi
Victoria Dean
Deepak Pathak
Abhinav Gupta
Venue: NeurIPS 2021 Oral (2021)
Keywords: reinforcement learning
exploration
intrinsic motivation
continual learning

Abstract: Common approaches for task-agnostic exploration learn tabula-rasa --the agent assumes isolated environments and no prior knowledge or experience. However, in the real world, agents learn in many environments and always come with prior experiences as they explore new ones. Exploration is a lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of task-agnostic exploration. In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner.
Later on, the agent effectively transfers the learned exploration policy to better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present a simple yet effective approach to learning task-agnostic exploration policies. Our key idea is that there are two components of exploration: (1) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent’s belief; (2) an environment-centric component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs. We also introduce benchmarks and metrics for evaluating task-agnostic exploration strategies. The source code is available at https://github.com/sparisi/cbet/.

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration
Simone Parisi !-
Victoria Dean
Deepak Pathak2 Carnegie Mellon University
Abhinav Gupta'
Facebook Al Research
Abstract
Common approaches for task-agnostic exploration learn tabula-rasa -the agent assumes isolated environments and prior knowledge experience. However, in the real world, agents learn in many environments and always come with prior experiences they explore new ones. Exploration is lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of task- agnostic exploration. In this setup, the agent first learns explore across many environments without any extrinsic goal in a task-agnostic manner. Later on, the agent effectively transfers the learned exploration policy better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present simple yet effective approach to learning task- agnostic exploration policies. Our key idea is that there are two components of exploration: (I) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent's belief; (2) an environment-centir component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testinn environment pairs. We also introduce benchmarks and metrics for evaluating task lask-agnostic exploration strategies. The source code is available https: //gi thub. cwm/ sptrsssisbbb////////
Introduction
Exploration is one of the key unsolved problems in building intelligent agents capable of behaving like humans. reinforcement learning (RL), exploration usually studied under two different settings. The first is task-driven exploration, where the reward is well-defined and the agent's goal is to explore in order to maximize long-term rewards. However, in real life, external rewards are either sparse unknown altogether. In this setting, exploration is task-agnostic: given new environment, the agent has to explore it in absence of any external reward. Common approaches to encourage task-agnostic exploration
intrinsically motivated rewards such as prediction curiosity [35, 47), empowerment [39], or visitation counts [4, 34). But does this setup represent how humans explore?
We argue that the commonly-used task-agnostic exploration setup is unrealistic, both from practical and academic viewpoints. This setup assumes environments in isolation and agents exploring tabula- i.e., with prior knowledge experience. By contrast, environment in isolation and we do not throw away our past knowledge every time we encounter environment [14]. Exploration is rather lifelong process: every time encounter new environments, we use our prior knowledge and experience to develop new efficient exploration strategies. In this paper, we view the exploration problem from continual learning lens. More specifically, in this setup, the learning agent interacts with one or many environments without any extrinsic goal. At this time, the agent learns explore the environments. Later on, the agent effectively transfers the learned exploration policy to explore new environments, rather than exploring the new environment tabula-rasa.
humans do not learn from one
Equal contribution. Contacts: sparisi@fb. com and vdean@cmu. edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Learn to Explore
A key question in learning how to explore is what to learn and how transfer prior knowledge from one environment to another. Most existing task-agnostic exploration approaches, such as visitation counts, curiosity, empowerment, define intrinsic rewards an agent-centric manner: they encourage explo- ration of unseen parts of the enviromment based on the agent's 'ss own belief. In these approaches, exploration is driven by what the agent knows about the world. However, most do not make a distinction between what the agent believes it is interested in and states that would make any agent interested. For example, if the agent seen many objects of one kind in one environment, would not explore the same type of objects again in new environment. This seems to be in stark contrast how humans explore. Consider a switch with bell sign. Even though we might have pressed hundreds of doorbell switches (and even this instance), we are still attracted to press it. Some objects in the world just demand curiosity. We argue that apart from an 'agent- centric' component, there is an emvironment--entriee component to exploration, which can be learned from prior knowledge and experiences.
Boring...
i Interesting!
know is interesting!
Transfer Interest-Base Exploration
What is it?
visitation count model and has
Change-Based Exploration Transfer (C-BET) trains task-agnostic ex- ploration agents that transfer to new environ- ments. Here the agent learns that keys are interesting, as they allow further interaction with the environment (opening doors). Later, when tasked with reaching box behind door, the agent starts by picking up the key.
Figure 1:
In this paper, we propose paradigm change move away from stand-alone isolated task-agnostic environment exploration to a more realistic multi- environment transfer-exploration setup". We show how learn exploration policies both from single- and muuli-environmeen interaction, and how to transfer them to unseen environments, This transfer-epploration setup allows agents to use prior experiences for learning task-agnostic exploration. Notably, classic stand-alone task-agnostic approaches were designed for tabula-rasa exploration and hence only explore in an agent-centric manner. They fail capture the inherent interestingness of some environment components. With this insight, we propose Change- Change-Based Based Exploration Transfer (C- BET), a simple yet effective approach learning joint agent-centric and environment-centrin exploration. The key idea is for an agent to seek out both surprises (unseen areas) and high-impar (interesting) components of the environment. The experiments show that C-BET (a) learns more effectively when placed in multi-environment setup, and (b) either outperforms performs competitively with prior methods across several unseen testing environments. We hope this paper will inspire exploration research focus more learning from multiple environments and transferring experiences rather than tabula-rasa exploration.
2 Preliminaries and Related Work
We consider environments governed by Markov Decision Processes (MDPs). In MDPs, an agent observes the state of the environment and selects actions according to a policy T(ass). In turn, the environment changes, providing a new observation s' and a reward r. Through environment interaction, the agent collects episodes, i.e., sequences of states, actions and rewards (st, az rt)==.... The goal of RI. is to learn a policy maximizing the sum of rewards during episodes, i.e., the return. In this setting, exploration poses many questions. If the environment provides rewards, what should the agent look for? When should it act greedily with respect to the rewards has found and stop looking for more? In the history of RL, many approaches have been proposed to tackle these questions. On one hand, classic single-environment approaches range from intrinsic motivation with visitation counts [2, 4, 13, 26, 53], optimism [1, 5, 25, 27, 31], or curiosity [7, 24, 35, 45, 49, 51], to bootstrapping [12, 33] or empowerment [29, 39]. On the other hand, we find approaches incrementally learn tasks, such transfer learning [58], continual learning [28], curriculum learning [32], and meta learning [38]. Below, we review approaches closely related to ours.
While it can be argued that the real world has no explicit distinction between training and testing, we use this dichotomy only for the purpose of evaluation.

---

Exploration Training
Exploration Transfer
Train
Tiak Specific Reward
Train
Infecest Reward
Task-Specilic
Exploration
Act
Eavironment Change
Policy
Policy
Act
Exploration
State
Curiosity
Policy
C-BET pre-training, Our agent inter- acts with environments and learns using intrinsic rewards computed from state and change counts.
Figure 2:
Figure 3: C-BET transfer. The pre-trained ex- ploration policy is fixed and guides tisc-specific policy learning in new environments.
Intrinsic motivation. Exploration strategies relying intrinsic rewards date back to Schmidhuber [47], who proposed to encourage exploration by visiting hard-to-predict states. More recently, the idea of auxiliary rewards to make up for the lack of external rewards has been extensively studied in RL, supported by evidence from psychology and neuroscience [20]. Several intrinsic rewards have been proposed, ranging from visitation count bonuses [4, 53] to bonuses based prediction error of some quantity. For example, the agent may learn a dynamics model and try to predict the next state [24, 35, 48, 51J. By giving bonus proportional to the prediction error, the agent is incentivized to explore unpredictable states. Schultheis et al. [49], instead, proposed to learn intrinsic rewards function by maximizing extrinsic rewards by meta- meta-gradient.
However, these approaches exploration is agent-centrie, i.e., based on an agent's belief such as the forward model error. In contrast, with this work we propose additionally learning environment-centric exploration policies. C-BET neither requires a model nor knowledge of extrinsic rewards. Instead, encourages the agent to perform actions causing interesting changes to the environment. We should note that while Raileanu and Rocktäschel [36] proposed a similar approach, their exploration policy lacks the transfer component and also requires to learn models.
Transfer learning. The idea of agents capable of incrementally learning tasks well-known in the field of machine learning, with the first approaches dating back to the 90s' [40, 41. 56]. In RL, recent methods have focused on policy and feature transfer. In the former, a pre-trained agent (teacher) is used to transfer behaviors to a new agent (student). Examples include policy distillation, where the student trained to minimize the Kullback-Leibler divergence to the teacher [44] or to multiple teachers at the same time [55]. Alternative approaches, instead, directly reuse policies from source tasks to build the student policy [3, 17, 21]. In feature transfer, a pre-learned state representation is used to encourage exploration when tasks are presented to the agent [22, 59]. Similar to transfer RL, continual RL studies how learning on one or more tasks can help accelerate learning on different tasks, and how to prevent catastrophic forgetting [28, 42, 50]. Meta RL. instead, tries to exploit underlying common structures between tasks to learn new tasks more quickly [18, 38].
However, the setup in these approaches is not task-agnostic, i.e., task-specific policies are transferred rather than exploration policies. For example, after learning policy maximizing the rewards of one task, the agent starts exploring guided by the same policy as a second task is given. Transfer is task- centric rather than task-agnostic and environmentt-cnntric. Consequently, if tasks are too dissimilar information cannot be reused, even if the environments are similar. By contrast, in this work we propose learning aask-agnostic exploration from one or many environments and show transfer to unseen environments. We should note that while Pathak et al. [35] did demonstrate fine-tuning on different maze maps, their focus and large-scale evaluations remain on tabula-rasa exploration.
3 Learning to Explore
Our goal is to decouple the enviromment-centric nature of exploration from its agent-centric compo- nent. Contrary to prior work, we propose to first learn an enviromment-centric exploration policy and then to transfer it to unseen environments. The policy is driven by the inherent interestingness of states and is learned over time via interaction. First, during a pre pre-traiinnn training phase, the agent interacts with many environments without any tasks and learns an exploration policy. Then, when new environments and tasks are presented, the agent uses the previously learned policy to explore more efficiently and learn task- policies. C-BET's key components are (1) novel intrinsic reward and the learning policy to disentangle exploration from exploitation, and (2) the use of such policy to help exploration for new tasks. Figures 2 and 3 summarize our framework.
3

---

Turn Left
Turn Right
Move Forward
Pick Up
Figure 4: Visualization of intrinsic re- wards (row) for the agent's actions (col- umn). Brighter color denotes higher re- ward. Rewarding only state counts (top) does provide useful feedback, and go- ing to the corners valued more than pick- ing up the key. With the L2 norm of state changes (middle), the agent is biased in fa- of moving, because its position en- coded with the highest value in the observa- tion space. The resulting policy will prefer to navigate without picking up the key. By contrast, C-BET (bottom) gives picking up the key the highest reward.
(N(e)
(N(())
Eq. (1)
Gridworld with a key und a door. Observations encode cells depending on their content legg- key,
agent downward. up only from the cell above Samples been cullected rundumly.
We should note that Rajendran al. [37] also proposed transfer framework based on intrinsic rewards. In their work, the agent switches between practice episodes -where the agent receives only intrinsic rewards- and match episodes -giving only extrinsic rewards. However, practice episodes are simpler variations of match episodes (e.g., in Atari Pong the agent practices against itself) rather than different tasks in C-BET. Furthermore, the intrinsic reward used practice episodes is given by function trained with meta-gradie graddggaasentss to improve the extrinsic-rewant return. That is, exploration is not task aago-sgio in C-BET, and extrinsic rewards are the main drive of the agent.
3.1 Interestingness of State-Action Pairs
The natural world is filled with states or scenarios that are inherently interesting and our goal is capture this inherent interestingness via intrinsic rewards. this paper, we propose adding an environment-cemee component of interestingness to the existing agent-centric component of surprise. Specifically, we hypothesize that the environment can change on interaction, and the changes that are rare are inherently interesting. That is, we penalize actions not affecting the environment, and favor actions producing rare changes. For instance, moving around, bumping into walls, or trying to open locked doors without keys all result in no change and thus will be of low interest.
also want to keep the agent-centric component in exploration -that is, the exploration policy should look for surprises unseen states. Thus, we further reward actions leading to less-visited states. By combining these two components, the resulting C-BET interest-based reward is
(s a, s') (/(N(s') N())),
where c(s, s') defines the environment change of transition (s, a, s'), and denotes (pseudo counts of changes and states. Figure 4 empirically shows its effectiveness. In Section 4 we discuss change encodings used in experiments.
3.2 Exploration Learning
In this phase, we want to learn agg-osic exploration policies from interaction with many environ- ments. The agent has no goal, but states where it can 'die' are still terminal. this setting, we would like to treat the problem of learning exploration as an MDP with intrinsic-rewards only, and train the agent maximize discounted intrinsic-returns averaged over episodes.
Formally, the agent explores many environments ε {F1 Feg.... F2 ). each governed by MDP (S,, A, P. r4, %... That is, each environment has own states but the action space is the same, and all environments obey the same dynamics /' and the same intrinsic reward function r;. The agent's goal
learn an exploration policy maximizing the sum of discounted intrinsic rewards, i... (sxr(s, a) = arg max, Eg.r 1E, ( az)]. To approximate the average, after maximum number of steps the environment is reset and a new episode starts, as typically done in RL.
However, both common [7, 35, 36] and Eq. (1) intrinsic rewards decrease over time as the agent explores, to the point that they vanish to zero given enough samples. For instance, counts will grow to infinity, prediction models error will go to zero. While this is not an issue in the tabula-rasa setup where the agent also gets extrinsic rewards, can be problematic the proposed task-agnostic exploration framework. Any policy, indeed, would be optimal if all rewards zero.

---

To prevent Eq. (1) from vanishing, we randomly reset counts any given time step. To explain why resets need to be random, we start by considering 'episodic counts' proposed by Raileanu and Rocktäschel [36]. These counts are reset at the beginning of every episode to ensure that the agent does not go back and forth between a sequence of states with high rewards. While this work fine when extrinsic rewards are also given, it can be problem if we learn only on intrinsic rewards. When counts are reset, the agent 'forgets' past trajectories and thinks that every state and change is new. If resets always happen at the end of an episode, then initial states will always get higher reward. Moreover, starting always with zero-counts may favor some trajectories and penalize others. Consider for example the chainworld in Figure 5. The agent always starts in A, from where it can go to B or D. From B, loops between B and C. From D. it cannot go anywhere else. The optimal exploration policy should visit all states uniformly, by randomly going B and D. However, if we reset counts at every episode the agent forgets that it has already visited B and C. Thus, the intrinsic rewards for B and C are high again, and trajectory ABCBCBC... gives higher intrinsic return than ADDD... Consequently, the optimal policy with respect to episodic counts will always prefer to visit B rather than D.
B
D
Figure 5: This chainworld illus- trates that if counts are resets at the beginning of every episode, the learned policy will never visit D.
The optimal exploration policy, instead, should have some randomness to visit the environment uniformly, while prioritizing interesting states. For this reason, we propose to reset counts at any given step with probability p. When new episode starts, counts may not be reset yet the agent remembers what it has visited before. As the agent explores, on average common states and changes will have higher count more often, and the agent will correctly prefer rarer ones. In this paper, propose p S 1 - % where % is the intrinsic reward discount factor. This is a fitting choice because in an MDP the sum of discounted rewards can be interpreted as the expected sum of undiscounted rewards if every time step had a 1 - probability of ending. Intuitively, this means that 74 implies a "life expectancy' of 11(1 - 74) steps, and thus resets should not happen more frequently than that.
The resulting MDP with Eq. (1) rewards and random count resets can be solved by any RL algorithm. However, we should note that this MDP is non-stationary, because the agent may receive different rewards for the same state, depending on how many times the state has been visited in the past. Nonetheless, classic intrinsic rewards -even in tabula-rasa exploration- either based on prediction errors [35, 36] or counts [4] also introduce non-stationarity because they change over time as well, In practice, this non-stationarity is not issue because intrinsic rewards change slowly over time.
3.3 Exploration Transfer
Now, the agent is presented with new environments and asked to solve tasks. Formally, each enviromment is governed by the standard MDP AS P,r,)) and the agent's goal is to learn a policy maximizing the sum of extrinsic rewards, i.e., Tass a) = arg max, E, " (r( a+)]. Note that while during pre-training the policy learned across all environments (one exploration policy for all environments), at transfer we learn one task-specific policy for each environment.
In this phase, the interest-policy learned earlier drives exploration as tasks and environments presented to the agent. In order not to forget interestingness over time, the exploration policy is added fixed bias to the task-specific policy, similarly to what Hailu and Sommer [21] proposed. Thanks to the decoupling of the interest-policy (based on the intrinsic reward) from the task- pask-policy (based on the extrinsic reward), the latter can be also learned independently via any RL algorithm.
In our experiments, we use IMPALA [16] to learn both TexP and TrASK. IMPALA learns policies of the form (s, a) (f a)). where a is the sofumax function. The policy is trained maximize function representing the value of states (s), trained on the given rewards. In our framework, we combine the two policies as follows.
During pre-training, by using intrinsic rewards we learn Vi(s) and Tis a) (f,(ss fi(s, a)).
offc(s))) At transfer, we learn (s(s) on extrinsic rewards. The policy is Tusa (s, af
The interestingness fü transferred but not trained, i.e., acts as fixed bias to encourage interaction. Initially the policy follows ft since fe is initialized randomly. As it finds extrinsic rewards, the sum fe+ Ja becomes greedier with respect to extrinsic rewards, and fa slowly overtakes ʃt².
'If exploration and the task goals are misaligned, we can decay exploration, e.g., TTss(s, (s) fe (s, a)). where decays over time, similarly to common e-greedy policies.
5

---

Figure 6: Examples of the environments used in our experiments. In MiniGrid (left), the agent navigates through grid and interacts with objects (keys, doors, boxes, and balls) to fulfill task. In Habitat (right), the agent navigates through visually realistic rooms.
Note that we transfer only fa (the policy) and not V; (the state value). We could think of transferring V as fixed bias as well, i... by having (s()) (s) + (s()). The policy would be trained on Ve -the states value with respect to the given task where V, fixed and only V is updated. However. we believe it is more beneficial to isolate the exploratory component within the policy, in order to keep the task-specific value function targeted on extrinsic rewards. By not transferring V₁, Ve can be accurately trained on extrinsic rewards -that the agent will see often thanks to f; from the pre-trained policy. Ve, in turn, can make Tas greedy with respect to extrinsic rewards as V. is learned.
4 Experiments
Our experiments are designed to highlight the benefits of disentangling the anvironment-centie nature of exploration from agent-centric behavior by learning a separate exploration policy and then transferring it to new environments. We stress that for learning task-agnostic exploration there are no standard benchmark environments, experimental setups, well-defined evaluation metrics, or even baselines to compare against. One of our contributions is to provide an exhaustive evaluation framework for the transfer exploration paradigm.
Environments. The experiments are divided into two main sections. The first is about MiniGrid [10] (Section 4. 1), a set of snceduurall--geerattes environments where the agent can interact with many objects. The second is about Habitat [46] (Section 4.2), navigation simulator showcasing the generality our MiniGrid experiments to a visually realistic domain.
Change encoding. In both MiniGrid and Habitat the agent partially observes the environment, since it cannot see through corners, closed door, or inside boxes, and has a limited field of view. Rather than egocentric views (i.e., what the agent sees in front of itself), we use 360 panoramic views to count environment changes, as this is a cotation-invarian representation of the observed state, Similar to Chaplot et al. [8], we concatenate four egocentric views taken from 0%, 9.90, 180", and 270" with respect to the North. Then, the change of a transition is the difference between panoramic views pano(s), i..., s's, == pano (s') pano
Baselines. We evaluate against the following algorithms. For more details, refer to Appendix A.1.
Count [4]. The intrinsic reward is inversely proportional to the next state visitation count. Random Network Distillation (RND) [7]. The intrinsic reward is the prediction error of states' random features between a trained network and a fixed one. This can be interpreted as similar to using state counts because the prediction improves states are more often.
Rewarding Impact-Driven Exploration (RIDE) [36]. The intrinsic reward is the prediction error between consecutive embedded states, normalized by episodic state counts.
Curiosity [35]. The intrinsic reward is the prediction error between consecutive states.
In Appendix B we investigate the importance of count resets, panoramic changes, and different count-based rewards. The source code is available at https: //gpt/ggtttt/////////gg wtt/ httm//parrii/cbb..
4.1 MiniGrid Experiments
MiniGrid environments [10] are procedurally-generaae gridworlds where the agent can interact with objects, such as keys, doors, and boxes (Figure 6). Exploration is challenging because rewards are sparse, observations are partial, and specific actions are needed visit all states (e.g., pickup key to open door). With MiniGrid, we can generate several pairs of train and test environments that are related but still different in many ways. These pairs enable evaluation of both the learning and transfer abilities of an exploration method and its ability to deal with unseen components.
Implementation details. All environments give a 7x7x3 x3 partial observation encoding the content of the 7 tiles in front of the agent (including the agent's tile). The agent cannot see through walls, closed doors, or inside boxes. The action space discrete: left, right, forward, pick up, drop, toggle, and done. For complete description of the environments, refer to Appendix A.4.

---

Setups. We present three setups, to study different exploration transfers against tabula-rasa.
transfer). The agent loops over three environments episode by episode, and learns the exploration policy using intrinsic rewards only. There is one state count and one change count for all three environments rather than separate counts for each. The environments are: KeyCorridorS3R3, BlockedUnlockKickup. and MultiRoom-N4-55. and have been chosen for size and interaction variety: the first has both locked and an unlocked door, a key, and a ball; the second adds a box; the third has more rooms. Note that even if these environments have all object types, the agent cannot experience all kinds of interactions. For example, it will not know that keys can be hidden in boxes, as in the ObstructedMazes. The policy is then transferred to ten environments, seven of which new. A good intrinsic reward should help learn better exploration faster from multiple environments, thanks to sharing experience from diverse interaction.
MultiEnv
SingleEnv (one-to- mone-to-many transfer). The policy is pre-trained on a single environment. DoorKey and KeyCorridor are used for pre-training because they have some -but not all- objects.
Tabula- rasa (no pre-training / transfer). A task task-specific policy is learned as in classic intrinsic motivation by summing intrinsic and extrinsic rewards. While it is non-realistic setup, it is the most common RL exploration approach, and thus serves as baseline against our transfer framework.
Evaluation metrics. Our goal to learn exploration policies that encourage interaction with the environments and transfer well new enviromments, i.c., that can further be trained to solve extrinsic tasks faster. Therefore, we evaluate policies according to the following criteria.
Unique interactions over 100 episodes at transfer to new environments, after intrinsic-rewant pre-training (no extrinsic-reward training yet). Unique interactions picks/drops/toggles resulting in new environment changes. For instance, repeatedly picking and dropping the same key in the same cell results in only two interactions.
Tasks success rate over 100 episodes at transfer to new environments, after intrinsic-reward pre- training (no extrinsic-reward training yet). The task success rate denotes in how many episodes the exploration policy visits goal states -thus, would have already solved the environment task. Extrinsic return during extrinsic-reward training, after intrinsic-reward training.
4.1.1 MiniGrid Pre- Trainiin Results
Figure shows results after pre-training in MultiEnv. In Appendix we report results for the two SingleEnv setups well. C-BET policy both interacts with the environment and find goal states more often than all baselines. As we will see in the next section, this will result in faster extrinsic- reward learning. Furthermore, C-BET's policy transfers well to all environments, even the ones with unknown dynamics (e.g.. boxes in ObstructedMazzes needs to be toggled to reveal keys). Of the baselines, only Count scores high average interactions and success rate, but it does not generalize as well as C-BET. Indeed, most of Count's 's success comes from environments visited at pre-training (the first five), but most of its interactions are in environments with unseen dynamics (ObstructedMazes). That is, Count's policy can explore familiar environments prioritizing state coverage (high success rate and few interactions), but not unfamiliar ones (low success rate yet high interactions).
Alporithe
Aug. 02
From MuitiRoon--N4553 KeyCorridor5383, BLockedUntockPickkg
Al Algorithn
Aug. SR
Unique Interactions (UI)
C-SET (ours)
160
C-BET (Ours)
40,
120
Count
35.10
Success ltate (SR)
Count
45.74%
160
RIDE
0.23
80
RIDE
1.60
Curiosity
6. 6.92
Curiosity
9.98%
60
60
4.33
60
Randon
7.
Randow
3.040
49
20
20
untock
Deartey
MeyCar
intecaPick
Staceede
w17. Ste
dbstuc 1ith
2aate Matttr 20th
Donnkay
seytore
UntoskPyok
Blocken
NR.
822 $18
Bhoter 1010
(hoter 2010
Seatar 2000
Figure 7: Unique interactions and success rate at the beginning of transfer of policies pre- trained in MultiEny. Not only C-BET interacts the most and achieves the highest success rate, but also interacts and succeeds in all environments. Naturally. it interacts more in environment with many keys/balls/boxes to pick (KeyCorridor, BlockedUnblockPickup Obszes,tedMazes) and less if there is nothing to pick (MultiRooms). On the contrary. Count overfits to the training environments and performs well only on the first five. Other baselines perform poorly, almost as a random policy.
7

---

Iinlock
DearKey-84a
KeyCarridorS38t
UnlorkPickup
Trassfe From VeeCertade Free Sourtay From Pultiber
Extrinsic Return
(Miltions)
Rolcciooon-6
Obstructeedara-101.
Obstructsddaze-222
ructedMaerz..
Trarsfer Frogs KoyCoridde Frix Nonrkyy
Extrinsic Raturn
Franes (Millions)
C-BET (Ours)
Curlosity
Extrinsic only
Figure 8: MiniGrid task learning, for both transfer and tabula-rasa exploration. The hardest tasks are outlined in red. C-BET (blue) from MultiEnv (top under each environment) performs the best, starting with nearly optimal policies in most environments. This demonstrates the effectiveness of pre-training multiple environments using the C-BET intrinsic reward.
Finally, RIDE, Curiosity, and RND baselines perform poorly. This is unsurprising if consider that they rely on predictive models and that MiniGrid dynamics are deterministic and simple. Dy- namies and embeddings models are learned quickly, without giving the policy time explore. In Appendix C.3 we support this claim by showing the intrinsic reward trend during pre-training.
4.1.2 MiniGrid Transfer Results
We transfer the exploration policies learned in Figure 7 discussed in Section 3.3. Figure 8 shows how transfer setups monny-to-mamy and one-to-many) perform against tabula-rass exploration. The first takeaway is that policies pre-trained with the C-BET intrinsic reward outperform baselines in both transfer and tabula-rasa. In MultiEnv transfer, C-BET performs the best, especially on the hardest environments (outlined in red). In particular, only C-BET is able to transfer MultiRoom-N6. On the contary, Count -that can solve it in tabula-rasa- fails at transfer. C-BET is also the only solving ObstructedMaze--DDhha -th hardest environment among the ten- even in tabula-rasa. The second takeaway is that baselines relying on models are not suited to the transfer framework. RIDE, Curiosity and RND perform better in the tabula-rasa setup (last row), except for the easiest environments (Unlock and DoorKey), meaning that transfer is actually harmful. These results are line with Figure? 7, where only C-BET and Count show success at offline transfer. Furthermore, RIDE, Curiosity and RND perform worst when transfer is from MultiEnv, highlighting that their intrinsic rewards are not suited for a multi-enviromment setup.
Finally, no algorithm learns MultiRoom-NI2-1 not even C-BET despite showing some success in Figure 7. This is due to the candomly-nntiiaizznn fe of the task-specific policy, hindering the pre-trained exploration policy success.

---

Apartmeett
Fron rtnent
States
Vistto States (15)
5090
Algoritha
C-BET (Burs)
Avg. VS
3238.51
4090
22000
Count
1449.46
10000
3090
REDE
983.50
2000
Curiosity
720.66
RND
1283.13
1000
Rendom
1260.77
F Frames (Nillions)
Hotel
Offsce
Figure Habitat pre-training. C- Figure 10: Habitat offline transfer. Bars denote the unique BET explores the scene faster and state count in scene during episode. C-BET visits scores the highest unique state count. more than twice as many states than all baselines.
C-BET
RIDE
Count
Curiosity
RND
Figure 11: Scene coverage of exploration policies during pre-training (2M steps) in Apartment 0. Darker red cells denote higher visitation rates. Only C-BET visits all of the scene uniformly.
4.2 Habitat Experiments
To demonstrate that C-BET's efficacy extends realistic settings with visual inputs, we perform experiments on Habitat [46] with Replica scenes [52].
Implementation details. Egocentric views have resolution 64x6 The action space is discrete: forward 0.25 meter, turn 10" left, and turn 10 right. To ease computational demands, we use #Exploration [54] with static hashing to map both egocentric and panoramic views to hash codes and count their occurrences with hash table. More details in Appendix A.6.
Setups. We evaluate Habitat on the one-to-many transfer. First, we pre-train exploration policies with only intrinsic rewards in one scene. Then, we evaluate them on new scenes without further learning. Given a fixed amount steps, better policies will visit more of the new scenes.
Evaluation metries. Unlike MiniGrid, we no extrinsic rewards in Habitat. Since the agent has to navigate through rooms and spaces, we evaluate exploration policies using scene coverage measured by the agent's 's true state in Cartesian coordinates (not accessible by the agent)', Faster, larger and more uniform coverage corresponds to better exploration. Plots show mean and confidence interval over seven random seeds per method with smoothing.
4.2.1 Habitat Pre-Training Results
We pre-train exploration policies on Apartment 0 (Figure 6), one of the largest Replica scene in the dataset. Figures 9 and 11 show state coverage throughout and at the end of pre-training, respectively. C-BET explores more efficiently, covering twice as much of the scene than all baselines. In particular, at the end of pre-training it has explored almost all Apartment O uniformly. In Appendix E. we also report C-BET results when environment changes are encoded with egocentric views rather than panoramic views.
4.2.2 Habitat Transfer Results
Here, we evaluate scene coverage of pre-trained policies in seven unseen scenes for episodes of fixed steps. A better exploration policy will exhibit generalization by covering larger portion of all scenes as evenly as possible, an impressive feat given the visual complexity of the observations. Indeed, generalization is harder than MiniGrid because the lighting, colors, objects, and layout can be very different between scenes (see Figure 14 in the Appendix). Figures 10 and 12 show that, once again, C-BET clearly outperforms all baselines. Its exploration policy transfer well to all scenes, it uniformly discovers more states. No baseline comes closer its results. Actually, in many scenes baselines perform worse than a random policy.
To ease memory usage, round states 0.05 precision, e.g., 1.26 is rounded to 1.25, and 1.28 1.30.

---

C-BET
RIDE
Count
Curiosity
RND
Random
Dotio
Room
Figure 12: Scene coverage of exploration policies after 100 episodes (50,000 total steps) at offline transfer to Room O. C-BET outperforms baselines and exhibits great transfer by visiting all of the scene uniformly. In Appendix E.2, we show heatmaps for all transfer scenes.
Discussion
In this paper, we proposed a paradigm change in task-agnostic exploration. Instead of studying task-agnostic exploration in isolated environments, we proposed to (1) learn task-agnostic exploration policies from one or multiple environments, and (2) transfer learned exploration policies to unseen environments at testing time. In our setup, the agent interacts with the environment without any extrinsic goal and learns to explore environments in a task-agnostic manner. To this end, we proposed novel intrinsic reward to encourage interaction with the environment and the visitation of unseen states. Subsequently, our agent effectively transfers its exploration policy to unseen environments. Ad Advantages. The proposed two-phase framework achieves two important features, making it funda- mentally different from prior work. First, we account for environment interestingness without relying on additional models. Instead, we use a data-driven approach, estimating the rarity of states and environment changes. Rare changes are considered more interesting, actions causing them receive higher intrinsic rewards, and the agent is encouraged to perform them again. For instance, when navigating through rooms, opening doors will be more interesting due to rarity: the agent must navigate to the corresponding key, collect it, navigate to the door, and finally open it. Thus opening a door rarer than picking up key, turn rarer than simple navigation movements. Furthermore, intrinsic rewards rather than task-centric extrinsic rewards facilitates
relying on environment-centee learning from multiple environments at the same time.
Second, contrary to prior transfer and continual learning algorithms we transfer policies learned on interestingness of the environment rather than task-specific policies. In the interest-basee pre-training phase, we learn through interaction with the environment in a task-agnostic fashion, i.e., the agent freely explores the environment without any extrinsic task.
Limitations. this paper, we assumed that interacting with the environment while looking for rare changes helps find better extrinsic rewards faster. However, exploration and the task goals may be misaligned, thus a highly exploratory policy may slow down the discovery of extrinsic rewards. For instance, the environment may have dangerous states or harmful objects that the agent should avoid, even though they would make it curious during pre-training. Furthermore, C-BET is currently tied to (pseudo) to compute the rarity of states and changes. While extensions to continuous spaces exist, count-based metrics are more suited for discrete spaces.
Impact. RL can positively impact real-world problems, e.g., healthcare [19], assistive robotics [15], and climate change [43]. Yet, RL may have negative impacts, e.g., in autonomous weapons or workforce displacement [6]. Our work focuses on exploration in RL. Better understanding of what interesting do visit helps exploration in unseen environments, time with random actions. Similarly, transferring policies learned in related setting aas we do- can help narrow the range of the agent's expected behavior. Conversely, in many real-world scenarios exploration by curiosity and interestingness is unacceptable. For instance, autonomous cars cannot run over pedestrians just for the sake of curiosity. At present, our work is far from these impacts, but we hope to direct research to focus more on learning from multiple environments and transferring experiences, while at the same time ensuring the safety and reliability of autonomous agents.
the agent will not waste
Acknowledgments and Disclosure of Funding
The authors would like to thank Davide Tateo for his thoughtful discussions, Roberta Raileanu for providing support for RIDE's codebase, and Sudeep Dasari for helping run experiments. VD and DP were supported in part by NSF Fellowship and DARPA Machine Common Sense grant, respectively.
10

---

References
[1] P. Auer and R. Ortner. Logarithmic online regret bounds for undiscounted reinforcement learning. In Advances Neural Information Processing Systems (NIPS), pages 49-56, 2007.
[2] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):255-256, 2002. 2
[3] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. Van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. In International Conference on Neural Information Processing Systems (NeurIPS), 2017.
[4] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS), 2016. 1, 6,
[5] R. I. Brafman and M. Tennenholtz. R-MAX - A general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research (JMLR), 3(Oet): 213-231, 2002.
[6] E. Brynjolfsson and T. Mitchell. What can machine learning do? workforce implications. Science, 358(6370), 2017. 10
[7] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation.
In International Conference on Learning Representations (ICLR), 2019. 2, 4, 6
[8] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12875-12884, 2020. 6
[9] M. S. Charikar. Similarity estimation techniques from rounding algorithms. In Symposium on Theory of Computing, 2002. 16, 18
[10] M. Chevalier-Boisvert, L. Willems, and S. Pal. Minimalistic Gridworld Environment for OpenAI Gym, 2018. URL https:
com/maximecb///eeemmocceeeeeeeee hoyoo//ggmm--mmg syym-minigrid... 6
[11] D.-A. Clevert, T. Unterthiner, and Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs), 2015. 16
[12] C. D' D'Eramo, A. Cini, and M. Restelli. Exploiting Action-Valie uncertainty to drive exploration in reinforcement learning. In International Joint Conference Neural Networks (IJCNN), 2019.
[13] K. Dong, Y. Wang, X. Chen, and L. Wang. Q-learning with UCB exploration is sample efficient for Infinite-Horizon MDP. In International Conference on Learning Representation (ICLR), 2020. 2
[14] R. Dubey, Agrawal, D. Pathak, L. Griffiths, and A. A. Efros. Investigating human priors for playing video games. In International Conference on Machine Learning (ICML), 2018.
[15] Z. Erickson, Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp. Assistive gym: A physics simulation framework for assistive robotics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10169-10176. IEEE, 2020. 10
[16] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018. 5, 16
[17] F. Fernández and M. Veloso. Probabilistic policy reuse in reinforcement learning agent. In International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2006.
3
11

---

[18] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning (ICML), 2017. 3
[19] O. Gottesman, F. Johansson, M. Komorowski, Faisal, D. Sontag, F. Doshi Velez, and A. Celi. Guidelines for reinforcement learning in healthcare. Nature medicine, 25(1):16-15) 18, 2019. 10
[20] Gottlieb, P. Oudeyer, M. Lopes, and Baranes. iiformation-seeking curiosity, and attention: computational and neural mechanisms. Trends in Cognitive Sciences, 17(11):585-59) 2013. 3
[21] G. Hailu and G. Sommer. On amount and quality of bias in reinforcement learning. In International Conference on Systems, Man, and Cybernetics (SMC), 1999. 3, 5
[22] S. Hansen, W. Dabney, Barreto, Van de Wiele, D. Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Representations (ICLR), 2020.
[23] S. Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 9()):
1780, 1997. 16
[24] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NIPS), 2016. 2,3
[25] T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research (JMLR), 1666 2010.
[26] C. Jin, Allen in Neural Information Processing Systems (NIPS), 2018. 2
Bubeck, and M. I. Jordan, Is Q-learning provably efficient? In Advances
[27] M. Kearns and S. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209-2 222,9 2002.
[28] Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. National Academy of Sciences, 11 114(13)3351-33526 2017. 2, 3
J. Quan, T. Ramalho, A.
[29] A. S. Klyubin, D. Polani, and C. L. Nehaniv. All else being equal be empowered. In European Conference on Artificial Life, 2005. 2
[30] H. Küttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, T. Rocktäschel, and E. Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL, 2019. URL https://gi htgggthtbb...... com/ facebookreeerrr//ooccbbaaa
16
[31] T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math., (1):4 Mar 1985. 2
[32] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. F. Taylor, and P. Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21 21(181):1-50 2020. 2
[33] I. Osband, B. V. Roy, D. J. Russo, and Z. Wen. Deep exploration via randomized value functions. Journal of Machine Leurning Research (JMLR), 20(124):1 2019.
[34] G. Ostrovski, M. G. Bellemare, A. van den Oord, and Munos. Count-bases exploration with neural density models. In International Conference on Machine Learning (ICML), 2017. 1
[35] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International Conference on Machine Learning (ICML), 2017. 1, 3,
4, 5, 6,
[36] R. Raileanu and T. Rocktäschel. RIDE: Rewarding Impact-Driven Exploration for Procedurally- Generated Environments. International Conference on Learning Representations (ICLR), 2020. 3, 4, 5, 6, 16, 28
12

---

[37] J. Rajendran, R. Lewis, V. Veeriah, H. Lee, and S. Singh. How should an agent practice? In Conference on Artificial Intelligence (AAAI), 2020. 4
[38] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinforceemnnt learning via probabilistic context variables. In International Conference on Machine Learning (ICML), 2019. 2,
[39] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), 2015. 1,2
[40] M. B. Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin Austin, Texas 787 78712, 1994. 3
[41] M. B. Ring. CHILD: A first step towards continual learning. In Learning to learn, pages 261-292. Springer, 1998. 3
[42] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 3
[43] D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. Milojevic-Dupont, N. Jaques, A. Waldman-Broww et al. Tackling climate change with machine learning. arXiv preprint arXiv:1906.054::
2019, 10
[44] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. In International Conference on Learning Representations (ICLR), 2015.
[45] R. M. Ryan and E. L. Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25(1):54-67, 2000. 2
[46] M. Savva, A. Kadian, O. Maksymets, Y. Thao, R. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In International Conference on Computer Vision (ICCV), 2019. 6,9
[47] Schmidhuber. A possibility for Implementing curiosity and boredom in Model-Bulding neural controllers. In International Conference on Simulation of Adaptive Behavior (SAB), 1991. 3,
[48] Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connection Science, 18(2):13--877, 2006. 3
[49] M. Schultheis, B. Belousov, H. Abdulsamad, and J. Peters. Receding horizon curiosity. In Conference on Robot Learning (CoRL), 2019. 3,
[50] J. Schwarz, J. Luketina, W. M. Czarnecki, A. Barwins-a Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine learning (ICML), 2018. 3
[51] B. C. Stadie, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. NIPS Workshop on Deep Reinforcement Learning, 2015. 2,
[52] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L.. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe. The Replica dataset: A digital replica of indoor spaces, 2019. 9
[53] A. L. Strehl and M. L. Littman. An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences (JCSS), 139 2008
2, 3
[54] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. X. Chen, Y. Duan, J. Schulman, F. DeTurck, and Abbeel. #Exploration: study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), 2017. 9, 16
13

---

[55] Y. W. Teh, V. Bapst, W. M. Czarnecki. J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. International Conference on Neural Information Processing Systems (NeurIPS), 3017.
[56] S. Thrun and T. M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, 15 (1-2):25-44) 46, 1995.
[57] T. Tieleman and G. Hinton. Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. Technical Report., 2017. 16
[58] K. Weiss, T. M. Khoshgoftaar, and Wang. A survey of transfer learning. Journal of Big data, 3(1):9, 2016. 2
[59] D. Yarats. R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical
representations. In International Conference on Machine Learning (ICMI.), 2021. 3
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss this in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The source code is available at /ectps:/ggiiin
com/ /sppsssiicbbe///////ess
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify all the training details in Appendix A.2.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] All our plots show confidence intervals either as shaded areas or error bars.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We specify all the compute details in Appendix A.3.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We provide biblio- graphic references or URLs for the relevant resources.
(b) Did you mention the license
assets? [No] We provide instead links or references that will allow the interested reader to find out about the licenses of the various asset.
(c) Did you include any assets either in the supplemental material
URL? [Yes]
The source code is available at https: / //gitttuut thub. comm/pprrssii ppparisi/chi /ssssttt//////sccccscccsssss
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We are using existing assets in compliance with their respective licenses, that do not require direct consent by the creators.
(e) Did you discuss whether the data you using/curating contains personally identifi- able information or offensive content? [N/A] Our code does not contain personally identifiable information or offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "eRdJF9uZtj_", "reviewer": "Reviewer_gCxP", "summary": "This paper introduces a novel problem setting within the subfield of exploration in reinforcement learning, and a novel method aimed at solving this problem setting. Specifically, in the new problem setting the agent first gathers experience in several environment with no access to extrinsic reward. The agent is then transferred to similar environments where it must perform the task at hand. What makes this interesting from an exploration perspective is the fact that all the environments are hard exploration problems, so one possible good approach is to learn a good exploration policy in the initial environments, such that when the agent transfers to the new environments it can quickly optimise the extrinsic reward. The new method (CBET) is based on combining standard state-visitation counts with a novel state-change-count, and episode resetting of the counts. The method gets better performance than existing methods (which are targeted at the tabula rasa exploration setting) in terms of exploration in the initial environments and speed of learning in the new environments, with experiments on MiniGrid and Habitat.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Overview\nOverall, I believe the paper is just about worthy of acceptance. I think the novel problem setting of transfer-exploration is a useful problem setting that hasn't been studied before, and is more relavant to real-world settings than pure tabula rasa exploration. The benchmarks provided for this problem setting aren't the most thorough or rigorous (another benchmark in which transfer-performance can be measured on a down-stream task would be beneficial), but do still match the capability the problem setting aims to capture. The extensive experiments show that existing methods focused on tabular rasa exploration fail to perform as well in this new setting. The novel method (CBET) proposed for this setting contains some interesting insights about the difference between this setting and tabula rasa exploration, and shows better performance in this setting. I believe the paper could be improved with more experiments ablating different parts of CBET, to see which components are most useful for learning an exploration policy which transfers well to down-stream tasks. Further, a variety of clarifications are needed to ensure that the methodology described in the paper is clear and unambiguous.\n\n## Strengths\n\nAs described above, I think the novel problem is interesting and useful, and more relevant to real-world problems than tabular-rasa exploration. The proposed benchmarks adaquately assess the capabilities required in the problem setting. The thorough experiments on existing methods shows that they fail in this problem setting (which isn't surprising, but is useful information), and further experiments in the appendix suggest why this is (due to model learning and hence intrinsic reward collapsing relatively early for these methods). Their novel method is well motivated to solve these problems, and does demonstrate better performance in that benchmarks that existing methods. In general the paper is well written and easy to read.\n\n## Weaknesses and suggestions for improvements\nThere are several ways in which the paper isn't as strong as it could be\n\n  - While the problem setting is defined informally, it would be useful to have a formal definition, which corresponding terminology. This would enable discussion of the various parts of the problem setting more easily, and make it clear exactly the kind of problem you're trying to present and subsequently solve.\n  - While CBET is demonstrated to perform better in the benchmark, and some analysis of the method of resetting is done in the appendix, I think further analysis, and specifically ablations, would be useful to understand why CBET performs better than a simple count-based method. Specifically, the two adjustments CBET makes to a count-based method is the (multiplicative) inclusion of the state-change-count as part of the intrinsic reward, and the use of count resetting after every episode during pretraining. Ablations which remove these two components would be useful to see which of them contribute to performance, and in what way.\n  - While I don't think it's necessary for acceptance, another benchmark which had a down-stream extrinsic reward to optimise for (or adjusting the Habitat benchmark to have a downstream task which the policy is transferred to) would improve the robustness of the results and ensure that the capability being measured matches the capability the problem setting desires.\n  - Again, I don't believe this is necessary for acceptance, but the current SOTAs on MiniGrid is (I believe) [BeBold: Exploration Beyond the Boundary of Explored Regions](https://arxiv.org/abs/2012.08621) and [Adversarially Guided Actor-Critic](https://arxiv.org/abs/2102.04376), neither of which are compared against in this paper. If these comparisons could be added, it would improve the results in the paper and give evidence that CBET is a valuable new contribution. In particular AGAC doesn't suffer from intrinsic reward disappearing during training, and BeBold uses episode count resets as part of it's intrinsic reward calculation.\n\n### Smaller points\n\n  - Related to Figure 7: It would be beneficial to have an average task-specific reward (or solve rate) for CBET and each of the baselines in MiniGrid, averaged across all transfer tasks, to have a number to compare between the methods. This could be a number related to sample efficiency as well as or instead of task reward.\n  - As part of improving our understanding of CBET's performance, It would be great to see the values of the state-count and change-count plotted for different environemnts and different settings. For example, my intuition is that the change counts (and the state counts) for the Habitat environment, given the observation is so rich, should never be the same. Is this the case, or movement in the state-space discrete in this environment, and hence states are often repeated?\n  - Related to the above point on BeBold and AGAC, it would be informative to see results of CBET on the standard \"No Transfer\" setting in MiniGrid across a wider range of environments, such that it can be compared to other exploration methods which are targetted at that setting. While it isn't relevant to the main concern of your paper, it would be informative to see whether CBET's performance on the transfer-exploration setting comes from just better exploration in general, or is specifically most useful in this setting.\n  - L23: If the rewards are unknown then the goal isn't defined, so this isn't an entirely realistic setting - there's nothing to achieve. I think you should stick to sparse rewards as a motivating example for exploration.\n  - L66: Which approaches? This sentence is unclear to me.\n  - L112: \"In feature transfer, during...\" you've missed a word in this sentence I think.\n  - L115: I think it might be worth citing work such as [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://arxiv.org/abs/2008.02790), which is a meta-RL work which aims to learn an exploration policy which is useful in down-stream tasks, and is hence similar in some sense to your problem setting. In general, I think it could be worth contrasting with this kind of Meta RL setting in the related work setting. In L117 you claim that task-specific policies are transferred, but this generally isn't the case in the Meta-RL setting.\n  - L139: \"[...]and the changes that are rare\": add \"that\"\n  - L158: At this point it would be worth discussing the limitations of count-resets - each episode the agent could explore the same parts of the environment, effectively memorising a policy which hits all the novel states and interesting changes it can, but never exploring the environment beyond that point.\n  - L226: \"...ten new environments, seven of which are new.\" - This should be just \"...ten environments, seven of which are new...\" I think.\n  - L240: That's quite a lot of smoothing - are evaluations very high variance? What do less smoothed curves look like?\n\n### Clarification Questions\n  - What's the difference between CBET (Ours) and Our Reward in Figure 7?\n  - In Figure 7 no-transfer MultiRoom-N12-S10, RIDE gets 0 reward after 100M time-steps, but in the original RIDE paper, RIDE achieves 0.6 extrinsic reward after ~20M time-steps (see Figure 3 (h) in that paper). Do you have an idea why there's that discrepancy in results? Other results from the RIDE paper roughly match the corresponding no-transfer experiments in your paper.\n  - In your experiments do you continue training with the intrinsic reward combined with the extrinsic reward after transfering to the new environments, or do you solely train with extrinsic reward after transfer and intrinsic reward before transfer? If you're only training the pre-trained policy with extrinsic reward in the new environment that might explain the discrepancy in scores pointed out in the bullet point above, but I also think that it's not a valid evaluation methodology - there's no reason to disallow intrinsic rewards and further exploration bonuses being used after transfer to a new environment.\n  - L178: Your method is unclear to me here: IMPALA learns a state-value function $V(s)$, and a policy $\\pi(a|s)$, but doesn't learn a Q-function $Q_i(s,a)$ as you've written it. How do you adapt IMPALA to learn a Q-function?\n\n## Conclusion\nI believe the paper is marginally above the acceptance threshold, but I think if the adjustments and additional experiments described above are added (especially the ablations) then I'd be willing to raise my score further.\n\n## Edit:\nGiven the comments in this response and your response to other authors, I believe my confusions have been clarified, and I believe the paper will be much improved with the improvements you've described here and elsewhere. Hence, I'm raising my score to an 8 - I believe the paper is an interesting, novel and relevant problem setting and presents a novel approach to solving it, while also bein rigorous in the evaluation of their and previous works to ensure that their contribution is robustly useful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Comments on limiations have been adaquately addressed in the paper, with any additional concerns raised in the review above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "NjGe3cVOqH", "reviewer": "Reviewer_QKaX", "summary": "This paper proposes a new paradigm for exploration based on transfer. They argue that we should not focus on exploration in isolated environments (it is not realistic for humans either), but instead focus on a sequence of tasks. More interestingly, they propose that we need to disentangle the exploration signal, into one agent-centric one based on visitation counts, but also a environment-centric one that identifies states where we get a change in the environment, and are thereby inherently interesting over tasks. They propose a new intrinsic reward that combines both properties, and test these in gridworld experiments and first-person navigation tasks, where they perform on par with or better than baseline exploration methods. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong:\n* I really like the main idea: the fact that certain objects are interesting by definition, not in a single task, but over repeated tasks. This is indeed a novel perspective that I have not seen before, but makes a lot of sense.\n* Extensive experimentation, interesting observations. For example that transfer using model characteristics is not efficient at all for transfer. \n\nWeaker:\n* I would like a more detailed comparison to Raileanu and Rocktäschel [36]. Since the claims in the introduction are quite strong (a completely new paradigm for exploration), you should motivate this carefully compared to related work (I think the statement is a bit strong).\n* I am a bit in doubt about the “count resets”. I understand the reasoning, but it may actually be harmfull within a task (if there is one state we have not visited yet, then keeping the counts this may be rewarded highly, while with a reset we have to start from scratch again). I think the paper at least needs to discuss this, and a more elegant solution (which allows optimal exploration within the task as well as for transfer) would be preferable.\n* I am a bit confused between c(s, a, s) (e.g. L215) which is a vector, and N( c) in Eq. 1. Is N( c) the sum over the vector? \n* I am also confused about the panoramic images (L283-288). How do you align these for the equation in L 290 (otherwise the element-wise difference does not make sense right, if you have rotated your view every element will change).\n* Fig 10: I find the Habitat interpretation really qualitative, the main result being “that C-Bet is the only policy visiting the right-hand corner”. Sure, but RIDE is the only one to visit the top-middle corner? \n\nSummary:\nI really like the idea of this paper, that task-agnostic exploration should be separated into generic environment-centric properties (objects that are always interesting) and agent-specific properties depending on the interaction history (such as novelty). I think this is an important insight, especially that one can estimate the former property during training. There are some smaller issue with the paper as mentioned above, but none of them is really big. The experiments are sounds as well, and indeed show that the proposed method works. I recommend this paper for acceptance, since I believe it will definitely interest the broader RL exploration community.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Gj9G0ZGNm9B", "reviewer": "Reviewer_baKZ", "summary": "This paper focuses on task-agnostic exploration in RL, with two key ideas:\n- introducing a new evaluation method for task-agnostic exploration, by measuring \"how well\" the learnt exploration policy behaves in new tasks; \"how well\" can either be measured based on extrinsic return in the withheld task, or on exploration-related quantities like coverage\n- proposing a new count-based intrinsic reward that is shown to outperform other baselines in the transfer task in procedurally-generated gridworlds, and in the coverage task in 3D scenes from Habitat.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is well written with a wealth of experimental results, as well as thorough in the Related work section.\n\nOriginality: \n- The idea of measuring how well a policy transfers is not new, but I have not seen it applied specifically to evaluate exploration policies.\n- As far as I know, combining state visitation count (already established in the literature) with change count has never been published.\n\nContribution/significance: I think good enough, with one \"medium\" and one minor contribution.\n- The first problem addressed by the paper -- evaluating task-agnostic exploration -- is an important one in the community, and I think the proposed protocol has merits. There are a few limitations that I think currently limit the applicability of the method, but they seem somewhat easy to overcome and are clearly identified in the paper. To me this is a significant enough contribution.\n- The proposed C-BET exploration algorithm seems to achieve good performance according to that metric. I have to say I'm always a bit wary of papers that both introduce a new benchmark and a new algorithm that outperforms its competitors on that benchmark, but in this instance the relatively thorough set of baselines offsets this issue. However I'm not sure how generally applicable C-BET is (both in terms of scalability and applicability).\n\nQuestions:\n- How scalable is C-BET when the state-action space grows? In the Habitat experiments, the authors had to use a very coarse action space with 3 actions (thus drastically reducing the complexity from a 3D env to a gridworld with rich observations). Does C-BET still perform reasonably well with, eg, 30° vertical and horizontal rotations?\n- How does C-BET perform in environments with intra-episode stochasticity (eg, noisy tv)? The 1/sqrt(N(c)) will ultimately compensate for this if the episode is long enough, but because counts are reset every episode, the additional |A| factor (|S||A| state-action pairs to visit instead of just |S| states) seems like it can significantly hurt performance in practical setups.\n- Figure 7: Shouldn't the results on MultiRoom-N12-S10 (\"no transfer\" row) match those of Fig 3h in https://arxiv.org/pdf/2002.12292.pdf? RIDE should get to ~0.6 in 25M frames.\n- Figure 9: it's not clear to me how to interpreter error bars. Is the colored bar showing the average and the top of the error bar the maximum over 5 seeds? If so, what about the minimum?\n- Section 5, \"advantage\" paragraph (and scattered throughout the rest of the paper): I'm not sure I see how C-BET accounts for \"rarity\" of change, since eg the agent's position is presumably part of the state, and C-BET doesn't have any form of (action, state) embedding that would be able to learn translational (or other) invariances? Presented with an environment with, eg, N on/off switches that are part of the state but do nothing interesting, won't C-BET be tempted to explore all 2^N combinations?\n\nExtra comment: unless negative rewards exist in MiniGrid, something seems off with the smoothing of curves (eg Fig7)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is a clear and fair acknowledgement of limitations in an explicit section, which I really appreciated.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "EIU8IIYZ-MM", "reviewer": "Reviewer_3Bvy", "summary": "* This paper challenges the contemporary experimental setup of exploration algorithms, and suggests that we should not start exploring in new environments tabula-rasa, and instead use knowledge derived from prior experience to bootstrap exploration. \n* The paper also identifies issues with using existing tabula-rasa exploration algorithms for the new setting, and propose a new method C-BET to overcome those shortcomings. \n* C-BET and a bunch of existing exploration algorithms are benchmarked on newly constructed transfer exploration setups in the MiniGrid and Habitat environments. ", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper offers a fresh perspective on training and evaluation exploration algorithms for RL environments. The current paradigm in training exploration methods is to start \"tabula-rasa\" in each environment, with no pre-conceived knowledge about the world. This means that the agent has to blindly try-out and explore a combinatorically large state-action space which then leads to requiring billions of frames or years of experience to solve sparse reward problems, something that's impractical for real-world usage. Not to mention the inherent safety issues involved with tabula-rasa exploration. The paper proposes a new experimental setup to evaluate exploration algorithms: the algorithms are first pre-trained in a task-agnostic manner on some environments and then transferred to unseen environments. \n\nThe writing in the paper is clear throughout and it does good job of explaining why this different experimental setup makes sense. I also appreciate the generous use of figures to make the exposition clear. One of the most important contributions of this paper is the construction of new experimental setups for exploration transfer, and benchmarking existing methods on them. These experimental setups cover MiniGrid (a suite of GridWorld environments) and Habitat (a suite of visual 3D environments), thus covering a wide spectrum of scenarios. \n\nIts interesting to see curiosity like methods fail during transfer, but somewhat obvious in hindsight. After learning a predictive model of the world, similar states are not so surprising anymore even though they might still be inherently interesting in the new environment. C-BET tries to fix this by using a pseudo count of interesting changes along with count resets every episode. The results on pre-training and task-transfer are quite compelling. The analysis in the supplementary material is quite informative and its interesting to see that a simple trick like count resets has such a large impact. \n\nOverall, I really like this paper, the ideas and perspective on exploration are original, presented well and are of great significance. I would have recommended acceptance even without the introduction of a new method, but that is an added bonus. I do have some comments / questions though and it would great if the authors can expand on it: \n* Can you benchmark / ablate the impact of *change* pseudo counts? Specifically, running your \"Count\" baseline along with episode resets, and comparing with C-BET (count + change + episode resets) would clarify this. \n* In the Habitat experiments, the use of panaromic images seems like leveraging access to privileged informtation that other methods don't have. Have you tried experimenting with using egocentric images for pseduo-counts and the change encoding? Does it lead to a significant performance drop? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new problem setup for exploration in RL, which is to allow the agent to learn an exploratory policy in the absence of reward/task and evaluate how quickly the agent learns when the task is given in the transfer phase. All of the reviewers highly appreciated this problem setup and agreed that this setup is reasonable and deserves more attention, though a similar setup has been considered in the past [1], which should be cited and discussed in the paper. In addition, the paper proposes a new count-based exploration method that encourages rare state changes and proposes to reset count at every episode to overcome the vanishing intrinsic reward issue specifically for the new proposed problem setup. The results on MiniGrid and Habitats environments show that the proposed method performs better than baselines in both the exploration phase and the transfer phase. The authors promised to address several minor questions around count reset and ablation in the revision. Overall, the new problem setup and the proposed method would be quite interesting to the research community. Thus, I recommend acceptance for this paper.

[1] Rajendran et al., How Should an Agent Practice?
Decision: Accept (Oral)