Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

1 Introduction . Exploration is one of the key unsolved problems in building intelligent agents capable of behaving like humans . In reinforcement learning ( RL ) , exploration is usually studied under two different settings . The first is task-driven exploration , where the reward is well-defined and the agent ’ s goal is to explore in order to maximize long-term rewards . However , in real life , external rewards are either sparse or unknown altogether . In this setting , exploration is task-agnostic : given a new environment , the agent has to explore it in absence of any external reward . Common approaches to encourage task-agnostic exploration use intrinsically motivated rewards such as prediction curiosity [ 35 , 47 ] , empowerment [ 39 ] , or visitation counts [ 4 , 34 ] . But does this setup represent how humans explore ? We argue that the commonly-used task-agnostic exploration setup is unrealistic , both from practical and academic viewpoints . This setup assumes environments in isolation and agents exploring tabularasa , i.e. , with no prior knowledge or experience . By contrast , we as humans do not learn from one environment in isolation and we do not throw away our past knowledge every time we encounter a new environment [ 14 ] . Exploration is rather a lifelong process : every time we encounter new environments , we use our prior knowledge and experience to develop new efficient exploration strategies . In this paper , we view the exploration problem from a continual learning lens . More specifically , in this setup , the learning agent interacts with one or many environments without any extrinsic goal . At this time , the agent learns to explore the environments . Later on , the agent effectively transfers the learned exploration policy to explore new environments , rather than exploring the new environment tabula-rasa . ∗Equal contribution . Contacts : sparisi @ fb.com and vdean @ cmu.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Learn to Explore Transfer Interest-Based Exploration away from stand-alone isolated task-agnostic environment exploration to a more realistic multienvironment transfer-exploration setup2 . We show how to learn exploration policies both from single- and multi-environment interaction , and how to transfer them to unseen environments . This transfer-exploration setup allows agents to use prior experiences for learning task-agnostic exploration . Notably , classic stand-alone task-agnostic approaches were designed for tabula-rasa exploration and hence only explore in an agent-centric manner . They fail to capture the inherent interestingness of some environment components . With this insight , we propose Change-Based Exploration Transfer ( CBET ) , a simple yet effective approach learning joint agent-centric and environment-centric exploration . The key idea is for an agent to seek out both surprises ( unseen areas ) and high-impact ( interesting ) components of the environment . The experiments show that C-BET ( a ) learns more effectively when placed in a multi-environment setup , and ( b ) either outperforms or performs competitively with prior methods across several unseen testing environments . We hope this paper will inspire exploration research to focus more on learning from multiple environments and transferring experiences rather than tabula-rasa exploration . 2 Preliminaries and Related Work . We consider environments governed by Markov Decision Processes ( MDPs ) . In MDPs , an agent observes the state of the environment s and selects actions a according to a policy π ( a|s ) . In turn , the environment changes , providing a new observation s′ and a reward r. Through environment interaction , the agent collects episodes , i.e. , sequences of states , actions and rewards ( st , at , rt ) t=1 ... T . The goal of RL is to learn a policy maximizing the sum of rewards during episodes , i.e. , the return . In this setting , exploration poses many questions . If the environment provides no rewards , what should the agent look for ? When should it act greedily with respect to the rewards it has found and stop looking for more ? In the history of RL , many approaches have been proposed to tackle these questions . On one hand , classic single-environment approaches range from intrinsic motivation with visitation counts [ 2 , 4 , 13 , 26 , 53 ] , optimism [ 1 , 5 , 25 , 27 , 31 ] , or curiosity [ 7 , 24 , 35 , 45 , 49 , 51 ] , to bootstrapping [ 12 , 33 ] or empowerment [ 29 , 39 ] . On the other hand , we find approaches to incrementally learn tasks , such as transfer learning [ 58 ] , continual learning [ 28 ] , curriculum learning [ 32 ] , and meta learning [ 38 ] . Below , we review approaches closely related to ours . 2While it can be argued that the real world has no explicit distinction between training and testing , we use this dichotomy only for the purpose of evaluation . Intrinsic motivation . Exploration strategies relying on intrinsic rewards date back to Schmidhuber [ 47 ] , who proposed to encourage exploration by visiting hard-to-predict states . More recently , the idea of auxiliary rewards to make up for the lack of external rewards has been extensively studied in RL , supported by evidence from psychology and neuroscience [ 20 ] . Several intrinsic rewards have been proposed , ranging from visitation count bonuses [ 4 , 53 ] to bonuses based on prediction error of some quantity . For example , the agent may learn a dynamics model and try to predict the next state [ 24 , 35 , 48 , 51 ] . By giving a bonus proportional to the prediction error , the agent is incentivized to explore unpredictable states . Schultheis et al . [ 49 ] , instead , proposed to learn intrinsic rewards function by maximizing extrinsic rewards by meta-gradient . However , in these approaches exploration is agent-centric , i.e. , based on an agent ’ s belief such as the forward model error . In contrast , with this work we propose additionally learning environment-centric exploration policies . C-BET neither requires a model nor knowledge of extrinsic rewards . Instead , it encourages the agent to perform actions causing interesting changes to the environment . We should note that while Raileanu and Rocktäschel [ 36 ] proposed a similar approach , their exploration policy lacks the transfer component and also requires to learn models . Transfer learning . The idea of agents capable of incrementally learning tasks is well-known in the field of machine learning , with the first approaches dating back to the 90s ’ [ 40 , 41 , 56 ] . In RL , recent methods have focused on policy and feature transfer . In the former , a pre-trained agent ( teacher ) is used to transfer behaviors to a new agent ( student ) . Examples include policy distillation , where the student is trained to minimize the Kullback-Leibler divergence to the teacher [ 44 ] or to multiple teachers at the same time [ 55 ] . Alternative approaches , instead , directly reuse policies from source tasks to build the student policy [ 3 , 17 , 21 ] . In feature transfer , a pre-learned state representation is used to encourage exploration when tasks are presented to the agent [ 22 , 59 ] . Similar to transfer RL , continual RL studies how learning on one or more tasks can help accelerate learning on different tasks , and how to prevent catastrophic forgetting [ 28 , 42 , 50 ] . Meta RL , instead , tries to exploit underlying common structures between tasks to learn new tasks more quickly [ 18 , 38 ] . However , the setup in these approaches is not task-agnostic , i.e. , task-specific policies are transferred rather than exploration policies . For example , after learning a policy maximizing the rewards of one task , the agent starts exploring guided by the same policy as a second task is given . Transfer is taskcentric rather than task-agnostic and environment-centric . Consequently , if tasks are too dissimilar information can not be reused , even if the environments are similar . By contrast , in this work we propose learning task-agnostic exploration from one or many environments and show transfer to unseen environments . We should note that while Pathak et al . [ 35 ] did demonstrate fine-tuning on different maze maps , their focus and large-scale evaluations remain on tabula-rasa exploration . 3 Learning to Explore . Our goal is to decouple the environment-centric nature of exploration from its agent-centric component . Contrary to prior work , we propose to first learn an environment-centric exploration policy and then to transfer it to unseen environments . The policy is driven by the inherent interestingness of states and is learned over time via interaction . First , during a pre-training phase , the agent interacts with many environments without any tasks and learns an exploration policy . Then , when new environments and tasks are presented , the agent uses the previously learned policy to explore more efficiently and learn task-specific policies . C-BET ’ s key components are ( 1 ) a novel intrinsic reward and the learning of a policy to disentangle exploration from exploitation , and ( 2 ) the use of such policy to help exploration for new tasks . Figures 2 and 3 summarize our framework . We should note that Rajendran et al . [ 37 ] also proposed a transfer framework based on intrinsic rewards . In their work , the agent switches between practice episodes –where the agent receives only intrinsic rewards– and match episodes –giving only extrinsic rewards . However , practice episodes are simpler variations of match episodes ( e.g. , in Atari Pong the agent practices against itself ) rather than different tasks as in C-BET . Furthermore , the intrinsic reward used in practice episodes is given by a function trained with meta-gradients to improve the extrinsic-reward return . That is , exploration is not task-agnostic as in C-BET , and extrinsic rewards are the main drive of the agent . 3.1 Interestingness of State-Action Pairs . The natural world is filled with states or scenarios that are inherently interesting and our goal is to capture this inherent interestingness via intrinsic rewards . In this paper , we propose adding an environment-centric component of interestingness to the existing agent-centric component of surprise . Specifically , we hypothesize that the environment can change on interaction , and the changes that are rare are inherently interesting . That is , we penalize actions not affecting the environment , and favor actions producing rare changes . For instance , moving around , bumping into walls , or trying to open locked doors without keys all result in no change and thus will be of low interest . We also want to keep the agent-centric component in exploration –that is , the exploration policy should look for surprises or unseen states . Thus , we further reward actions leading to less-visited states . By combining these two components , the resulting C-BET interest-based reward is ri ( s , a , s ′ ) = 1/ ( N ( s′ ) +N ( c ) ) , ( 1 ) where c ( s , s′ ) defines the environment change of a transition ( s , a , s′ ) , and N denotes ( pseudo ) counts of changes and states . Figure 4 empirically shows its effectiveness . In Section 4 we discuss change encodings used in our experiments . 3.2 Exploration Learning . In this phase , we want to learn task-agnostic exploration policies from interaction with many environments . The agent has no goal , but states where it can ‘ die ’ are still terminal . In this setting , we would like to treat the problem of learning exploration as an MDP with intrinsic-rewards only , and train the agent to maximize discounted intrinsic-returns averaged over episodes . Formally , the agent explores many environments EEXP = { E1 , E2 , . . . , EN } , each governed by MDP 〈Sn , A , P , ri , γi〉 . That is , each environment has its own states but the action space is the same , and all environments obey the same dynamics P and the same intrinsic reward function ri . The agent ’ s goal is to learn an exploration policy maximizing the sum of discounted intrinsic rewards , i.e. , πEXP ( s , a ) = argmaxπ EE , π [ ∑ t γ t iri ( st , at ) ] . To approximate the average , after a maximum number of steps the environment is reset and a new episode starts , as typically done in RL . However , both common [ 7 , 35 , 36 ] and Eq . ( 1 ) intrinsic rewards decrease over time as the agent explores , to the point that they vanish to zero given enough samples . For instance , counts will grow to infinity , or prediction models error will go to zero . While this is not an issue in the tabula-rasa setup where the agent also gets extrinsic rewards , it can be problematic in the proposed task-agnostic exploration framework . Any policy , indeed , would be optimal if all rewards are zero . To prevent Eq . ( 1 ) from vanishing , we randomly reset counts any given time step . To explain why resets need to be random , we start by considering ‘ episodic counts ’ proposed by Raileanu and Rocktäschel [ 36 ] . These counts are reset at the beginning of every episode to ensure that the agent does not go back and forth between a sequence of states with high rewards . While this work fine when extrinsic rewards are also given , it can be a problem if we learn only on intrinsic rewards . When counts are reset , the agent ‘ forgets ’ past trajectories and thinks that every state and change is new . If resets always happen at the end of an episode , then initial states will always get higher reward . Moreover , starting always with zero-counts may favor some trajectories and penalize others . A DBC Consider for example the chainworld in Figure 5 . The agent always starts in A , from where it can go to B or D. From B , it loops between B and C. From D , it can not go anywhere else . The optimal exploration policy should visit all states uniformly , by randomly going to B and D. However , if we reset counts at every episode the agent forgets that it has already visited B and C. Thus , the intrinsic rewards for B and C are high again , and trajectory ABCBCBC ... gives higher intrinsic return than ADDD ... Consequently , the optimal policy with respect to episodic counts will always prefer to visit B rather than D .

Summary: This paper introduces a novel problem setting within the subfield of exploration in reinforcement learning, and a novel method aimed at solving this problem setting. Specifically, in the new problem setting the agent first gathers experience in several environment with no access to extrinsic reward. The agent is then transferred to similar environments where it must perform the task at hand. What makes this interesting from an exploration perspective is the fact that all the environments are hard exploration problems, so one possible good approach is to learn a good exploration policy in the initial environments, such that when the agent transfers to the new environments it can quickly optimise the extrinsic reward. The new method (CBET) is based on combining standard state-visitation counts with a novel state-change-count, and episode resetting of the counts. The method gets better performance than existing methods (which are targeted at the tabula rasa exploration setting) in terms of exploration in the initial environments and speed of learning in the new environments, with experiments on MiniGrid and Habitat.