Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

1 Introduction . Exploration is one of the key unsolved problems in building intelligent agents capable of behaving like humans . In reinforcement learning ( RL ) , exploration is usually studied under two different settings . The first is task-driven exploration , where the reward is well-defined and the agent ’ s goal is to explore in order to maximize long-term rewards . However , in real life , external rewards are either sparse or unknown altogether . In this setting , exploration is task-agnostic : given a new environment , the agent has to explore it in absence of any external reward . Common approaches to encourage task-agnostic exploration use intrinsically motivated rewards such as prediction curiosity [ 35 , 47 ] , empowerment [ 39 ] , or visitation counts [ 4 , 34 ] . But does this setup represent how humans explore ? We argue that the commonly-used task-agnostic exploration setup is unrealistic , both from practical and academic viewpoints . This setup assumes environments in isolation and agents exploring tabularasa , i.e. , with no prior knowledge or experience . By contrast , we as humans do not learn from one environment in isolation and we do not throw away our past knowledge every time we encounter a new environment [ 14 ] . Exploration is rather a lifelong process : every time we encounter new environments , we use our prior knowledge and experience to develop new efficient exploration strategies . In this paper , we view the exploration problem from a continual learning lens . More specifically , in this setup , the learning agent interacts with one or many environments without any extrinsic goal . At this time , the agent learns to explore the environments . Later on , the agent effectively transfers the learned exploration policy to explore new environments , rather than exploring the new environment tabula-rasa . ∗Equal contribution . Contacts : sparisi @ fb.com and vdean @ cmu.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Learn to Explore Transfer Interest-Based Exploration away from stand-alone isolated task-agnostic environment exploration to a more realistic multienvironment transfer-exploration setup2 . We show how to learn exploration policies both from single- and multi-environment interaction , and how to transfer them to unseen environments . This transfer-exploration setup allows agents to use prior experiences for learning task-agnostic exploration . Notably , classic stand-alone task-agnostic approaches were designed for tabula-rasa exploration and hence only explore in an agent-centric manner . They fail to capture the inherent interestingness of some environment components . With this insight , we propose Change-Based Exploration Transfer ( CBET ) , a simple yet effective approach learning joint agent-centric and environment-centric exploration . The key idea is for an agent to seek out both surprises ( unseen areas ) and high-impact ( interesting ) components of the environment . The experiments show that C-BET ( a ) learns more effectively when placed in a multi-environment setup , and ( b ) either outperforms or performs competitively with prior methods across several unseen testing environments . We hope this paper will inspire exploration research to focus more on learning from multiple environments and transferring experiences rather than tabula-rasa exploration . 2 Preliminaries and Related Work . We consider environments governed by Markov Decision Processes ( MDPs ) . In MDPs , an agent observes the state of the environment s and selects actions a according to a policy π ( a|s ) . In turn , the environment changes , providing a new observation s′ and a reward r. Through environment interaction , the agent collects episodes , i.e. , sequences of states , actions and rewards ( st , at , rt ) t=1 ... T . The goal of RL is to learn a policy maximizing the sum of rewards during episodes , i.e. , the return . In this setting , exploration poses many questions . If the environment provides no rewards , what should the agent look for ? When should it act greedily with respect to the rewards it has found and stop looking for more ? In the history of RL , many approaches have been proposed to tackle these questions . On one hand , classic single-environment approaches range from intrinsic motivation with visitation counts [ 2 , 4 , 13 , 26 , 53 ] , optimism [ 1 , 5 , 25 , 27 , 31 ] , or curiosity [ 7 , 24 , 35 , 45 , 49 , 51 ] , to bootstrapping [ 12 , 33 ] or empowerment [ 29 , 39 ] . On the other hand , we find approaches to incrementally learn tasks , such as transfer learning [ 58 ] , continual learning [ 28 ] , curriculum learning [ 32 ] , and meta learning [ 38 ] . Below , we review approaches closely related to ours . 2While it can be argued that the real world has no explicit distinction between training and testing , we use this dichotomy only for the purpose of evaluation . Intrinsic motivation . Exploration strategies relying on intrinsic rewards date back to Schmidhuber [ 47 ] , who proposed to encourage exploration by visiting hard-to-predict states . More recently , the idea of auxiliary rewards to make up for the lack of external rewards has been extensively studied in RL , supported by evidence from psychology and neuroscience [ 20 ] . Several intrinsic rewards have been proposed , ranging from visitation count bonuses [ 4 , 53 ] to bonuses based on prediction error of some quantity . For example , the agent may learn a dynamics model and try to predict the next state [ 24 , 35 , 48 , 51 ] . By giving a bonus proportional to the prediction error , the agent is incentivized to explore unpredictable states . Schultheis et al . [ 49 ] , instead , proposed to learn intrinsic rewards function by maximizing extrinsic rewards by meta-gradient . However , in these approaches exploration is agent-centric , i.e. , based on an agent ’ s belief such as the forward model error . In contrast , with this work we propose additionally learning environment-centric exploration policies . C-BET neither requires a model nor knowledge of extrinsic rewards . Instead , it encourages the agent to perform actions causing interesting changes to the environment . We should note that while Raileanu and Rocktäschel [ 36 ] proposed a similar approach , their exploration policy lacks the transfer component and also requires to learn models . Transfer learning . The idea of agents capable of incrementally learning tasks is well-known in the field of machine learning , with the first approaches dating back to the 90s ’ [ 40 , 41 , 56 ] . In RL , recent methods have focused on policy and feature transfer . In the former , a pre-trained agent ( teacher ) is used to transfer behaviors to a new agent ( student ) . Examples include policy distillation , where the student is trained to minimize the Kullback-Leibler divergence to the teacher [ 44 ] or to multiple teachers at the same time [ 55 ] . Alternative approaches , instead , directly reuse policies from source tasks to build the student policy [ 3 , 17 , 21 ] . In feature transfer , a pre-learned state representation is used to encourage exploration when tasks are presented to the agent [ 22 , 59 ] . Similar to transfer RL , continual RL studies how learning on one or more tasks can help accelerate learning on different tasks , and how to prevent catastrophic forgetting [ 28 , 42 , 50 ] . Meta RL , instead , tries to exploit underlying common structures between tasks to learn new tasks more quickly [ 18 , 38 ] . However , the setup in these approaches is not task-agnostic , i.e. , task-specific policies are transferred rather than exploration policies . For example , after learning a policy maximizing the rewards of one task , the agent starts exploring guided by the same policy as a second task is given . Transfer is taskcentric rather than task-agnostic and environment-centric . Consequently , if tasks are too dissimilar information can not be reused , even if the environments are similar . By contrast , in this work we propose learning task-agnostic exploration from one or many environments and show transfer to unseen environments . We should note that while Pathak et al . [ 35 ] did demonstrate fine-tuning on different maze maps , their focus and large-scale evaluations remain on tabula-rasa exploration . 3 Learning to Explore . Our goal is to decouple the environment-centric nature of exploration from its agent-centric component . Contrary to prior work , we propose to first learn an environment-centric exploration policy and then to transfer it to unseen environments . The policy is driven by the inherent interestingness of states and is learned over time via interaction . First , during a pre-training phase , the agent interacts with many environments without any tasks and learns an exploration policy . Then , when new environments and tasks are presented , the agent uses the previously learned policy to explore more efficiently and learn task-specific policies . C-BET ’ s key components are ( 1 ) a novel intrinsic reward and the learning of a policy to disentangle exploration from exploitation , and ( 2 ) the use of such policy to help exploration for new tasks . Figures 2 and 3 summarize our framework . We should note that Rajendran et al . [ 37 ] also proposed a transfer framework based on intrinsic rewards . In their work , the agent switches between practice episodes –where the agent receives only intrinsic rewards– and match episodes –giving only extrinsic rewards . However , practice episodes are simpler variations of match episodes ( e.g. , in Atari Pong the agent practices against itself ) rather than different tasks as in C-BET . Furthermore , the intrinsic reward used in practice episodes is given by a function trained with meta-gradients to improve the extrinsic-reward return . That is , exploration is not task-agnostic as in C-BET , and extrinsic rewards are the main drive of the agent . 3.1 Interestingness of State-Action Pairs . The natural world is filled with states or scenarios that are inherently interesting and our goal is to capture this inherent interestingness via intrinsic rewards . In this paper , we propose adding an environment-centric component of interestingness to the existing agent-centric component of surprise . Specifically , we hypothesize that the environment can change on interaction , and the changes that are rare are inherently interesting . That is , we penalize actions not affecting the environment , and favor actions producing rare changes . For instance , moving around , bumping into walls , or trying to open locked doors without keys all result in no change and thus will be of low interest . We also want to keep the agent-centric component in exploration –that is , the exploration policy should look for surprises or unseen states . Thus , we further reward actions leading to less-visited states . By combining these two components , the resulting C-BET interest-based reward is ri ( s , a , s ′ ) = 1/ ( N ( s′ ) +N ( c ) ) , ( 1 ) where c ( s , s′ ) defines the environment change of a transition ( s , a , s′ ) , and N denotes ( pseudo ) counts of changes and states . Figure 4 empirically shows its effectiveness . In Section 4 we discuss change encodings used in our experiments . 3.2 Exploration Learning . In this phase , we want to learn task-agnostic exploration policies from interaction with many environments . The agent has no goal , but states where it can ‘ die ’ are still terminal . In this setting , we would like to treat the problem of learning exploration as an MDP with intrinsic-rewards only , and train the agent to maximize discounted intrinsic-returns averaged over episodes . Formally , the agent explores many environments EEXP = { E1 , E2 , . . . , EN } , each governed by MDP 〈Sn , A , P , ri , γi〉 . That is , each environment has its own states but the action space is the same , and all environments obey the same dynamics P and the same intrinsic reward function ri . The agent ’ s goal is to learn an exploration policy maximizing the sum of discounted intrinsic rewards , i.e. , πEXP ( s , a ) = argmaxπ EE , π [ ∑ t γ t iri ( st , at ) ] . To approximate the average , after a maximum number of steps the environment is reset and a new episode starts , as typically done in RL . However , both common [ 7 , 35 , 36 ] and Eq . ( 1 ) intrinsic rewards decrease over time as the agent explores , to the point that they vanish to zero given enough samples . For instance , counts will grow to infinity , or prediction models error will go to zero . While this is not an issue in the tabula-rasa setup where the agent also gets extrinsic rewards , it can be problematic in the proposed task-agnostic exploration framework . Any policy , indeed , would be optimal if all rewards are zero . To prevent Eq . ( 1 ) from vanishing , we randomly reset counts any given time step . To explain why resets need to be random , we start by considering ‘ episodic counts ’ proposed by Raileanu and Rocktäschel [ 36 ] . These counts are reset at the beginning of every episode to ensure that the agent does not go back and forth between a sequence of states with high rewards . While this work fine when extrinsic rewards are also given , it can be a problem if we learn only on intrinsic rewards . When counts are reset , the agent ‘ forgets ’ past trajectories and thinks that every state and change is new . If resets always happen at the end of an episode , then initial states will always get higher reward . Moreover , starting always with zero-counts may favor some trajectories and penalize others . A DBC Consider for example the chainworld in Figure 5 . The agent always starts in A , from where it can go to B or D. From B , it loops between B and C. From D , it can not go anywhere else . The optimal exploration policy should visit all states uniformly , by randomly going to B and D. However , if we reset counts at every episode the agent forgets that it has already visited B and C. Thus , the intrinsic rewards for B and C are high again , and trajectory ABCBCBC ... gives higher intrinsic return than ADDD ... Consequently , the optimal policy with respect to episodic counts will always prefer to visit B rather than D .

Summary: This paper proposes a new paradigm for exploration based on transfer. They argue that we should not focus on exploration in isolated environments (it is not realistic for humans either), but instead focus on a sequence of tasks. More interestingly, they propose that we need to disentangle the exploration signal, into one agent-centric one based on visitation counts, but also a environment-centric one that identifies states where we get a change in the environment, and are thereby inherently interesting over tasks. They propose a new intrinsic reward that combines both properties, and test these in gridworld experiments and first-person navigation tasks, where they perform on par with or better than baseline exploration methods.