BeBold: Exploration Beyond the Boundary of Explored Regions

1 INTRODUCTION . Deep reinforcement learning ( RL ) has experienced significant progress over the last several years , with impressive performance in games like Atari ( Mnih et al. , 2015 ; Badia et al. , 2020a ) , StarCraft ( Vinyals et al. , 2019 ) and Chess ( Silver et al. , 2016 ; 2017 ; 2018 ) . However , most work requires either a manually-designed dense reward ( Brockman et al. , 2016 ) or a perfect environment model ( Silver et al. , 2017 ; Moravčı́k et al. , 2017 ) . This is impractical for real-world settings , where the reward is sparse ; in fact , the proper reward function for a task is often even unknown due to lack of domain knowledge . Random exploration ( e.g. , -greedy ) in these environments is often insufficient and leads to poor performance ( Bellemare et al. , 2016 ) . Recent approaches have proposed to use intrinsic rewards ( IR ) ( Schmidhuber , 2010 ) to motivate agents for exploration before any extrinsic rewards are obtained . Various criteria have been proposed , including curiosity/surprise-driven ( Pathak et al. , 2017 ) , count-based ( Bellemare et al. , 2016 ; Burda et al. , 2018a ; b ; Ostrovski et al. , 2017 ; Badia et al. , 2020b ) , and state-diff approaches ( Zhang et al. , 2019 ; Marino et al. , 2019 ) . Each approach has its upsides and downsides : Curiosity-driven approaches look for prediction errors in the learned dynamics model and may be misled by the noisy TV ( Burda et al. , 2018b ) problem , where environment dynamics are inherently stochastic . Count-based approaches favor novel states in the environment but suffer from detachment and derailment ( Ecoffet et al. , 2019 ) , in which the agent gets trapped into one ( long ) corridor and fails to try other choices . Count-based approaches are also short-sighted : the agent often settles in local minima , sometimes oscillating between two states that alternately feature lower visitation counts ( Burda et al. , 2018b ) . Finally , state-diff approaches offer rewards if , for each trajectory , representations of consecutive states differ significantly . While these approaches consider the entire trajectory of the agent rather than a local state , it is asymptotically inconsistent : the intrinsic reward remains positive when the visitation counts approach infinity . As a result , the final policy does not necessarily maximize the cumulative extrinsic reward . In this paper , we propose a novel exploration criterion that combines count-based and state-diff approaches : instead of using the difference of state representations , we use the regulated difference of inverse visitation counts of consecutive states in a trajectory . The inverse visitation count is approximated by Random Network Distillation ( Burda et al. , 2018b ) . Our IR provides two benefits : ( 1 ) This addresses asymptotic inconsistency in the state-diff , since the inverse visitation count vanishes with sufficient explorations . ( 2 ) Our IR is large at the end of a trajectory and at the boundary between the explored and the unexplored regions ( Fig . 1 ) . This motivates the agent to move Beyond the Boundary of the explored regions and step into the unknown , mitigating the short-sighted issue in count-based approaches . Following this simple criterion , we propose a novel algorithm BeBold and evaluate it on two very challenging procedurally-generated ( PG ) environments : MiniGrid ( Chevalier-Boisvert et al. , 2018 ) and NetHack ( Küttler et al. , 2020 ) . MiniGrid is a popular benchmark for evaluating exploration algorithms ( Raileanu and Rocktäschel , 2020 ; Campero et al. , 2020 ; Goyal et al. , 2019 ) and NetHack is a much more realistic environment with complex goals and skills . BeBold manages to solve the 12 most challenging environments in MiniGrid within 120M environment steps , without curriculum learning . In contrast , ( Campero et al. , 2020 ) solves 50 % of the tasks , which were categorized as “ easy ” and “ medium ” , by training a separate goal-generating teacher network in 500M steps . In NetHack , a more challenging procedurally-generated environment , BeBold also outperforms all baselines with a significant margin on various tasks . In addition , we analyze BeBold extensively in MiniGrid . The quantitative results show that BeBold largely mitigates the detachment problem , with a much simpler design than Go-Explore ( Ecoffet et al. , 2020 ) which contains multiple handtune stages and hyper-parameters . Most Related Works . RIDE ( Raileanu and Rocktäschel , 2020 ) also combines multiple criteria together . RIDE learns the state representation with curiosity-driven approaches , and then uses the difference of learned representation along a trajectory as the reward , weighted by pseudo counts of the state . However , as a two-stage approach , RIDE heavily relies on the quality of generalization of the learned representation on novel states . As a result , BeBold shows substantially better performance in the same procedurally-generated environments . Go-Explore ( Ecoffet et al. , 2020 ) stores many visited states ( including boundaries ) , reaches these states without exploration , and explores from them . BeBold focuses on boundaries , perform exploration without human-designed cell representation ( e.g. , image downsampling ) and is end-to-end . Frontier-based exploration ( Yamauchi , 1997 ; 1998 ; Topiwala et al. , 2018 ) is used to help specific robots explore the map by maximizing the information gain . The “ frontier ” is defined as the 2D spatial regions out of the explored parts . No automatic policy optimization with deep models is used . In contrast , BeBold can be applied to more general partial observable MDPs with deep policies . 2 BACKGROUND . Following single agent Markov Decision Process ( MDP ) , we define a state space S , an action space A , and a ( non-deterministic ) transition function T : S × A → P ( S ) where P ( S ) is the probability of next state given the current state and action . The goal is to maximize the expected reward R = E [ ∑T k=0 γ krt+k=1 ] where rt is the reward , γ is the discount factor , and the expectation is taken w.r.t . the policy π and MDP transition P ( S ) . In this paper , the total reward received at time step t is given by rt = ret +αr i t , where r e t is the extrinsic reward given by the environment , r i t is the intrinsic reward from the exploration criterion , and α is a scaling hyperparameter . 3 EXPLORATION BEYOND THE BOUNDARY . Our new exploration criterion combines both counting-based and state-diff-based criteria . Our criterion doesn ’ t suffer from short-sightedness and is asymptomatically consistent . We ’ ll first introduce BeBold and then analyse the advantages of BeBold over existing criteria in Sec . 4 . Exploration Beyond the Boundary . BeBold gives intrinsic reward ( IR ) to the agent when it explores beyond the boundary of explored regions , i.e. , along a trajectory , the previous state st has been sufficiently explored but st+1 is new : ri ( st , at , st+1 ) = max ( 1 N ( st+1 ) − 1 N ( st ) , 0 ) , ( 1 ) Here N is the visitation counts . We clip the IR here because we don ’ t want to give a negative IR to the agent if it transits back from a novel state to a familiar state . From the equation , only crossing the frontier matters to the intrinsic reward ; if both N ( st ) and N ( st+1 ) are high or low , their difference would be small . As we will show in Sec . 4 , for each trajectory going towards the frontier/boundary , BeBold assigns an approximately equal IR , regardless of their length . As a result , the agent will continue pushing the frontier of exploration in a much more uniform manner than RND and won ’ t suffer from short-sightedness . This motivates the agent to explore different trajectories uniformly . Also Eq . 1 is asymptotically consistent as ri → 0 when N →∞ . Like RIDE ( Raileanu and Rocktäschel , 2020 ) , in our implementation , partial observation ot are used instead of the real state st , when st is not available . Episodic Restriction on Intrinsic Reward ( ERIR ) . In many environments where the state transition is reversible , simply using intrinsic reward to guide exploration would result in the agent going back and forth between novel states st+1 and their previous states st. RIDE ( Raileanu and Rocktäschel , 2020 ) avoids this by scaling the intrinsic reward r ( s ) by the inverse of the state visitation counts . BeBold puts a more aggressive restriction : the agent is only rewarded when it visits the state s for the first time in an episode . Thus , the intrinsic reward of BeBold becomes : ri ( st , at , st+1 ) = max ( 1 N ( st+1 ) − 1 N ( st ) , 0 ) ∗ 1 { Ne ( st+1 ) = 1 } ( 2 ) Ne here stands for episodic state count and is reset every episode . In contrast , the visitation count N is a life-long memory bank counting state visitation across all of training . Inverse visitation counts as prediction difference . We use the difference between a teacher φ and a student network φ′ to approximate visitation counts : N ( st+1 ) ≈ 1||φ ( ot+1 ) −φ′ ( ot+1 ) ||2 , here ot+1 is the observation of the agent in state st+1 . This yields the following implementation of BeBold : ri ( st , at , st+1 ) =max ( ||φ ( ot+1 ) − φ′ ( ot+1 ) ||2 − ||φ ( ot ) − φ′ ( ot ) ||2 , 0 ) ∗ 1 { Ne ( ot+1 ) = 1 } ) ( 3 ) Shared visitation counts N ( st ) in the training of Procedurally-Generated ( PG ) Environments . During training , the environment changes constantly ( e.g. , blue keys becomes red ) , while the semantic links of these objects remain the same . We use a shared RND ( φ , φ′ ) across different PG environments , and treat these semantically similar states as new without using domain knowledge ( e.g. , image downsampling like in Go-Explore ( Ecoffet et al. , 2019 ) ) . Partial observability and generalization of neural network φ handles these differences and leads to count-sharing . For episodic count Ne ( ot+1 ) , since it is not shared across episodes ( and environments ) , we use a hash table . 4 CONCEPTUAL ADVANTAGES OF BEBOLD OVER EXISTING CRITERIA . Short-sightedness and Detachment . One issue in the count-based approach is its short-sightedness . Let ’ s assume in a simple environment , there are M corridors { τj } Mj=1 starting at s0 and extending to different parts of the environment . The corridor τj has a length of Tj . The agent starts at s0 . For each visited state , the agent receives the reward of 1N ( s ) where N ( · ) is the visitation count , and learns with Q-learning . Then with some calculation ( See Appendix ) , we see that the agent has a strong preference on exploring the longest corridor first ( say τ1 ) , and only after a long period does it start to explore the second longest . This is because the agent initially receives high IR in τ1 due to its length , which makes the policy π visit τ1 more often , until it depletes the IR in τ1 . This behavior of “ dedication ” could lead to serious issues . If M ≥ 3 and 2 corridors are long enough ( say τ1 and τ2 are long ) , then before the agent is able to explore other corridors , its policy π has already been trained long enough so that it only remembers how to get into τ1 and τ2 . When τ1 has depleted its IR , the agent goes to τ2 following the policy . After that , the IR in τ1 revives since the visitation counts in τ1 is now comparable or even smaller than τ2 , which lures the agent to explore τ1 again following the policy . This leaves other corridors ( e.g. , τ3 ) unexplored for a very long time . Note that using a neural-network-approximated IR ( RND ) instead of tabular IR could potentially alleviate this issue , but it is often far less than enough in complex environments . As mentioned in Go-Explore series ( Ecoffet et al. , 2019 ; 2020 ) , count-based approaches also suffer from detachment : if the agent by chance starts exploring τ2 after briefly exploring the first few states of τ1 , it would not return and explore τ1 further since τ1 is now “ shorter ” than τ2 and has lower IR than τ2 for a long period . Go-Explore tries to resolve this dilemma between “ dedication ” and “ exploration ” by using a two-stage approach with many hand-tuned parameters . In contrast , IR of BeBold depends on the difference of the visitation counts along the trajectory , and is insensitive to the length of the corridor . This leads to simultaneous exploration of multiple corridors and yields a diverse policy π ( See Sec . 5.2 for empirical evidence ) . Moreover , the IR focuses on the boundary between explored and unexplored regions , where the two goals ( dedication and exploration ) align , yielding a much cleaner , one-stage method . Asymptotic Inconsistency . Approaches that define IR as the difference between state representations ‖ψ ( s ) − ψ ( s′ ) ‖ ( ψ is a learned embedding network ) ( Zhang et al. , 2019 ; Marino et al. , 2019 ) suffer from asymptotic inconsistency . In other words , their IR does not vanish even after sufficient exploration : ri 6→ 0 whenN →∞ . This is because when the embedding network ψ converges after sufficient exploration , the agent can always obtain non-zero IR if a major change in state representation occurs ( e.g. , opening a door or picking up a key in MiniGrid ) . Therefore , the learned policy does not maximize the extrinsic reward re , deviating from the goal of RL . Automatic curriculum approaches ( Campero et al. , 2020 ) ) have similar issues due to an ever-present IR . For this , ( Zhang et al. , 2019 ) proposes to learn a separate scheduler to switch between intrinsic and extrinsic rewards , and ( Raileanu and Rocktäschel , 2020 ) divides the state representation difference by the square root of visitation counts . In comparison , BeBold does not require any extra stage and is a much simpler solution .

Summary: This paper is a presentation of BeBold, a new method using an intrinsic reward for exploration, meant for procedurally generated, episodic environments. The method includes two major components: the first being intrinsically rewarding the agent for entering states that are less visited than the current state and the second being only intrinsically rewarding the agent for the first time it enters a state during an episode. The agent's intrinsic reward is larger if the difference in visit counts is larger. The paper includes some discussion of the conceptual advantages of BeBold over prior work as well as empirical demonstrations in MiniGrid and NetHack.