BeBold: Exploration Beyond the Boundary of Explored Regions

1 INTRODUCTION . Deep reinforcement learning ( RL ) has experienced significant progress over the last several years , with impressive performance in games like Atari ( Mnih et al. , 2015 ; Badia et al. , 2020a ) , StarCraft ( Vinyals et al. , 2019 ) and Chess ( Silver et al. , 2016 ; 2017 ; 2018 ) . However , most work requires either a manually-designed dense reward ( Brockman et al. , 2016 ) or a perfect environment model ( Silver et al. , 2017 ; Moravčı́k et al. , 2017 ) . This is impractical for real-world settings , where the reward is sparse ; in fact , the proper reward function for a task is often even unknown due to lack of domain knowledge . Random exploration ( e.g. , -greedy ) in these environments is often insufficient and leads to poor performance ( Bellemare et al. , 2016 ) . Recent approaches have proposed to use intrinsic rewards ( IR ) ( Schmidhuber , 2010 ) to motivate agents for exploration before any extrinsic rewards are obtained . Various criteria have been proposed , including curiosity/surprise-driven ( Pathak et al. , 2017 ) , count-based ( Bellemare et al. , 2016 ; Burda et al. , 2018a ; b ; Ostrovski et al. , 2017 ; Badia et al. , 2020b ) , and state-diff approaches ( Zhang et al. , 2019 ; Marino et al. , 2019 ) . Each approach has its upsides and downsides : Curiosity-driven approaches look for prediction errors in the learned dynamics model and may be misled by the noisy TV ( Burda et al. , 2018b ) problem , where environment dynamics are inherently stochastic . Count-based approaches favor novel states in the environment but suffer from detachment and derailment ( Ecoffet et al. , 2019 ) , in which the agent gets trapped into one ( long ) corridor and fails to try other choices . Count-based approaches are also short-sighted : the agent often settles in local minima , sometimes oscillating between two states that alternately feature lower visitation counts ( Burda et al. , 2018b ) . Finally , state-diff approaches offer rewards if , for each trajectory , representations of consecutive states differ significantly . While these approaches consider the entire trajectory of the agent rather than a local state , it is asymptotically inconsistent : the intrinsic reward remains positive when the visitation counts approach infinity . As a result , the final policy does not necessarily maximize the cumulative extrinsic reward . In this paper , we propose a novel exploration criterion that combines count-based and state-diff approaches : instead of using the difference of state representations , we use the regulated difference of inverse visitation counts of consecutive states in a trajectory . The inverse visitation count is approximated by Random Network Distillation ( Burda et al. , 2018b ) . Our IR provides two benefits : ( 1 ) This addresses asymptotic inconsistency in the state-diff , since the inverse visitation count vanishes with sufficient explorations . ( 2 ) Our IR is large at the end of a trajectory and at the boundary between the explored and the unexplored regions ( Fig . 1 ) . This motivates the agent to move Beyond the Boundary of the explored regions and step into the unknown , mitigating the short-sighted issue in count-based approaches . Following this simple criterion , we propose a novel algorithm BeBold and evaluate it on two very challenging procedurally-generated ( PG ) environments : MiniGrid ( Chevalier-Boisvert et al. , 2018 ) and NetHack ( Küttler et al. , 2020 ) . MiniGrid is a popular benchmark for evaluating exploration algorithms ( Raileanu and Rocktäschel , 2020 ; Campero et al. , 2020 ; Goyal et al. , 2019 ) and NetHack is a much more realistic environment with complex goals and skills . BeBold manages to solve the 12 most challenging environments in MiniGrid within 120M environment steps , without curriculum learning . In contrast , ( Campero et al. , 2020 ) solves 50 % of the tasks , which were categorized as “ easy ” and “ medium ” , by training a separate goal-generating teacher network in 500M steps . In NetHack , a more challenging procedurally-generated environment , BeBold also outperforms all baselines with a significant margin on various tasks . In addition , we analyze BeBold extensively in MiniGrid . The quantitative results show that BeBold largely mitigates the detachment problem , with a much simpler design than Go-Explore ( Ecoffet et al. , 2020 ) which contains multiple handtune stages and hyper-parameters . Most Related Works . RIDE ( Raileanu and Rocktäschel , 2020 ) also combines multiple criteria together . RIDE learns the state representation with curiosity-driven approaches , and then uses the difference of learned representation along a trajectory as the reward , weighted by pseudo counts of the state . However , as a two-stage approach , RIDE heavily relies on the quality of generalization of the learned representation on novel states . As a result , BeBold shows substantially better performance in the same procedurally-generated environments . Go-Explore ( Ecoffet et al. , 2020 ) stores many visited states ( including boundaries ) , reaches these states without exploration , and explores from them . BeBold focuses on boundaries , perform exploration without human-designed cell representation ( e.g. , image downsampling ) and is end-to-end . Frontier-based exploration ( Yamauchi , 1997 ; 1998 ; Topiwala et al. , 2018 ) is used to help specific robots explore the map by maximizing the information gain . The “ frontier ” is defined as the 2D spatial regions out of the explored parts . No automatic policy optimization with deep models is used . In contrast , BeBold can be applied to more general partial observable MDPs with deep policies . 2 BACKGROUND . Following single agent Markov Decision Process ( MDP ) , we define a state space S , an action space A , and a ( non-deterministic ) transition function T : S × A → P ( S ) where P ( S ) is the probability of next state given the current state and action . The goal is to maximize the expected reward R = E [ ∑T k=0 γ krt+k=1 ] where rt is the reward , γ is the discount factor , and the expectation is taken w.r.t . the policy π and MDP transition P ( S ) . In this paper , the total reward received at time step t is given by rt = ret +αr i t , where r e t is the extrinsic reward given by the environment , r i t is the intrinsic reward from the exploration criterion , and α is a scaling hyperparameter . 3 EXPLORATION BEYOND THE BOUNDARY . Our new exploration criterion combines both counting-based and state-diff-based criteria . Our criterion doesn ’ t suffer from short-sightedness and is asymptomatically consistent . We ’ ll first introduce BeBold and then analyse the advantages of BeBold over existing criteria in Sec . 4 . Exploration Beyond the Boundary . BeBold gives intrinsic reward ( IR ) to the agent when it explores beyond the boundary of explored regions , i.e. , along a trajectory , the previous state st has been sufficiently explored but st+1 is new : ri ( st , at , st+1 ) = max ( 1 N ( st+1 ) − 1 N ( st ) , 0 ) , ( 1 ) Here N is the visitation counts . We clip the IR here because we don ’ t want to give a negative IR to the agent if it transits back from a novel state to a familiar state . From the equation , only crossing the frontier matters to the intrinsic reward ; if both N ( st ) and N ( st+1 ) are high or low , their difference would be small . As we will show in Sec . 4 , for each trajectory going towards the frontier/boundary , BeBold assigns an approximately equal IR , regardless of their length . As a result , the agent will continue pushing the frontier of exploration in a much more uniform manner than RND and won ’ t suffer from short-sightedness . This motivates the agent to explore different trajectories uniformly . Also Eq . 1 is asymptotically consistent as ri → 0 when N →∞ . Like RIDE ( Raileanu and Rocktäschel , 2020 ) , in our implementation , partial observation ot are used instead of the real state st , when st is not available . Episodic Restriction on Intrinsic Reward ( ERIR ) . In many environments where the state transition is reversible , simply using intrinsic reward to guide exploration would result in the agent going back and forth between novel states st+1 and their previous states st. RIDE ( Raileanu and Rocktäschel , 2020 ) avoids this by scaling the intrinsic reward r ( s ) by the inverse of the state visitation counts . BeBold puts a more aggressive restriction : the agent is only rewarded when it visits the state s for the first time in an episode . Thus , the intrinsic reward of BeBold becomes : ri ( st , at , st+1 ) = max ( 1 N ( st+1 ) − 1 N ( st ) , 0 ) ∗ 1 { Ne ( st+1 ) = 1 } ( 2 ) Ne here stands for episodic state count and is reset every episode . In contrast , the visitation count N is a life-long memory bank counting state visitation across all of training . Inverse visitation counts as prediction difference . We use the difference between a teacher φ and a student network φ′ to approximate visitation counts : N ( st+1 ) ≈ 1||φ ( ot+1 ) −φ′ ( ot+1 ) ||2 , here ot+1 is the observation of the agent in state st+1 . This yields the following implementation of BeBold : ri ( st , at , st+1 ) =max ( ||φ ( ot+1 ) − φ′ ( ot+1 ) ||2 − ||φ ( ot ) − φ′ ( ot ) ||2 , 0 ) ∗ 1 { Ne ( ot+1 ) = 1 } ) ( 3 ) Shared visitation counts N ( st ) in the training of Procedurally-Generated ( PG ) Environments . During training , the environment changes constantly ( e.g. , blue keys becomes red ) , while the semantic links of these objects remain the same . We use a shared RND ( φ , φ′ ) across different PG environments , and treat these semantically similar states as new without using domain knowledge ( e.g. , image downsampling like in Go-Explore ( Ecoffet et al. , 2019 ) ) . Partial observability and generalization of neural network φ handles these differences and leads to count-sharing . For episodic count Ne ( ot+1 ) , since it is not shared across episodes ( and environments ) , we use a hash table . 4 CONCEPTUAL ADVANTAGES OF BEBOLD OVER EXISTING CRITERIA . Short-sightedness and Detachment . One issue in the count-based approach is its short-sightedness . Let ’ s assume in a simple environment , there are M corridors { τj } Mj=1 starting at s0 and extending to different parts of the environment . The corridor τj has a length of Tj . The agent starts at s0 . For each visited state , the agent receives the reward of 1N ( s ) where N ( · ) is the visitation count , and learns with Q-learning . Then with some calculation ( See Appendix ) , we see that the agent has a strong preference on exploring the longest corridor first ( say τ1 ) , and only after a long period does it start to explore the second longest . This is because the agent initially receives high IR in τ1 due to its length , which makes the policy π visit τ1 more often , until it depletes the IR in τ1 . This behavior of “ dedication ” could lead to serious issues . If M ≥ 3 and 2 corridors are long enough ( say τ1 and τ2 are long ) , then before the agent is able to explore other corridors , its policy π has already been trained long enough so that it only remembers how to get into τ1 and τ2 . When τ1 has depleted its IR , the agent goes to τ2 following the policy . After that , the IR in τ1 revives since the visitation counts in τ1 is now comparable or even smaller than τ2 , which lures the agent to explore τ1 again following the policy . This leaves other corridors ( e.g. , τ3 ) unexplored for a very long time . Note that using a neural-network-approximated IR ( RND ) instead of tabular IR could potentially alleviate this issue , but it is often far less than enough in complex environments . As mentioned in Go-Explore series ( Ecoffet et al. , 2019 ; 2020 ) , count-based approaches also suffer from detachment : if the agent by chance starts exploring τ2 after briefly exploring the first few states of τ1 , it would not return and explore τ1 further since τ1 is now “ shorter ” than τ2 and has lower IR than τ2 for a long period . Go-Explore tries to resolve this dilemma between “ dedication ” and “ exploration ” by using a two-stage approach with many hand-tuned parameters . In contrast , IR of BeBold depends on the difference of the visitation counts along the trajectory , and is insensitive to the length of the corridor . This leads to simultaneous exploration of multiple corridors and yields a diverse policy π ( See Sec . 5.2 for empirical evidence ) . Moreover , the IR focuses on the boundary between explored and unexplored regions , where the two goals ( dedication and exploration ) align , yielding a much cleaner , one-stage method . Asymptotic Inconsistency . Approaches that define IR as the difference between state representations ‖ψ ( s ) − ψ ( s′ ) ‖ ( ψ is a learned embedding network ) ( Zhang et al. , 2019 ; Marino et al. , 2019 ) suffer from asymptotic inconsistency . In other words , their IR does not vanish even after sufficient exploration : ri 6→ 0 whenN →∞ . This is because when the embedding network ψ converges after sufficient exploration , the agent can always obtain non-zero IR if a major change in state representation occurs ( e.g. , opening a door or picking up a key in MiniGrid ) . Therefore , the learned policy does not maximize the extrinsic reward re , deviating from the goal of RL . Automatic curriculum approaches ( Campero et al. , 2020 ) ) have similar issues due to an ever-present IR . For this , ( Zhang et al. , 2019 ) proposes to learn a separate scheduler to switch between intrinsic and extrinsic rewards , and ( Raileanu and Rocktäschel , 2020 ) divides the state representation difference by the square root of visitation counts . In comparison , BeBold does not require any extra stage and is a much simpler solution .

Summary: The authors propose a novel intrinsic reward based on the difference of inverse visitation counts for consecutive states. This reward encourages the agent to explore beyond the boundary of already explored regions. Using a few simple examples, they show that the proposed intrinsic reward mitigates the problems of detachment and short-sightedness which are common for count-based methods. The method shows superior performance on a number of tasks from two procedurally-generated benchmarks, MiniGrid and NetHack. The paper also contains comparisons with a few strong baselines (including SOTA on these benchmarks), analysis of the learned behavior and intrinsic reward, as well as ablations of the proposed approach.