Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

1 INTRODUCTION . Deep reinforcement learning ( RL ) is widely applied in various domains ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Mnih et al. , 2016 ; Lillicrap et al. , 2015 ; Andrychowicz et al. , 2017 ; Zha et al. , 2019a ; Liu et al. , 2020 ) . However , RL algorithms often require tremendous number of samples to achieve reasonable performance ( Hessel et al. , 2018 ) . This sample efficiency issue becomes more pronounced in sparse reward environments , where the agent may take an extremely long time before bumping into a reward signal ( Riedmiller et al. , 2018 ) . Thus , how to efficiently explore the environment under sparse reward remains an open challenge ( Osband et al. , 2019 ) . To address the above challenge , many exploration methods have been investigated and demonstrated to be effective on hard-exploration environments . One of the most popular techniques is to use intrinsic rewards to encourage exploration ( Schmidhuber , 1991 ; Oudeyer & Kaplan , 2009 ; Guo et al. , 2016 ; Zheng et al. , 2018 ; Du et al. , 2019 ) . The key idea is to give intrinsic bonus based on uncertainty , e.g. , assigning higher rewards to novel states ( Ostrovski et al. , 2017 ) , or using the prediction error of a dynamic model as the intrinsic reward ( Pathak et al. , 2017 ) . While many intrinsic reward methods have demonstrated superior performance on hard-exploration environments , such as Montezuma ’ s Revenge ( Burda et al. , 2018b ) and Pitfall ! ( Badia et al. , 2019 ) , most of the previous studies use the same singleton environment for training and testing , i.e. , the agent aims to solve the same environment in each episode . However , recent studies show that the agent trained in this way is susceptible to overfitting and may fail to generalize to even a slightly different environment ( Rajeswaran et al. , 2017 ; Zhang et al. , 2018a ) . To deal with this issue , a few procedually-generated environments are designed to test the generalization of RL , such as ( Beattie et al. , 2016 ; Chevalier-Boisvert et al. , 2018 ; Nichol et al. , 2018 ; Côté et al. , 2018 ; Cobbe et al. , 2019 ; Küttler et al. , 2020 ) , in which the agent aims to solve the same task , but a different environment is generated in each episode . Unfortunately , encouraging exploration in procedually-generated environments is a very challenging task . Many intrinsic reward methods , such as count-based exploration and curiosity-driven exploration , often fall short in procedually-generated environments ( Raileanu & Rocktäschel , 2020 ; Campero et al. , 2020 ) . Figure 1a shows a motivating example of why count-based exploration is less effective in procedually-generated environments . We consider a 1D grid-world , where the agent ( red ) needs to explore the environment to reach the goal within 7 timesteps , that is , the agent needs to move right in all the steps . While count-based exploration assigns reasonable intrinsic rewards in singleton setting , it may generate misleading intrinsic rewards in procedually-generated setting because visiting a novel state does not necessarily mean a good exploration behavior . This issue becomes more pronounced in more challenging procedually-generated environments , where the agent is not likely to visit the same state more than once . To tackle the above challenge , this work studies how to reward good exploration behaviors that can generalize to different environments . When a human judges how well an agent explores the environment , she often views the agent from the episode-level instead of the state-level . For instance , one can easily tell whether an agent has explored a maze well by looking into the coverage rate of the current episode , even if the current environment is different from the previous ones . In Figure 1b , we can confidently tell that the episode in the right-hand-side is a good exploration behavior because the agent achieves 0.875 coverage rate . Similarly , we can safely conclude that the episode in the left-hand-side does not explore the environment well due to its low coverage rate . Motivated by this , we hypothesize that episode-level exploration score could be a more general criterion to distinguish good exploration behaviors than state-level intrinsic rewards in procedually-generated setting . To verify our hypothesis , we propose exploration via Ranking the Episodes ( RAPID ) . Specifically , we identify good exploration behaviors by assigning episodic exploration scores to past episodes . To efficiently learn the past good exploration behaviors , we use a ranking buffer to store the highlyscored episodes . The agent then learns to reproduce the past good exploration behaviors with imitation learning . We demonstrate that RAPID significantly outperforms the state-of-the-art intrinsic reward methods on several procedually-generated benchmarks . Moreover , we present extensive ablations and analysis for RAPID , showing that RAPID can well explore the environment even without extrinsic rewards and could be generally applied in tasks with continuous state/action spaces . 2 EXPLORATION VIA RANKING THE EPISODES . An overview of RAPID is shown in Figure 2 . The key idea is to assign episodic exploration scores to past episodes and store those highly-scored episodes into a small buffer . The agent then treats the episodes in the buffer as demonstrations and learns to reproduce these episodes with imitation learning . RAPID encourages the agent to explore the environment by reproducing the past good exploration behaviors . This section introduces how we define the episodic exploration score for procedurally-generated environments and how the proposed method can be combined with state-ofthe-art reinforcement learning agents . 2.1 EPISODIC EXPLORATION SCORE . Each episode is scored from both local and global perspectives . The local score is a per-episode view of the exploration behavior , which measures how well the current episode itself explores the environment . The global score provides a long-term and historical view of exploration , i.e. , whether the current episode has explored the regions that are not well explored in the past . Additionally , we consider extrinsic reward , which is the episodic reward received from the environment . It is an external criterion of exploration since a high extrinsic reward often suggests good exploration in sparse environments . The overall episodic exploration score is obtained by the weighted sum of the above three scores . We expect that these scores can model different aspects of exploration behaviors and they are complementary in procedurally-generated environments . Local Score . The intuition of the local exploration bonus is that the agent is expected to visit as many distinct states as possible in one episode . In our preliminary experiments on MiniGrid , we observe that many poor exploration episodes repeatedly visit the same state . As a result , the agent may get stuck in a well-known region but never reach out to unexplored regions . From a local view , we can quantify this exploration behavior based on the diversity of the visited states . Specifically , we define the local exploration score as Slocal = Ndistinct Ntotal , ( 1 ) where Ntotal is the total number of states in the episode , and Ndistinct is the number of distinct states in the episode . Intuitively , optimizing the local score will encourage the agent to reach out to the unexplored regions in the current episode . In an ideal case , the episodes that never visit the same state twice will receive a local bonus of 1 . There could be various ways to extend the above definition to continuous state space . In this work , we empirically define the local score in continuous state space as the mean standard deviation of all the states in the episode : Slocal = ∑l i=1 std ( si ) l , ( 2 ) where l is the dimension of the state , and std ( si ) is the standard deviation along the i-dimension of the states in the episode . Intuitively , this score encourages the episodes that visit diverse states . Global Score . The global score is designed to model long-term exploration behavior . While a different environment will be generated in each episode in the procedurally-generated setting , the environments in different episodes usually share some common characteristics . For example , in the Algorithm 1 Exploration via Ranking the Episodes ( RAPID ) 1 : Input : Training steps S , buffer size D , RL rollout steps T 2 : Initialize the policy πθ , replay buffer D 3 : for iteration = 1 , 2 , ... until convergence do 4 : Execute πθ for T timesteps 5 : Update πθ with RL objective ( also update value functions if any ) 6 : for each generated episode τ do 7 : Compute episodic exploration score Sτ based on Eq . ( 4 ) 8 : Give score Sτ to all the state-action pairs in τ and store them to the buffer 9 : Rank the state-action pairs in D based on their exploration scores 10 : if D.length > D then 11 : Discard the state-action pairs with low scores so that D.length = D 12 : end if 13 : for step = 1 , 2 , .. , S do 14 : Sample a batch from D and train πθ using the data in D with behavior cloning 15 : end for 16 : end for 17 : end for MiniGrid MultiRoom environment ( Figure 3a ) , although a different maze will be generated in a new episode , the basic building blocks ( e.g. , walls , doors , etc . ) , the number of rooms , and the size of each room will be similar . From a global view , the agent should explore the regions that are not well explored in the past . Motivated by count-based exploration , we define the global score as Sglobal = 1 Ntotal ∑ s 1√ N ( s ) , ( 3 ) where N ( s ) is the state count of s throughout the training process , that is , the global score is the mean count-based intrinsic reward of all the states in the episode . Episodic Exploration Score . Suppose Sext is the total extrinsic reward of an episode received from the environment . The episodic exploration score is obtained by the weighted sum of the extrinsic reward Sext , the local score Slocal , and the global score Sglobal : S = w0Sext + w1Slocal + w2Sglobal , ( 4 ) where w0 , w1 and w2 are hyperparameters . We speculate that the three scores are complementary and the optimal w0 , w1 and w2 could vary in different environments . For example , a higher w1 will give a higher weight to the per-episode view of exploration behavior . A higher w2 will focus more on whether the current episode has explored the past unexplored regions . Nevertheless , we find that a single set of w0 , w1 and w2 works well across many tasks . Comparison with Prior Work . Several papers have studied episode-level feedback to improve the sample efficiency of RL . To the best of our knowledge , the existing work mainly focuses on learning a meta-policy based on episodic rewards ( Xu et al. , 2018 ; Zha et al. , 2019b ) , or imitating the episodes with high episodic rewards ( Guo et al. , 2018 ; Srivastava et al. , 2019 ) . However , in very sparse environments , it is not likely to receive a non-zero reward without exploration . These methods are not suitable for very sparse environments since episodic rewards could be always zero . In this work , we propose local and global episodic scores to quantify the exploration behaviors and encourage the agent to visit distinct states from per-episode and long-term views .

Summary: This paper tackles the problem of improving exploration in deep RL for procedurally-generated environments, where state-of-the-art exploration techniques typically fail. In the proposed approach, called RAPID, each agent-generated episode is evaluated with respect to its local exploration score (for the given episode), global exploration score (across all previous episodes), and extrinsic reward obtained. Episodes with high scores are stored in a replay buffer, and a policy is trained via behavioral cloning on batches of state-action pairs from this buffer. This policy is also used to produce the agent-generated episodes.