Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

1 INTRODUCTION . Deep reinforcement learning ( RL ) is widely applied in various domains ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Mnih et al. , 2016 ; Lillicrap et al. , 2015 ; Andrychowicz et al. , 2017 ; Zha et al. , 2019a ; Liu et al. , 2020 ) . However , RL algorithms often require tremendous number of samples to achieve reasonable performance ( Hessel et al. , 2018 ) . This sample efficiency issue becomes more pronounced in sparse reward environments , where the agent may take an extremely long time before bumping into a reward signal ( Riedmiller et al. , 2018 ) . Thus , how to efficiently explore the environment under sparse reward remains an open challenge ( Osband et al. , 2019 ) . To address the above challenge , many exploration methods have been investigated and demonstrated to be effective on hard-exploration environments . One of the most popular techniques is to use intrinsic rewards to encourage exploration ( Schmidhuber , 1991 ; Oudeyer & Kaplan , 2009 ; Guo et al. , 2016 ; Zheng et al. , 2018 ; Du et al. , 2019 ) . The key idea is to give intrinsic bonus based on uncertainty , e.g. , assigning higher rewards to novel states ( Ostrovski et al. , 2017 ) , or using the prediction error of a dynamic model as the intrinsic reward ( Pathak et al. , 2017 ) . While many intrinsic reward methods have demonstrated superior performance on hard-exploration environments , such as Montezuma ’ s Revenge ( Burda et al. , 2018b ) and Pitfall ! ( Badia et al. , 2019 ) , most of the previous studies use the same singleton environment for training and testing , i.e. , the agent aims to solve the same environment in each episode . However , recent studies show that the agent trained in this way is susceptible to overfitting and may fail to generalize to even a slightly different environment ( Rajeswaran et al. , 2017 ; Zhang et al. , 2018a ) . To deal with this issue , a few procedually-generated environments are designed to test the generalization of RL , such as ( Beattie et al. , 2016 ; Chevalier-Boisvert et al. , 2018 ; Nichol et al. , 2018 ; Côté et al. , 2018 ; Cobbe et al. , 2019 ; Küttler et al. , 2020 ) , in which the agent aims to solve the same task , but a different environment is generated in each episode . Unfortunately , encouraging exploration in procedually-generated environments is a very challenging task . Many intrinsic reward methods , such as count-based exploration and curiosity-driven exploration , often fall short in procedually-generated environments ( Raileanu & Rocktäschel , 2020 ; Campero et al. , 2020 ) . Figure 1a shows a motivating example of why count-based exploration is less effective in procedually-generated environments . We consider a 1D grid-world , where the agent ( red ) needs to explore the environment to reach the goal within 7 timesteps , that is , the agent needs to move right in all the steps . While count-based exploration assigns reasonable intrinsic rewards in singleton setting , it may generate misleading intrinsic rewards in procedually-generated setting because visiting a novel state does not necessarily mean a good exploration behavior . This issue becomes more pronounced in more challenging procedually-generated environments , where the agent is not likely to visit the same state more than once . To tackle the above challenge , this work studies how to reward good exploration behaviors that can generalize to different environments . When a human judges how well an agent explores the environment , she often views the agent from the episode-level instead of the state-level . For instance , one can easily tell whether an agent has explored a maze well by looking into the coverage rate of the current episode , even if the current environment is different from the previous ones . In Figure 1b , we can confidently tell that the episode in the right-hand-side is a good exploration behavior because the agent achieves 0.875 coverage rate . Similarly , we can safely conclude that the episode in the left-hand-side does not explore the environment well due to its low coverage rate . Motivated by this , we hypothesize that episode-level exploration score could be a more general criterion to distinguish good exploration behaviors than state-level intrinsic rewards in procedually-generated setting . To verify our hypothesis , we propose exploration via Ranking the Episodes ( RAPID ) . Specifically , we identify good exploration behaviors by assigning episodic exploration scores to past episodes . To efficiently learn the past good exploration behaviors , we use a ranking buffer to store the highlyscored episodes . The agent then learns to reproduce the past good exploration behaviors with imitation learning . We demonstrate that RAPID significantly outperforms the state-of-the-art intrinsic reward methods on several procedually-generated benchmarks . Moreover , we present extensive ablations and analysis for RAPID , showing that RAPID can well explore the environment even without extrinsic rewards and could be generally applied in tasks with continuous state/action spaces . 2 EXPLORATION VIA RANKING THE EPISODES . An overview of RAPID is shown in Figure 2 . The key idea is to assign episodic exploration scores to past episodes and store those highly-scored episodes into a small buffer . The agent then treats the episodes in the buffer as demonstrations and learns to reproduce these episodes with imitation learning . RAPID encourages the agent to explore the environment by reproducing the past good exploration behaviors . This section introduces how we define the episodic exploration score for procedurally-generated environments and how the proposed method can be combined with state-ofthe-art reinforcement learning agents . 2.1 EPISODIC EXPLORATION SCORE . Each episode is scored from both local and global perspectives . The local score is a per-episode view of the exploration behavior , which measures how well the current episode itself explores the environment . The global score provides a long-term and historical view of exploration , i.e. , whether the current episode has explored the regions that are not well explored in the past . Additionally , we consider extrinsic reward , which is the episodic reward received from the environment . It is an external criterion of exploration since a high extrinsic reward often suggests good exploration in sparse environments . The overall episodic exploration score is obtained by the weighted sum of the above three scores . We expect that these scores can model different aspects of exploration behaviors and they are complementary in procedurally-generated environments . Local Score . The intuition of the local exploration bonus is that the agent is expected to visit as many distinct states as possible in one episode . In our preliminary experiments on MiniGrid , we observe that many poor exploration episodes repeatedly visit the same state . As a result , the agent may get stuck in a well-known region but never reach out to unexplored regions . From a local view , we can quantify this exploration behavior based on the diversity of the visited states . Specifically , we define the local exploration score as Slocal = Ndistinct Ntotal , ( 1 ) where Ntotal is the total number of states in the episode , and Ndistinct is the number of distinct states in the episode . Intuitively , optimizing the local score will encourage the agent to reach out to the unexplored regions in the current episode . In an ideal case , the episodes that never visit the same state twice will receive a local bonus of 1 . There could be various ways to extend the above definition to continuous state space . In this work , we empirically define the local score in continuous state space as the mean standard deviation of all the states in the episode : Slocal = ∑l i=1 std ( si ) l , ( 2 ) where l is the dimension of the state , and std ( si ) is the standard deviation along the i-dimension of the states in the episode . Intuitively , this score encourages the episodes that visit diverse states . Global Score . The global score is designed to model long-term exploration behavior . While a different environment will be generated in each episode in the procedurally-generated setting , the environments in different episodes usually share some common characteristics . For example , in the Algorithm 1 Exploration via Ranking the Episodes ( RAPID ) 1 : Input : Training steps S , buffer size D , RL rollout steps T 2 : Initialize the policy πθ , replay buffer D 3 : for iteration = 1 , 2 , ... until convergence do 4 : Execute πθ for T timesteps 5 : Update πθ with RL objective ( also update value functions if any ) 6 : for each generated episode τ do 7 : Compute episodic exploration score Sτ based on Eq . ( 4 ) 8 : Give score Sτ to all the state-action pairs in τ and store them to the buffer 9 : Rank the state-action pairs in D based on their exploration scores 10 : if D.length > D then 11 : Discard the state-action pairs with low scores so that D.length = D 12 : end if 13 : for step = 1 , 2 , .. , S do 14 : Sample a batch from D and train πθ using the data in D with behavior cloning 15 : end for 16 : end for 17 : end for MiniGrid MultiRoom environment ( Figure 3a ) , although a different maze will be generated in a new episode , the basic building blocks ( e.g. , walls , doors , etc . ) , the number of rooms , and the size of each room will be similar . From a global view , the agent should explore the regions that are not well explored in the past . Motivated by count-based exploration , we define the global score as Sglobal = 1 Ntotal ∑ s 1√ N ( s ) , ( 3 ) where N ( s ) is the state count of s throughout the training process , that is , the global score is the mean count-based intrinsic reward of all the states in the episode . Episodic Exploration Score . Suppose Sext is the total extrinsic reward of an episode received from the environment . The episodic exploration score is obtained by the weighted sum of the extrinsic reward Sext , the local score Slocal , and the global score Sglobal : S = w0Sext + w1Slocal + w2Sglobal , ( 4 ) where w0 , w1 and w2 are hyperparameters . We speculate that the three scores are complementary and the optimal w0 , w1 and w2 could vary in different environments . For example , a higher w1 will give a higher weight to the per-episode view of exploration behavior . A higher w2 will focus more on whether the current episode has explored the past unexplored regions . Nevertheless , we find that a single set of w0 , w1 and w2 works well across many tasks . Comparison with Prior Work . Several papers have studied episode-level feedback to improve the sample efficiency of RL . To the best of our knowledge , the existing work mainly focuses on learning a meta-policy based on episodic rewards ( Xu et al. , 2018 ; Zha et al. , 2019b ) , or imitating the episodes with high episodic rewards ( Guo et al. , 2018 ; Srivastava et al. , 2019 ) . However , in very sparse environments , it is not likely to receive a non-zero reward without exploration . These methods are not suitable for very sparse environments since episodic rewards could be always zero . In this work , we propose local and global episodic scores to quantify the exploration behaviors and encourage the agent to visit distinct states from per-episode and long-term views .

Summary: This paper presents an exploration method for procedurally-generated environments, RAPID, which imitates the past episodes that have a good exploration behavior. First, authors introduce exploration scores, local score for per-episode view of the exploration behavior, and global score for long-term and historical view of exploration. The authors use the weighted sum of these two exploration scores and extrinsic reward as a final episodic exploration score. They rank the state-action pairs based on episodic exploration score and train the agent to imitate behaviors with high score. In experiments, they show the results by comparing state-of-the-art algorithms in several procedurally-generated environments.