Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
Authors: Daochen Zha
Wenye Ma
Lei Yuan
Xia Hu
Ji Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement Learning
Exploration
Generalization of Reinforcement Learning
Self-Imitation

Abstract: Exploration under sparse reward is a long-standing challenge of model-free reinforcement learning. The state-of-the-art methods address this challenge by introducing intrinsic rewards to encourage exploration in novel states or uncertain environment dynamics. Unfortunately, methods based on intrinsic rewards often fall short in procedurally-generated environments, where a different environment is generated in each episode so that the agent is not likely to visit the same state more than once. Motivated by how humans distinguish good exploration behaviors by looking into the entire episode, we introduce RAPID, a simple yet effective episode-level exploration method for procedurally-generated environments. RAPID regards each episode as a whole and gives an episodic exploration score from both per-episode and long-term views. Those highly scored episodes are treated as good exploration behaviors and are stored in a small ranking buffer. The agent then imitates the episodes in the buffer to reproduce the past good exploration behaviors. We demonstrate our method on several procedurally-generated MiniGrid environments, a first-person-view 3D Maze navigation task from MiniWorld, and several sparse MuJoCo tasks. The results show that RAPID significantly outperforms the state-of-the-art intrinsic reward strategies in terms of sample efficiency and final performance. The code is available at https://github.com/daochenzha/rapid

Published as a conference paper at ICLR 2021
RANK THE EPISODES: A SIMPLE APPROACH FOR EXPLORATION IN PROCEDURALLY-GENERAEE ENVIRONMENTS
Daochen Zha', Wenye Ma², Lei Yuan', Xia Hu', , Ji Liu²
'Department of Computer Science and Engineering, Texas A&M University
2 AA Platform, Kwai Inc.
[daochen. zha, xiahu] (arnuutttann @t.mm.................. edu
mmawenye, 1eiyuan03] @kuai shou.. com, jil jiliuekwai.com
ABSTRACT
Exploration under sparse reward is a long-standing challenge of model-free re- inforcement learning. The state-of-the-art methods address this challenge by in- troducing intrinsic rewards to encourage exploration novel states uncertain environment dynamics. Unfortunately, methods based on intrinsic rewards of- ten fall short in procedurally -egerrated environments, where different environ- ment is generated in each episode so that the agent is not likely to visit the same state more than once, Motivated by how humans distinguish good exploration be- haviors by looking into the entire episode, we introduce RAPID, a simple yet environ-
effective cpisode-level exploration method for procedurally-generaae ments. RAPID regards each episode as whole and gives episodic explo- ration score from both per-episode and long-term views. Those highly scored episodes treated good exploration behaviors and are stored in small ranking buffer. The agent then imitates the episodes in the buffer to reproduce the past good exploration behaviors. We demonstrate our method on several procedurally-generne
MiniGrid environments, a first-person-view 3D Maze nav- igation task from Mini World, and several sparse MuJoCo tasks. The results show that RAPID significantly outperforms the state-of-the- intrinsic reward strate- gies in terms of sample efficiency and final performance. The code is available https: g thub. com http///acchehhaaaaecce rapid
INTRODUCTION
Deep reinforcement learning (RL) is widely applied in various domains (Mnih et al. 2013 Silver et al. 2016 Mnih et al. 2016 Lillicrap et al. 2015 Andrychowicz et al. 2017 Zha et al. 2019 Liu
cet al. 2020). However, RL algorithms often require tremendous number of samples to achieve rea- sonable performance (Hessel et al. 2018). This sample efficiency issue becomes more pronounced in sparse reward environments, where the agent may take an extremely long time before bumping into a reward signal (Riedmiller al. 20188) Thus, how to efficiently explore the environment under sparse reward remains an open challenge (Osband et al. 2019))
To address the above challenge, many exploration methods have been investigated and demonstrated to be effective on hard-exploration environments. One of the most popular techniques is to use in- trinsic rewards encourage exploration (Schnidhuuer 1991 Oudeyer & Kaplan 2009 Guo al.) 2016 Zheng et al. 2018 Du et al. 2019). The key idea is to give intrinsic bonus based on uncer- tainty, e-g., assigning higher rewards to novel states (Ostrovski et al. 2017), or using the prediction error of a dynamic model as the intrinsic reward (Pathak et al. (017). While many intrinsic reward methods have demonstrated superior performance on hard-exploraticn environments, such as Mon- tezuma's Revenge (Burda et al. 2018b) and Pitfall! (Badia et al., 2019), most of the previous studies use the same singleton environment for training and testing, i.e., the agent aims to solve the same environment in each episode. However, recent studies show that the agent trained in this way is ceptible to overfitting and may fail to generalize to even a slightly different environment (Rajeswaran et al. 2017 Zhang al. 2018a). To deal with this issue, few procedually: generrated environments

---

Published as a conference paper at ICLR 2021
Intrttso: zewazd
possiibe
Visited
Not visited
Sample episodes
100 episodes
100 episodes
Episodic score - 0.25 Poor exploration
Episodic score - 0.875 Good exploration
Singleton
Procedurally-Gnnente
(a) Count-based exploration
(b) Episodic exploration score
motivating example of count-based exploration versus episode-lveel exploration score. While count-based exploration works well in singleton setting, i.e., the environment is the same in each episode, may be brittle block in an episode is sampled from ten possible values, the third block will have very high intrinsic reward because the agent uncertain on the new states. This will reward the agent for exploring the third block and also its neighbors, and hence the agent may get stuck around the third block. In (b), we score behaviors in episode-level with # visited states/ # total states. The episodic exploration score can effectively distinguish good exploration behaviors in proceduully--eeneaated setting.
Figure I:
pocceuulll--generatee setting. In (a), if the observation of the third
designed to test the generalization of RL, such as (Beattie et al.) 2016 Cheveeier-Boisvert al., 2018 Nichol et al. 2018 Côté et al. 2018 Cobbe et al. 2019 Küttler et al., 2020), in which the agent aims to solve the same task, but different environment is generated in each episode.
Unfortunately, encouraging exploration in poocedually-generatte environments is a very challeng- ing task. Many intrinsic reward methods, such count-based exploration and curiosity- ploration, often fall short in procedually-generater environments (Raileanu & Rocktäschel 2020) Campero al.) 2020). Figure la shows motivating example of why count-based exploration is less effective in procedually-generat (red) needs to explore the environment reach the goal within timesteps, that is, the agent needs to move right in all the steps. While count-based exploration assigns reasonable intrinsic rewards in singleton setting, it may generate misleading intrinsic rewards because visiting novel state does not necessarily mean good exploration behavior. This issue be- comes more pronounced in more challenging procedually- -eeeerrree environments, where the agent is not likely to visit the same state more than once. To tackle the above challenge, this work studies how to reward good exploration behaviors that can generalize to different environments.
environments. We consider a ID grid-world, where the agent
poocedually-generacee setting
When human judges how well agent explores the environment, she often views the agent from the episode-level instead of the state-level. For instance, one can easily tell whether an agent has explored
well by looking into the coverage environment is different from the previous ones. In Figure we can confidently tell that the episode in the night-hand-side is good exploration behavior because the agent achieves 0.875 coverage rate. Similarly, we can safely conclude that the episode in the left-hand-side does not explore the environment well due its low coverage rate. Motivated by this, hypothesize that episode-level exploration score could be more general criterion to distinguish good exploration behaviors than state-level intrinsic rewards in procedually-generatte setting.
of the current episode, even if the current
To verify our hypothesis, we propose exploration via Ranking the Episodes (RAPID). Specifically, we identify good exploration behaviors by assigning episodic exploration scores to past episodes. To efficiently learn the past good exploration behaviors, we use ranking buffer to store the highly- scored episodes. The agent then learns to reproduce the past good exploration behaviors with imi- tation learning. We demonstrate that RAPID significantly outperforms the stae-oofthh--s the-art intrinsic reward methods on several procedually generated benchmarks. Moreover, we present extensive ab- lations and analysis for RAPID, showing that RAPID can well explore the environment even without extrinsic rewards and could be generally applied tasks with continuous state/action spaces.
eereduually-geerrtteed
2 EXPLORATION VIA RANKING THE EPISODES
An overview of RAPID is shown in Figure The key idea is to assign episodic exploration scores to past episodes and store those highly-score scored episodes into small buffer. The agent then treats the episodes in the buffer as demonstrations and learns to reproduce these episodes with imitation learning. RAPID encourages the agent explore the environment by reproducing the past good
2

---

Published as a conference paper at ICLR 2021
Episodes
Local Score
Environment
Global Score
Rank episodic scores
Intereaction
Agent
Imitation Learning
RL Update
Ranking Buffer
Figure 2: An overview of RAPID. The past episodes assigned episodic exploration scores based on the local view, the global view, and the extrinsic reward. Those highly scored episodes are stored in small ranking buffer. The agent is then encouraged to reproduce the past good exploration behaviors, i.e., the episodes in the buffer, with imitation learning.
exploration behaviors. This section introduces how define the episodic exploration score for procedurally--eeeraaee environments and how the proposed method can be combined with state-of- the-art reinforcement learning agents.
2.1 EPISODIC EXPLORATION SCORE
Each episode is scored from both local and global perspectives. The local score is a per-episode view of the exploration behavior, which measures how well the current episode itself explores the environment. The global score provides a long-term and historical view of exploration, i.e., whether the current episode has explored the regions that are not well explored in the past. Additionally, we consider extrinsic reward, which is the episodic reward received from the environment. It is an external criterion of exploration since a high extrinsic reward often suggests good exploration in sparse environments. The overall episodic exploration score is obtained by the weighted sum of the above three scores. We expect that these scores can model different aspects of exploration behaviors and they are complementary in procedurally-genenee
environments.
Local Score. The intuition of the local exploration bonus is that the agent is expected to visit as many distinct states possible in one episode. In preliminary experiments on MiniGrid, we observe that many poor exploration episodes repeatedly visit the same state. As a result, the agent may get stuck in well-known region but never reach out to unexplored regions. From a local view, we can quantify this exploration behavior based on the diversity of the visited states. Specifically, define the local exploration
Naistinet
Speal
Nivtal
where Nosal is the total number of states in the episode, and Naistiner is the number of distinct states in the episode. Intuitively, optimizing the local score will encourage the agent to reach out to the unexplored regions the current episode. In ideal case, the episodes that never visit the same state twice will receive a local bonus of 1. There could be various ways to extend the above definition continuous state space. In this work, we empirically define the local score in continuous state space as the mean standard deviation of all the states in the episode:
Siocal Σl std(s:)
where 1 is the dimension of the state, and std(s;) s:) is the standard deviation along the i-dimension of the states in the episode. Intuitively, this score encourages the episodes that visit diverse states.
Global Score. The global score is designed to model long-term exploration behavior. While different environment will be generated each episode in the sercedurally-generaae setting, the environments in different episodes usually share some common characteristies. For example, in the
3

---

end for Published as a conference paper at ICLR 2021
Algorithm Exploration via Ranking the Episodes (RAPID)
1: Input: Training steps S, buffer size D, RL rollout steps T
2: Initialize the policy we, replay buffer DD
3: for iteration = 1, 2, until convergence do
4: Execute T for T timesteps
5: Update тe with RL objective (also update value functions if any)
6: for each generated episode r do
7:
Compute episodic exploration score S, based Eq. Give score S, to all the state-action pairs in r and store them to the buffer Rank the state-action pairs in DD based on their exploration scores if "Dllenghh > D then
8:
9:
10:
Discard the state-action pairs with low scores so that D.length end if
12:
for step = 1, 2, ... do
14:
Sample batch from D and train using the data in D with behavior cloning
15:
end for
16:
end for
17:
MiniGrid MultiRoom enviromment (FigureBal although a different maze will be generated in a new episode, the basic building blocks (e.g., walls, doors, etc.), the number of rooms, and the size of each room will be similar. From a global view, the agent should explore the regions that are not well explored in the past. Motivated by count-based exploration, we define the global score
Sglobaal
(3)
Niotal
(s)
where (s)) is the state count of s throughout the training process, that is, the global score is the mean count-based intrinsic reward of all the states in the episode.
Episodic Exploration Score. Suppose Sox is the total extrinsic reward of an episode received from the environment. The episodic exploration score is obtained by the weighted sum of the extrinsic reward Sex. the local score Spocal. and the global score Sglobal:
5= Spree + W1 Stoeal + W2 Sglabal,
(4)
where Wo. Wi and We are hyperparameters. We speculate that the three scores are complementary and the optimal wp, and could vary in different environments. For example, higher will give higher weight to the per-episode view of exploration behavior. A higher W2 will focus more on whether the current episode has explored the past unexplored regions. Nevertheless, we find that a single set of Wo, Wi and w2 works well across many tasks.
Comparison with Prior Work. Several papers have studied episode-level feedback to improve the sample efficiency of RL. To the best of our knowledge, the existing work mainly focuses on learning meta-polle meta-policy based episodic rewards (Xu et al. 2018 Zha et al. 2019b), imitating the episodes with high episodic rewards (Guo et al.] 2018 Srivastava et al., 2019). However, in very sparse environments, it is not likely to receive a non-zero reward without exploration. These methods are not suitable for very sparse environments since episodic rewards could be always zero. In this work, we propose local and global episodic scores quantify the exploration behaviors and encourage the agent to visit distinct states from per-episode and long-term views.
2.2 REPRODUCING PAST GOOD EXPLORATION BEHAVIORS WITH IMITATION LEARNING
straightforward idea is to treat the episodic exploration score as reward signal and assign it to the last timestamp of the episode. In this way, the agent can be updated with any reinforcement learning objective. However, this strategy not very effective with three potential limitations. First, the reward remains sparse that the reinforcement learning agents may not be able to effectively exploit the reward signals. Second, the intrinsic exploration bonus may introduce bias in the objective. While the local score and global score may encourage exploration at the beginning stage, they
4

---

Published as a conference paper at ICLR 2021
(a) SaltiRoom-N12-S51 (b) KeyCorridor- -4--3r
Maze
(d) Maze (top view)
Figure 3: Rendering of the poocedulll--genencc environments in our experiments.
true objectives and may degrade the final performance. Third, it only uses a good episode once. In fact, many past good episodes can be exploited many times to improve sample efficiency.
To overcome the above limitations, we propose to use small ranking buffer store past good state-action pairs and then use imitation objective to enhance exploration. Specifically, we assign the same episodic exploration score to all the state-action pairs in an episode and rank all the state- action pairs based on scores. In addition to the original RL objective, we employ behavior cloning, the simplest form of imitation learning, to encourage the agent to reproduce the state-action pairs in the buffer. This simple ranking buffer can effectively exploit the past good experiences since good state-action pair could be sampled more than once. The procedure is summarized Algorithm A possible variant is to keep the entire episodes in the buffer. We have tried this idea and do not observe a clear difference between this variant and Algorithm (see Appendix for more discussions).
3 EXPERIMENTS
The experiments are designed to answer the following research questions: RQ1: how is RAPID compared with the lfate-of-the-art intrinsic reward methods on benchmark environments (Section 3 RQ2: how will RAPID perform if removing one of the scores (i.e., the local score, the global score, and the extrinsic reward) or the buffer (Section 3.3)? RQ3: how will the hyperparameters impact the performance of RAPID (Section 3.3 RQ4: can RAPID explore the environment without extrinsic reward (Section 3.3) RQ5: how will RAPID perform with larger
seeeeduualll-generrte
and more challenging grid-world (Section 3.3 ? RQ6: can RAPID generalize to 3D navigation task
and continuous state/action space, such as MuJoCo (Section 3.4
3.1 ENVIRONMENTS AND BASELINES
Figure 3 shows the rendering of procedurally generated environments used in this work. Fol- lowing the previous studies of exploration in procedurally-generaaee environments (Raileanu & Rocktäschel, 2020 Campero et al. 2020), we mainly focus on MiniGrid environments (Chevalier- Boisvert al. 2018)), which provide sereeduurall-geneertte of hard exploration tasks: MultRoom--NX-SY, where X and Y indicate the number of rooms and the room size, respectively, and KeyCorridor-SX-IY where X and Y indicate the room size and the number of rows, respectively. A large X or Y suggests larger environment, which will be more difficult explore. The agent has partial view of the environment and needs to navigate the green block or the red ball under sparse rewards. These tasks are challenging because the agent needs to explore many rooms order to receive reward signal so that standard RL algorithms will typi- cally fail. To test whether RAPID can generalize to continuous state space, we consider MiniWorld Maze Chevaller-Boisverr 2018). whose observation space 60 x 80 images. Similarly, the agent in MiniWorld Maze has 3D partial view of the environment and needs to navigate the red box. The maze is procedurally -enerrated, i.e., a different maze will be generated in each episode. We further study RAPID in continuous state and action spaces sparse MuJoCo tasks TToorov et al. 2012 Brockman et al. 2016). More details of the above environments are described in Appendx B
goid-worlds. We consider two types
While exploration has been extensively studied in the literature, very few methods designed for errceurrllyyggeneratedd
procedurally generated environments. To better understand the performance of RAPID, we con- sider the baselines in the following categories. Traditional Methods: consider several popular methods designed for singleton environments, including count- based exploration (COUNT) (Belle- mare et al. 2016), Random Network Distillation Exploration (RANDOM) (Burda et al. 2018h).
5

---

Published as a conference paper at ICLR 2021
RAPID
ANDGO
COUNT
0.8
200m
0.4
0.4
0.
0.2
0.0
0.0
1.5
2.0
0.0
1.5
2.0
0.0
3.0 Grnsstop
1.5
2.0
0.0
3.5 Grisstos
1.5
7.0
Dimestogs
Smestops
(a) MlultiRoom-N7-S5
(b) MultiRoom-NN -11-54
(c) MultiRoom-N7-S3
(d) MultiRoom -N1-S00
1.00
0.8
0.75
0.50
0.5
0.2
0.2
0.2
8.00
0.0
0.0
-025
0.0
0.5
1.0 tirendaps
1.5
2.0
9.0
0.5
3.0 ttrendaps
1.5
2.0
0.0
0.5
3.0 trrentape
1.5
2.0
0.0
0.5
3.0
1.5
2.0
107
L07
tiramteps
(c) KeyCorridor-
(f) KeyCorridonSS3-R.
(g) KeyCoridor-SS4-3 (h) MultiRoom-N12-53
Figure 4: Performance of RAPID against baselines on hard-exploration environments in MiniGrid.
All the experiments are run times. The shaded area represents mean + standard deviation.
RAPID
w/o local
w/o global
w/o reward
w/o buffer
w/o ranking
MR-N7S4
0.787 + 0,001
0.787 + 0.000
0.787 + 0.001
0.781 :: 0.002
0,000 1 0.000
0.002 1 0.002
MR-N NR-N
0.778 + 0.000
0.778 1 0.000
0.778 + 0.001
0.775 : 0.002
0.000 1: 0.000
0.000 1 0.000
MR-N758
0.678 : 0.001
0.677 0.002
0.677 0.002
0.652 = 0.004
0.000 + 0.000
0.000 0.000
MR-N10510
0.632 + 0.001
0.238 + 0.288
0.630 = 0.002
0.601 = 0.010
0.000
0.000
0.000 + 0.000
MR-NI2S10
0.644 + 0.001
0.001 0.001
0.633 0.005
0.613 - 0.007
0.000
0.000
0.000 +: 0.000
KC-S3R2
0.934 = 0.004
0.933 1333+00 0.002
0.934 = 0.000
0.929 = 0.003
0.018
- 0.008
0.527 = 0.380
KC-S3R3
0.922 + 0.001
0.885 0.022
0.912 0.003
0.903
= 0.002
0.012
+ 0.007
0.013 0.006
KC3S4R3
0.473 1 0.087
0.150 0.095
0.244 0.144
0.035 1 0.035
0.000 1 0.000
0.001 0.001
Table 1: Performance of RAPID and the ablations on MiniGrid environments. The mean maximum
returns and standard deviations are reported. MR stands for MultiRoom; KC stands for KeyCorridor Learning curves are in Appendix C
and curiosity-driven exploration (CURIOSITY) Pathak et al. 2017). Methods for Procedurally- Generated Environments: consider two of exploration methods, including Impact- Driven Exploration (RIDE) (Raileanu & Rocktäschel 2020). and exploration with Adversarially Motivated Intrinsic Goals (AMIGO) (Campero et al. 2020). Imitatimt Methods: self- self-imitation learning (SIL) (Oh et al., 2018) also exploits past good experiences to encourage exploration. We further include PPO (Schulman al. (017) reinforcement leaming baseline. All the algorithms are run the same number timesteps for a fair comparison. More details are provided in Appendix A
3.2 MINIGRID RESULTS
To study RQ1, we plot the learning curves of RAPID and the baselines on MiniGrid benchmarks in Figure 4 We can make three observations. First, RAPID performs significantly better than the state-of-the-art methods designed for singleton or pnocedurally-generale environments in terms of sample efficiency and final performance. For example, on MultiRoom-N7-SS, RAPID is at least 10 times more sample efficient than RIDE, On KeyCoridor-S4-R3, RAPID the only algorithm that successfully navigates the goal within 20 million timesteps. Second, the traditional intrinsic reward methods perform well on easy procedurally generrted environments but perform poorly on hard- exploration tasks. For example, COUNT performs well as RAPID on KeyCorridor-S3-R? but fails on more challenging tasks, such as MultiRoom-N7-S This suggests that traditional methods may fall short in challenging procedurally-geenraaed environments, which is consistent with the results in previous work (Raileanu & Rocktäschel] 2020). Third, RAPID outperforms SIL across all the tasks. Recall that both RAPID and SIL learn to reproduce past good experiences. While SIL reproduces experiences with high rewards, RAPID additionally considers local and global exploration scores. This verifies the effectiveness of the episodic exploration scores.

---

Published as a conference paper at ICLR 2021
MRQ
MRR
0.4
06
033
02
04
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.
Emwccaps
107
timeddppe
trmeddape
(a) Impact of training steps (b) Imapet of buffer size
(c) Pure exploration
(d) Local exploration score
Figure 5: Analysis of RAPID on MutiiRoomNN12-510. Results for other environments are in Ap-
pendix D (b): the impact of the hyperparameters. (c) the returns without extrinsic rewards
(pure exploration) and the corresponding local exploration scores.
0.00
AGL
3.5
0.4
3.4
0.65
0.50
3.
0.2
3.0
15 25
10
dreestes
Figure 6: Left: the maximum returns w.I.t. the number of rooms under room size 4 (full curves Middle: the maximum returns w.r.t the room sizes with 4 rooms (full curves in Appendix F. Right: learning curves on Mini WolrW Maze (more results in Appendix G
in Appendix E..
3.3 ABLATIONS AND ANALYSIS
To study RQ2, we perform ablation studies on RAPID. Specifically, we remove one of the local score, the global score, and the extrinsic reward, and compare each ablation with RAPID. Besides, we consider a variant that directly uses the episodic exploration score as the reward signal without buffer. To demonstrate the effectiveness of the ranking, further include a baseline that uses the replay buffer without ranking. Table shows the results. We observe that the local score contributes the most the performance. While most existing exploration methods reward exploration globally, the global score could be not aware of generalization, which may explain the unsatisfactory perfor- mance of COUNT and CURIOSITY. While the global score seems play a relatively small role as compared to local score in most MiniGrid environments, it plays an important role in KeyCorridor- S4-R3. A possible reason is that the local score is a too weak signal in KeyCorridor S4-R3, and the global score may help alleviate this issue since it provides some historical information. We also observe that the agent fails in almost all the tasks without buffer or ranking. This suggests that the ranking buffer is an effective way to exploit past good exploration behaviors.
To study RQ3, we plot the learning curves with different training steps S and buffer size in Fig- ure 5a and 5b We observe that larger number of training steps usually leads to faster learning speed at the beginning stage but may result in sub-optimal performance. For the buffer size, we observe that large buffer will lead to slow learning speed with more stable performance. We speculate that there is a trade-off between learning speed and the final performance. In our experiments, we set training steps S to be and buffer size D to be 10000, respectively, across all the tasks.
To investigate RQ4, we remove extrinsic rewards and show the learning curves of RAPID and the baselines in with the corresponding local exploration scores in Figure/5d We make two in- teresting observations. First, without extrinsic rewards, RAPID can deliver reasonable performance. For example, on MultiRoom-NI2-S10 RAPID achieves nearly 0.5 average return with pure explo- ration, compared with around 0.6 if using extrinsic rewards. We look into the episodes generated by RAPID with pure exploration and observe that, most cases, the agent is able to reach the goal but struggles to find the optimal path to the goal. This is expected because the extrinsic reward is obtained by whether the agent can reach the goal subtracting some penalties of the timesteps spent. The extrinsic reward can encourage the agent to choose the shortest path to reach the goal. For other baselines, we observe that only RIDE and AMIGO can occasionally reach the goal. Second, we observe that some baselines, such as RIDE, are indirectly maximizing the local exploration scores. For example, we observe that the local exploration score increases for RIDE on MultiRoom-N12- S10. However, the local score of RAPID drops after about 10 million timesteps. This suggests the episode-leved score may better align with good exploration behaviors. A possible explanation for the superiority of RAPID is that it is designed directly optimize episodic exploration score.
7

---

Published as a conference paper at ICLR 2021
200
302
300
250
150
130
100
100
100
100
50
3
Emessops
1e5
#mmestps
1e5
#mmestpss
1c5
#imesoops
los
(a) Walker2d
(b) Hopper
(c) InvertedPendulum
(d) Swimmer
Figure 7: Performance on MuloCo tasks with episodic reward. i.e,, non-zero reward is only provided at the end of an episode. All the experiments are run 5 times with different random seeds.
To answer RQS, we focus on MuliRoom--NX-SY and ambitiously vary X and Y to make the en- vironments more challenging. Specifically, we fix Y to be 4 and vary X in the left-hand-sinn of F and fix X to be 4 and vary Y in the middle of Figure Compared with RIDE, our RAPID delivers stronger performance on more challenging mini-worlds.
3.4 MINIWORLD MAZE AND SPARSE MuJoCo RESULTS
For RQ6, the learning curves on 3D Maze are shown in the right-hand-side of Figure 6 RAPID performs significantly better than the baselines. We also conduct experiments on subset of sparse MuloCo tasks (Figure 7). The results show that RAPID improves over PPO and SII.. Interestingly, RAPID achieves more than 200 average return on Swimmer, which surpasses some previous results reported on the dense reward counterpart (Duan et al. 2016 Henderson et al. 2018 Lai et al., 2020). Note that the MuloCo environments are singleton. We use these environments solely to test whether RAPID can deal with continuous state/action spaces. We have also tried adapting Swimmer to be procedurally-
and observe similar results (Appendix
RELATED WORK
Count-Based and Curiosity-Drive Exploration. These two classes of intrinsic motivations are the most popular and proven to be effective in the literature. Count- bast- exploration encourages the agent visit the novel states, which is first studied in tabular setting (Strehl & Littman| (008) and has been recently extended to more complex state space (Bellemare et 2016 Ostrovski et al.)
2017 Tang et al. 2017: Martin et al., 2017 Machado et al.) 2018: Osband et al.) 2019). Curiosity- driven exploration learns the environment dynamics encourage exploration (Stadie et al. 2015 Pathak et al., 2017 Sukhbaatar et al. 2018 Burda et al., 2018a). However, these methods are designed for singleton environments and often struggle to generalize procedurally-geneee
en-
vironments (Raileanu & Rocktäschel] 2022))
Exploration for the generalization of reinforcement learning (Rajeswaran et al., 2017: Zhang et al. 2018a bb Choi al. 2018) and designed procedurally-
Generated Environments. Several recent studies have discussed
environments test the generalization of rein- forcement learning (Beattie et al.) 2016,; Nichol et al.) 2018 Küttler et al. 2020). More recent environments and address this issue with new exploration methods (Raileanu & Rocktäschel 2020 Campero et al.) 2020). This work studies new perspective of exploration bonus in episode-leve level and achieves sig- nificantly better performance than the previous methods on eereedurally-gneerrte benchmarks.
papers show that traditional exploration methods fall short in procedurally-genetee
Experience Replay Buffer and Self-Imitation Learning. Experience replay mechanism is widely used and is initially designed to stabilize deep reinforcement learning (Lin 1992 1993 Mnih et al.) 2015 Zhang & Sutton 2017 Zha et al. 2019b; Fedus al., (020). Seif-Imitation Learning extends the experience replay buffer and uses to store past good experiences (Oh et al. 2018 Guo et al.) 2018 2019) Gangwani et al., 2018). However, Self-Imitation only exploits highly rewarded experi- ences and thus is not suitable for very sparse environments. Our work incorporates local and global scores to encourage past good exploration behaviors in sparse weeeduurall-geneeatte environments.
Episodic Memory, Inspired by the episodic memory of the animals, some studies propose to episodic memory to enhance sample efficiency (Blundell et al.] 2016]] Pritzel et al., 2017). The key idea to memorize and replay the good episodic experiences. More recently, episodic mem-
8

---

Published as a conference paper at ICLR 2021
ory is used to generate intrinsic rewards to guide exploration (Savinov et al. 2018). Never Give Up (NGU) (Badia et al. 2019) further considers local and global novelties in the intrinsic rewards. Our work differs from NGU in that approach the sample efficiency from different angle by treating each episode as a whole and using a ranking mechanism to select good experiences. Thus, good experiences can be exploited multiple times to improve sample efficiency.
LIMITATIONS AND DISCUSSIONS
RAPID is designed to study our hypothesis that imitating episode-level good exploration behaviors can encourage exploration in procedurally-geenraaee
environments. In the presented algorithm, we
have made some simple choices. In order to make RAPID more competitive to the state-of-the-an methods in more complicated environments, in this section, we highlight some limitations of RAPID and the potential research directions.
First, in our work, the local and global scores are mainly calculated on low-dimensional state space. We have not yet tested the scores on more complex procedurally--eenrraa environments, such as ProcGen benchmark (Cobbe et al., 2019), TextWorld (Côté et al.) 2018), and NetHack Learning Environment (Küttler al.] 2020), some of which require high-dimensional image inputs. In this context, count-based scores may not well-suited. Learning the state embedding (Raileanu & Rocktäschel 2020) and pseudo-counts (Bellemare et al. 2016 Ostrovski et al. (017) could be the potential ways to address this issue. Moreover, high extrinsic rewards may not necessarily mean good exploration behaviors in these environments. The applicability of RAPID in these environ- ments needs future research efforts.
Second, the ranking buffer greedily selects the highly-reward state-action pairs, which is the key to boosting the performance. However, with this greedy strategy, some highly-ranked state-action pairs may never be forgotten even if they are out-of-date, which may introduce bias. As shown in Figure 5b] a larger buffer size will lead to more stable performance but will sacrifice the sample effi- ciency. In more complicated environments, selecting the proper buffer size could be non-trivial. Some forgetting mechanism (Novati & Koumoutsakos/ 2019) and meta-learning strategies (Zhal al. 2019b) could be the potential directions better manage the buffer.
Third, in our implementation, we employ behavior cloning, the simplest form of imitation learn- ing, train the policy. In recent years, many more advanced imitation learning methods have been proposed (Ho & Ermon] 2016 Hester et al. (017)). In particular, the idea of inverse reinforcement learning (Abbeel & Ng 2004) is to learn reward functions based on the demonstrations. An interest- ing future direction is study whether
derive intrinsic rewards based the experiences in the buffer via inverse reinforcement learning. Specifically, is there a connection between RAPID and intrinsic rewards from the view of inverse reinforcement learning? Can RAPID be combined with intrinsic rewards to improve the performance? We hope that RAPID can facilitate future research to understand these questions.
CONCLUSIONS AND FUTURE WORK
This work studies how we can reward good exploration behaviors in procedurally--eenraae envi- ronments. We hypothesize that episode-level exploration score could be more general criterion to distinguish good exploration behaviors. To verify our hypothesis, we design RAPID, a practical algorithm that learns to reproduce past good exploration behaviors with a ranking buffer. Experi- ments on benchmark hard- hard-exxlooatton procedurally-geneee nificantly outperforms state-of tf-thear-t intrinsic reward methods terms of sample efficiency and final performance. Further analysis shows that RAPID has robust performance across different con- figurations, even without extrinsic rewards. Moreover, experiments on a 3D navigation task and a subset of sparse MuJoCo tasks show that RAPID could generalize to continuous state/action spaces, showing the promise in rewarding good episode-level exploration behaviors to encourage explo- ration. In the future, will extend idea and test it in more challenging procedurally-geeeene generated environments. We will also explore the possibility combining the episode-level exploration bonus with intrinsic rewards. Last, will try other imitation learning techniques beyond behavior cloning.
environments show that RAPID sig-

---

Published as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, 2017.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Rep- resentations, 2019.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint vrX2:v:612..3801, 2016.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint
2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018b.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 12122, 2020.
Maxime Chevalier- gym miniiwiidorldd environment for openai gym. httts:://gtthi 2018.
com/
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. Gitllub repository, 2018
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. In International Con- ference on Learning Representations, 2018.
Karl Cobbe. Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion benchmark reinforcement learning. arXiv preprint arXiv:1912.01550 018888. 2019.
Marc-Alexandre Côté. Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, al. Textworld: A learning environment for text-based games. In Workshop on Computer Games, 2018.
Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Liir: Learning individ- ual intrinsic reward multi-agent reinforcement learning. Advances Neural Information Processing Systems, 2019.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, 2016.
10

---

Published as a conference paper at ICLR 2021
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle. Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Con- ference Machine Learning, 2020.
Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. Interna- lional Conference on Learning Representations, 2018.
Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lee. Deep learning for reward design to improve monte carlo tree search in atari games. In International Joint Conference on Artificial Intelligence, 2016.
Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning. arXiv preprint arXiv:1812.00950. 2018.
Yijie Guo, Jongwook Choi, Marcin Moczulski, Samy Bengio, Mohammad Norouzi, and Honglak Lee. Efficient exploration with self-imitation learning via trajectory-condttioned policy, arXiv preprint
2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Amold, et al. Deep q-learning from demonstra- tions. 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, 2016.
Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici. Ed- ward Grefenstette, and Tim Rocktäschel. The nethack learning environment. arXiv preprint arXiv:2006. 10:0061.7700 13760, 2020.
Kwei-Herng Lai, Daochen Zha, Yuening Li, and Xia Hu. Dual policy distillation. In International Joint Conference on Artificial Intelligence, 2020.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02979 2015.
Long-Ji Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321 1992.
Long-Ji Lin, Reinforcement learning for robots using neural networks. Technical report, Carnegie-
Mellon Univ Pittsburgh PA School of Computer Science, 1993.
Xiao-Yani Liu, Hongyang Yang, Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. arXiv preprint urXiv:2011.09607 2020.
Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count-based exploration with the successor representation. arXiv preprint arXiv: 1807. 1622, 2018.
Jarryd Martin, S Suraj Narayanan, Tom Everitt, and Mareus Hutter. Count-based exploration in fea- ture space for reinforcement learning. In International Joint Conference on Artificial Intelligence, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):59-5333 2015.
11

---

Published as a conference paper at ICLR 2021
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforeement learning. In International Conference Machine Learning, 2016.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. arXiv preprinl arXiv: 180.003000 2018.
Guido Novati and Petros Koumoutsakos, Remember and forget for experience replay. In Interna- tional Conference on Machine Learning, 2019.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, 2018.
Ian Oshand, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Reseurch, 20(124):1-62, 2019.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning. 2017.
Pierre-Y Yves Oudeyer and Frederie Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. arXiv preprint 17X3v:1703.0/99 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-geneeee environments. In International Conference on Learning Representations, 2020.
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- cralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, 2017.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning, 2018.
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability, In International Conference on Learning Representations, 2018.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In International conference simulation of adaptive behavior: From animals to animats, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaskowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: wxXiv:912.00287, 2019.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 1507.0884 2015.
12

---

Published as a conference paper at ICLR 2021
Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133, 2008.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to explore with meta-policy gradient. In International Conference on Machine Learning. 2018.
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: A toolkit for reinforcement learning in card games. arXiv preprint arXiv: arXiv:1910.0499 1910.0437, 2019a.
Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. In International Joint Conference Artificial Intelligence, 2019b.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv: 07937, 2018a.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv: 06893, 2018b.
Shangtong Zhang and Richard S Sutton. deeper look at experience replay. NIPS Deep Reinforce- ment Learning Symposium, 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, 2018.
13

---

Published as a conference paper at ICLR 2021
A HYPERPARAMETERS AND NEURAL NETWORK ARCHITECTURE
For the proposed RAPID, the common hyperparameters are summarized in Table RAPID based on the PPO implementation from OpenAl baselines The nstep is 128 for all MiniGrid envi- ronments and MuJoCo environments, and 512 for KiniWorld-Mazze report the results with only imitation loss since find that the policy loss of PPO will harm the performance in the episodic reward setting. The learning rate 10 for all MiniGrid environments and
For MuJoCo environments,
and 10 10-4 for MuJoCo environments. Since MiniWorld-MMazeSS and MuloCo environments have continuous state space, we set W2 = 0. In practice, in MiniWon World- Maze-S5, observe that counting the states will usually lead memory error because it is likely to encounter the same state again in MiniWorld-Mazz due memory issues). Note that possible use pseudo-counts (Bellemare et al. 2016 Os- trovski et al. 2017), which we will study in our future work. For Keynrrid--KyCCrroorSSS2 MiniGrid KiniGrid-KeyCortii KeCCridoosSRR3-vl and MiniWorld -azed-, the update frequency of imitation learn- ing is linearly annealed to 0 throughout the training process. For other environments, the imitation learning is performed after each episode. We keep all other hyperparameters the default CNN architecture provided in OpenAI baselines for Mini HiniWorld-Mann -azee-S and MLP with 64-64 for other environments. The PPO also uses the same hyperparameters. We summarize the state space of each environment and how the local and global scores computed in Table
(COUNT is excluded in comparison
PPO as default and
Hyperparameter
Value
Wo
wi W2 buffer Size
0.001
10000
batch Size
256
number of update steps
entropy coefficient
0.01
value function coefficient
0.5
X
0.99
0.95
clip range
0.2
Table 2: Common hyperparameters of the proposed RAPID (top) and the hyperparameters of PPO
baseline (bottom).
Environment
State Space
Local Score
Global Score
8 MiniGrid Environments
7 x 7 x 3, discrete
Eq. 0 Eq
MiniWorld Maze
60 80 3, continuous
Eq.
No
MuJoCo Walker2d
17, continuous
Eq.
No
MuJoCo Walker2d
11, continuous
Eq.
No
MuloCo Walker2d
4. continuous
Eq.
No
MuJoCo Walker2d
continuous
Eq.
No
Table 3: The calculation of local and global scores for all the environments. Note that if the state space continuous. We disable the global score since counting continuous states meaningless. A possible solution is to pseudo-counts (Bellemare et al. 2016 Ostrovski al. 2017).
For RANDOM. we implement the two networks with 64-64 MLP. For CURIOSITY and RIDE, we use 64-64 MLP for state embedding model, forward dynamics model and inverse dynamics model, respectively. However, with extensive hyperparameters search, we are not able to reproduce the RIDE results in MiniGrid environments even with the exactly same neural architecture and hyper- parameters as suggested in (Raileanu & Rocktäschell (0000). Thus, we use the authors' implemen- tation Raileaauu & Rocktäschel 2020), which is well-tuned MiniGrid tasks. Our reported result
httes://ggthha.cccm/aaassoeeeere hwtp://eaacacciccccc--eae dripattdrivee-eepp ooatiinn
14

---

Published as a conference paper at ICLR 2021
par with that reported in (Raileanu & Rocktäschel 2020 For SIL and AMIGO, the implementations from the authors with the recommended hyperparameters.
For the methods based intrinsic rewards, i.c. RIDE.. AMIGO CURIOSITY, RANDOM, and COUNT, we search intrinsic reward coefficient from {0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001). For RIDE, the intrinsic reward coefficient is 0.. for MiniGrid-Key Saanfnnddd.eyyyoordonn223----- MiniGrid- KeyCoridorS3RS-v0 MiniiCri-MulliRRo-NNSSA--4 and MiniWorld-Maze--S N7S8-v0, MiniGrid-MulliRoom-NNIi 10-v0 and MiniGridMMuiiRRo-NNISSIII and CURIOSITY, the intrinsic reward coefficient is 0.1 for all the environments. For COUNT, the intrinsic reward coefficient is 0.005 for all the environments. For RANDOM, the intrinsic reward coefficient is 0.001 for all the environemnts. We also tune the entropy coefficient RIDE from {0.001, 0.0005), as suggested by the original paper. The entropy coefficient is 0.0005 for MiniGrid Seerrii-KeCCrrriooSS2eeee MiniGrid- MutGGri-MMtiiRoo--NOSA-AAA 0.001 for MiniGrid-MutiRoomNNSS-v00 MiniGrid- MuniGridMMlliRRooNNNOSSIIII MultiRoomNN1210--00
and 0.5 for MiniGrid-MuuliRoom- For AMIGO
MiniGrid-Ke MiniGrid-KeyCorridde KeyCoridooS3333- MiniGrid- SenoGrdd-KeCCorrdooSSSRCCCC and MiniWorld-Maze- and and MiniGrid-
The Wg, W1. and uh are selected as follows. We first run RAPID on
vO. We choose this environment because not too hard too easy, representative among all the considered MiniGrid environments, We fix wi to be 1, and select wi and W2 from (10-1, 10-3 10-2, 10-1 10 10 102, 103 104), that is, there 9 x 9 81 combinations. Note that we can fix wo because only the relative ranking matters in the ranking buffer. The selected w and w2 are then directly used in other environments without tuning. We further conduct a sensitivity analysis on that RAPID has good performance in a very wide range of hyperparameter choices.
expect that it is
in Figure g We observe
0.3
Figure 8: Sensitivity analysis of wi and (wo is fixed to 1) in MiniGrid-MutiiRom--T77 the experiments are run 3 x 10 timesteps. Note that 3 x 10 is more than enough for RAPID to converge in this environment (see 4). The average results over 5 independent runs are plotted.
All
B ENVIRONMENTS
B. MINIGRID ENVIRONMENTS
MiniGrid Fiis suite of liight and fast gridworld environment with OpenAl gym interfaces. The environments are partially observable with grid size of N x N. Each tile the gird can have most object, that is. or object. The possible objects are wall, floor, lava, door, key, ball, box and goal. Each object will have an associated color. The agent can pick at most one object (ball key). To open locked door, the agent must use key. Most of the environments in
https: httee///hhaa.aeaannee////aaaattt wtt.iiaatteeediiiiiiiin..i
15

---

Published as a conference paper at ICLR 2021
MiniGrid snccedurally-generatce i.e. different grid will be sampled in each episode. Figure shows the grids in four different episodes. The procedurally-geneeeee challenging because the RL agents need to learn the skills that can generalize different grids. The environments can be easily modified so that we can create different levels of the environments. For example, modify the number of rooms and the size of each room create hard-exploration or casy-exploration environments. The flexibility of MiniGrid environments enables us to conduct systematic comparison of algorithms under different difficulties.
nature makes training RL
Figure 9: Rendering of MultiRomm-N1-SID (top row) and CoCorrrdoo-S-RR(oooar row) different episodes. The environments are proceually-generaate i.c., different room is generated new episode.
The original observations of MiniGrid dictionaries, which consist of image field providing partially observable view of the environment, and mission field describing the goal with texts. In our experiments, we use ImgObsWrapper which only keeps the image field. The environment uses compact encoding with 3 input values per visible grid cell. Note that the cells behind the wall or unopened doors are invisible.
The possible actions in MiniGrid are (1) turn left, (2) turn right, (3) move forward, (4) pick up an object, (5) drop the carried object, (6) open doors/interact with objects, and (7) done. The agent will remain in the same state if the action is not legal. For example, if there is no object in front of the agent, the agent will remain in the same state when performing action (4) pick up an object.
We focus MultiRoom-NX-SY and KeyCorridor-SX-RY where X and Y are hyperparameters specifying the number of rooms or the room sizes in the environments. With larger X and Y, the environments will become more difficult to solve due to the sparse rewards. Figure I0 shows the environments (with different difficulties) in this work. MultiRoom-NX-ST the agent needs to navigate the green tile in the last room. If the agent successfully navigates the goal, the agent will receive a reward of subtracting some penalties the timesteps spent. In KeyCorridor-SX--B agent needs pick up the key, use the key open the locked door, and pick up the ball. Similarly, the agent will receive reward of I subtracting some penalties of the timesteps spent if picks up the ball. Both environments are extremely difficult to solve using RL alone due the sparse rewards.
the
B.2 MINIWORLD MAZE ENVIRONMENT
MiniWorld is minimalistic 3D interior environment simulator, Similar to Minigrid, we can easily create environments with different levels. We focus on MiniWorld-Maze, where the agent is asked to navigate the goal through a seredurrlly-ggnerrate maze. The agent has first-person partially observable view of the environment. This environment extremely difficult solve due to sparse reward, long-horizon, and the procedurally-generaa nature of the environments.
world
16

---

Published as a conference paper at ICLR 2021
(a) MultiRoom-N7-S4
(b) MultiRoom -10-m-4 (c) MultiRoom-NN-S8
(d) N
(e) KeyCorridor-3344
(l) KeyCorridor-S3-4
(g) KeyCorridor-44-R3 (h) SilliRooo-NN1-SSS
Figure 10: Rendering of Minigird environments used the work.
In this work, we focus MiniWorld-Mazz-SS variant of the MiniWorld Maze environment with x tiles. Figure IT the top view of the mazes four different episodes. each episode, agent and the goal are initialized in the top-left and bottom-right corners of the map, respectively. In this way, we ensure that the agent and the goal are far away enough so that the agent can not easily see the goal without exploration. A random maze will be generated in each episode. There are three possible actions: (1) move forward, (2) turn left, and (3) turn right. The agent will move 0.4 x tile Jength if moving forward, where tele_lnngth is the length of one side of tile. The agent will rotate 22.5 degrees turning right/left. The time budget of each episode is 600. The agent will receive any positive reward if it can not navigate the goal under the time budget.
Figure 11: Example generated mazes diffident episodes.
B.3 SPARSE MuJoCo ENVIRONMENTS
Mujoco is physical engine for continuous control (Todorov et al. 2012). Figure shows the rendering of the four MuloCo tasks used in our work. The original MuloCo environments use dense rewards, i.e., well-defined reward is given in each timestep. However, in many scenarios, well-defined dense rewards may be not available, We consider variant of MuJoCo task using only episodic reward. Specifically, the original rewards are accumulated and only episodic reward is given the final timestep. The agent will receive a reward in all the other timesteps. This sparse setting makes the tasks much more challenging solve. Contemporary RL algorithms will usually suffer from the sparse rewards and deliver unsatisfactory performance. We aim to use these sparse variants study whether our RAPID effectively capture the sparse rewards.
17

---

Published as a conference paper at ICLR 2021
Walker2d
(b) Hopper
InvertedPendulum
Swimmer
Figure 12: Rendering of Mujoco environments.
C ABLATIONS
Figure 13 shows the learning curves of RAPID against different ablations. Without local scores, observe Significant performance drop on challenging environments in terms of sample efficiency and final performance, i.e., on MultiRoomNN7-S. KeyCorridorS333 -orridor-S3-R3, and R4 This suggests that the proposed local score plays an important role in encouraging exploration. We also observe that without buffer, the agent fails to learn good policy. Naively using the proposed scores as intrinsic reward will harm the per- formance. Using the extrinsic reward and the global score rank the episodes also contribute in Therefore, the three proposed scoring
MultiRoom-N1O-S)
MultiRoom-N12-5
the challenging environments, such as KeyCorridor-S4-PR methods may be complementary and play different roles in different environments.
RAPID
W/o reward
wio local
wio global
w/o butfer
0.5
0.4
0.4
0.2
5.0
0.9
3.5
2.0
0.0
3.0 Umnstepa
3.0 Umiictes
3.5 Umrictess
Emezaggs
167
le7
le7
1e7
(a) MultiRoom-N7-S4
(b) MultiRoom-N10-S4
(c) MultiRoom-N7-S8
MultiRoom-N10-110
0.4
0.2
52
0.1
0.0
5.0
Emmecips
limecheps
167
Birecheps
107
limenheps
KeyCorridor-S3--oe
KeyCorridor-S3-RR
(g) KeyCorridor-44-R3 SoltiiRoom-N12-35
Figure 13: Learning curves of RAPID and the ablations
D FULL ANALYSIS RESULTS
D.1 IMPACT OF THE NUMBER OF UPDATE STEPS
18

---

Published as a conference paper at ICLR 2021
0.8
0.6
0.2
5.0
0.9
0.0
0.3
2.0
3.5
2.0
2.0
3.5
3.0
1.5
2.0
0.0
0.5
3.0 Grrestep
1.3
2.8
0.0
0.5
3.5 Brresteps
1.3
2.0
Emrespps
Grrestps
(a) MultiRoom-N7-S4
(b) MultiRoom-N10-A4
(c) MultiRoom-N7-S8
(d) MultiRoom-N10-S1
1.0
0.6
0.0
2.2
1.5
2.0
0.0
3.0 timesteps
0.0
3.0
1.5
3,0 timenteps
1.5
Emessipss
167
167
timesteps
l6?
167
(c) KeyCorridor-S3-RF (f) KeyCorridor-S3-R3
KeyCorridor-S4-R
(h) MultiRoom-Niz N12-S10
Figure 14: Impact of training steps S on MiniGrid enviromments.
D.2 IMPACT OF BUFFER SIZE
50000
0.5
0.4
0.2
0.2
0.0
3.0
107
Brrechpp
127
Brrechppe
127
(a) MultiRoom-N7-S5
MultiRoom-NiO-SA
(c) MultiRoom-N7-S8
MultiRoom-NlOs.
5.8
0.4
0.6
5.8
3.4
02
9.0
1n7
Smeekaps
Umiskaps
Umistaps
(e) KeyCorridorSS S3-R2
KeyCorridor-SS3
(g) KeyCorridor-S4an
(h) MultiRoom-N122
Figure 15: Impact of buffer size D on MiniGrid environments.
19

---

Published as a conference paper at ICLR 2021
D.3
PURE EXPLORATION ON MINIGRID
CURIOSITY
RANDOM
0.4
0.4
0.4
3.4
0.0
0.5
1.5
3.5
2.0
0.0
3.5
1.0 Gmistpps
1.5
2.0
0.0
0.5
3.5 Grnsstps
1.5
2.0
0.5
0.5
3.5 Grnesteps
1.5
2.0
Emirzaps
1+7
1+7
1+?
(a) MultiRoom-N7-S4
(b) MultiRoom-N10-a
(c) MultiRoom-N7-S8
(d) MultiRoom N10-5S10
2.6
0.35
2.4
0.4
0.3
0.00
0.0
0.5
1.0 Emeeaeps
3.5
2.0
0.0
0.5
3.0 tirandeps
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
3.0 tirantape
1.5
2.0
167
107
Jimmbeps
16?
167
(c) KeyCorridor-S3-R2
(f) KeyCorridor-S3-R3
(g) K4yCorrido-S44-33
(h) MultiRoom-Ni2-S10
Figure 16: Extrinsic rewards achieved by RAPID and baselines with pure exploration
D.4 LOCAL EXPLORATION SCORE OF PURE EXPLORATION ON MINIGRID
RAPID
AMIGO
CURIOSITY
RANDOM
COUNT
S31.
190
eetoraiin 2005
0.8
0.6
0.2
0.2
0.0
1.0
1.0
3.0
(a) MultiRoom-N7-S4
(b) MultiRoom-N10-SA
(c) MultiRoom-N7-S3
(d) MultiRoom-N10-
0.25
eoiratin zore
0.35
0.30
5.2
0.2
0.30
0.0
3.5
2.0
0.0
3.0
1.0
2.0
3.0 Brisstes
Emersags
dresstup
(e) KeyCorridor S3-2
(f) KeyCorridor-S3-PP
(g) KeyCorridor R3
(h) MultiRoom MolliiRoom-N12-110
Figure 17: Local exploration scores achieved by RAPID and baselines with pure exploration
20

---

Published as a conference paper at ICLR 2021
E
LEARNING CURVES WITH MORE ROOMS
0.5
0.8
0.6
3.4
0.4
0.9
0.0
0.5
2.0 Emresipps
1.5
2.0
0.0
0.5
3.0 timesteps
1.5
2.8
0.0
0.5
3.0 tiresteps
1.5
2.
0.2
0.5
3.3 liresteps
1.5
2.
(a) MultiRoom-N4-Se
(b) MultiRomm-N8S4
(c) MuliRoom--N1-SS
(d) MuliRRomm-N16-44
0.6
0.4
0.4
0.0
-02
-02
0.5
0.5
3.0 Fimecteps
155
2.0
0.0
3.5
1.0 Timesteps
1.5
2.0
0.0
0.5
3.0 timesteps
3.5
2.0
le?
1e7
167
(c) MultiRoom-N200 -22-S44
(f) SictiRRoo--24----
SoltiRRom-N28--11
Figure 18: The learning curves with more rooms
F LEARNING CURVES WITH LARGE ROOMS
0.8
0.1
02
1.0
1.5
2.0
0.0
1.0
2.0
0.0
1.0 Umieztps
2.2
dmesttpp
Emiezape
(a) MultiRnom-N4-S4
(b) MultiRoom--44-
(c) MultiRnom-N4-S12
0.2
0.1
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.2
1.5
2.2
0.0
0.5
1.0
1.5
2.2
Umestops
Emnezaps
Uriezope
(d) Multiooom-N4
(e) MultiRoom-Ve N4-S220
(f) MultiRoom-N44 -4-S244
0.4
0.3
0.0
0.5
1.0 Umandapp
1.5
2.0
0.0
0.5
1.0 Emeccags
1.5
2.5
0.0
0.5
1.0
15
2.
167
167
Bmmadags
167
MultiRoom-Ne N4-S28
(h) -4-iRooom-N4-S32
(i) MultiRoom-N4-S36
Figure 19: The learning curves with larger rooms sizes.
21

---

Published as a conference paper at ICLR 2021
G
ABLATION AND ANALYSIS OF MINIWORLD-MAZE
RAPID
0.6
W/o cal
w/oo buffer
0.4
0.2
0.0
timesteps
1e6
Figure 20: Learning curves of RAPID and the ablations on MiniWorld Maze. We observe minor
performance drop when removing the local score, substantial performance drop when removing extrinsic rewards or the buffer.
0.8
0.6
10
20
0.4
0.2
0.0
0
2
4
5
timesteps
1e6
Figure 21: Impact of training steps on MiniWorld Maze.
0.8
5000
0.6
10000
20000
0.4
50000
0.2
0.0
0
2
4
timesteps
1e6
Figure 22: Impact of buffer size on MiniWorke Maze.
22

---

Published as a conference paper at ICLR 2021
0.20
RAPID
PPO
0.15
SIL
RANDOM
0.10
M& Vh umy
CURIOSITY
0.05
0.00
1
2
timesteps
1e6
Figure 23: Extrinsic rewards achieved by RAPID and baselines on MiniWorld Maze with pure exploration.
3000
RAPID
2800
SIL
RANDOM
2600
CURIOSITY
2400
V
2200
2000
4
5
timesteps
1e6
Figure 24: Local exploration scores achieved by RAPID and baselines on MiniWorld Maze with pure exploration.
H COMPARISON WITH STORING ENTIRE EPISODES IN THE BUFFER
variant of RAPID is keep the entire episodes in the buffer instead of state-action pairs. The intuition of keeping the whole episode is that it is possible that particular state-action pair may only be good terms of exploration, episode. To test this variant, we force the buffer to keep the entire episode by allowing the episode the end of the buffer to exceed the buffer size. For example, given that the buffer size 5000, the length of the end episode is 160 and the number of state-action pairs in the buffer is 5120, we do not cut off the end episode and exceptionally allow the buffer to store 5120 state-action pairs. In this way, we ensure that the entire episodes are kept in the buffer. We run this experiment MiniGrid- MiniGrid-MultiRone MultiRoomm-N--S--W (Figure 25). We do not observe a clear difference the learning curves.
the context of the rest of the agent's trajectory in that
23

---

Published as a conference paper at ICLR 2021
0.6
return
0.2
Not Keeping
Keeping All
0.0
0.0
0.5
1.0
1.5
2.0
timesteps
1e7
Figure 25: Comparison of keeping all the state-action pairs of an episode and not keeping all the state-action pairs (i.e., fixed buffer size) on MultiRoom-N7-S8. The experiments run times with different random seeds. We observe no clear difference in the learning curves of these two implementations.
I RESULTS ON SEEEEDRRALLYGGEEERAA SWIMMER
The standard MuJoCo environments are singleton. Here, we modify the Swimmer-y2 environment make proceduralll--genrratee Specifically, make the environment in each episode different by modifying the XML configuration file of MuJoCo engine in every new episode. We consider two environmental properties, i.e., density and viscosity. Density is used to simulate lift and drag forces, which scale quadratically with velocity. Viscosity is used to simulate viscous forces, which scale linearly with velocity. In the standard Swimmer-v2 environment, the density and the viscosity are fixed to 4000 and 0.1, respectively. In the first experiment, we uniformly sample the density in (2000, 4000] each new episode make Similarly, in the second environment, uniformly sample the velocity in [0.1, 0.5], denoted as Velocity-
-eaettrraldygggeaeeeedd.eee denoted as Density-Swimmen
These two variants make RL training more challenging since the agent needs to learn a policy that generalize to different densities and velocities. Similarly, we accumulate the rewards to the final timestep of an episode to make the environments sparse. Both environments provided in our code for reproducibility. The results reported in Figure 26 We observe that all methods learn slower on these two variants. For example, RAPID only achieves around 190 while RAPID achieves more than 200 return in Swimmer. Similarly, SIL reaches around 100 return after around 5 x 10t timesteps in around 100 return after around 3 x 10 timesteps in Swimmer. Nevertheless, RAPID can discover very good policies even in these challenging poneduralll-ggenrraee settings.
return in Density-Swimnet
while it achieves
24

---

Published as a conference paper at ICLR 2021
200
300
200
250
150
100
150
100
100
0
1
2
3
timestees
timesteps
1e6
(a) Swimmer
(b) -onnity-Swiimerr
(c) Velocity-Swimmer
Figure 26: (a) is the singleton swimmer environment. (b) the density is seoeedurally-generatte In (c) the velocity is procedurally-geneeee All the methods tend to learn slower in the procedurally- generated settings. Nevertheless, RAPID still able discover good policies in these challenging variants.
DISCUSSIONS OF ANNEALING
In our experiments, we find that annealing the updating step to sometimes helps improve the sample efficiency. However, in some environments, annealing will have negative effect on the final performance. To show the effect of annealing, we plot learning curves with or without annealing on MiniGrid-Keee KeyCorridorS3R2-0 and Mut-Grrd-MullRRoo-N222222
in Figure 27 We observe that annealing can help in KeyCorridor S3-R2. The possible explanation is that, after the agent navigates the goal, it has access the reward signal. the PPO objective is sufficient to train good policy with the extrinsic rewards. The imitation loss may harm the performance since could be better to perform exploitation rather than exploration at this stage. However, in MultiRoom- -N12- 1-2 S10, annealing will harm the performance. In particular, the performance drops significantly in the final stage when the updating frequency of imitation loss approaches that MultiRomm-N12-1 can navigate the goal, it may easily suffer from insufficient exploration when encounters new environment. As result, the agent needs to keep exploring the environment in order to generalize. Introducing imitation loss will ensure that the agent keeps exploring the environments.
The possible reason is needs very deep exploration to find the goal. Even though the PPO agent
0.6
0.8
0.6
0.4
return
return
0.4
0.2
0.2
Annealing
Annealing Arnea ling
0.0
Annealing
0.0
0
1
2
0.0
0.5
1.0
1.5
2.0
timesteps
1e6
timesteps
1e7
(a) KeyCorridor-S3-PR
(b) MultiRoom-N122
Figure 27: Annealing versus not annealing. Annealing is helpful in KeyCorridor-S3a However, the policy fails converge with annealing in MultiRoo-NN12S10..
25


Reviews:
[{"review_id": "ymO8bcCnU6V", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper focuses on building reinforcement learning agents for procedurally generated environments. In particular it presents a method RAPID to intrinsically reward previously seen *good* exploration behaviors by attempting to imitate them in the future. Strong results are presented across multiple procedurally generated environments.\n\nPros:\n1. Procedurally generated environments provide a rigorous test of an agent's ability to generalize as exploration strategies learned cannot be reused as a whole. To counter this, they operate on the hypothesis that high rewarding trajectories are more likely to contain more generalizable exploration behaviors. The break down of this reward into local (per episode) and global extrinsic rewards gives the agent some idea of areas of the game it historically performs poorly on.   \n2. Given that the idea itself is simple (yet effective!), I appreciate that most of the main paper is dedicated to experiments - with each set of experiments designed to answer a particular research question. This make it easily accessible and digestible - answering many of the doubts I had when reading through the intro and methodology sections.\n\nCons: I have a couple of primary concerns mostly dealing with the underlying assumptions made.\n1. This method rewards good exploration behavior historically and as such depends on the ability of the agent to discover such trajectories in the first place.\n2. High extrinsic reward implies good exploration behavior.\nBoth of these assumptions likely do not hold in environments with sparse/ill-placed/deceptive rewards, challenges that environments such as Montezuma's Revenge/Net Hack/Interactive Fiction games are notorious for. In such environments, it is not only difficult to discover good exploration behaviors but also the overall extrinsic rewards themselves often lead to dead ends and globally suboptimal trajectories. The global score, being effectively count-based, alleviates some of this, but it is likely that this form of intrinsic that is based on extrinsic reward will likely fail in such scenarios. It would strengthen the paper to see positive results in such an environment.\n\nMinor: A few of citations to the ProcGen benchmark (Cobbe at al. 2020 https://arxiv.org/abs/1912.01588) and TextWorld (Cote et al. 2018 https://arxiv.org/abs/1806.11532), and the NetHack Learning Environment (Kuttler et al. https://arxiv.org/abs/2006.13760) are missing, being frameworks for looking RL in procedurally generated settings (and also good candidates for future environments to try RAPID in!!).\n\nRegardless, this paper provides a valuable contribution is studying exploration in procedurally generated environments - with the core idea of rewarding historically good behavior being one well proven in the past. The motivation is clear and the experiments well designed - proving that there are indeed environments where this idea works on, within the limits of the assumptions made. I would see this work accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a7cdAE1Gf7a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents RAPID, an exploration algorithm for procedurally generated environments. The paper introduces an exploration scores composed of a local and global score. The local score is computed per-episode, it is the fraction of distinct states visited during an episode, the global score keeps track of the exploratory effort of the agent over the whole training procedure.\nThese two scores are then added to the environment reward and used to improve the agent's policy with imitation learning. RAPID is shown have a faster convergence and lead to a higher performance than previous exploration algorithms for procedurally generated environments.\n\nI found the paper well written and easy to follow. The authors provided many explanations and figures to illustrate the behaviour of their algorithm.\nThe experiment section was great, it included many experiments on different environments as well as an extensive ablation study to showcase the benefits of RAPID. I think the key insight of this paper is using the local score as a reward for exploration and using it with imitation learning. It seems to be a great proxy for counts in the setting of procedurally generated games in the tabular case. I was actually surprised by the performance of the local score, as it is a per-episode metric it seemed to be a weak signal and that would be hard to learn from it. I'm guessing that the ranking of episodes is key here to make the algorithm work.\n\nOn the other hand I was not convinced by the utility of the global score. First of all it is not clear if the global score is state dependent or not, I assume it is not. The authors say: \"the basic building blocks (e.g., walls, doors, etc.), the number of rooms, and the size of each room will be similar. From a global view, the agent should explore the regions that are not well explored in the past\", unfortunately while similar states may be encountered they will likely not be equal, a metric defined using raw counts will often not be useful. \n\"From a global view, the agent should explore the regions that are not well explored in the past\" This sentence hints at the notion that should be more clearly discussed: generalization. While the local score only quantifies exploration within a single episode, the global score tries to leverage past experience to improve exploration, however to do so as the environment changes constantly some kind of generalization is necessary. I am not sure how a metric that is not aware of the generalization can be helpful. This is also shown in the experiment section, the global score that does not appear to significantly contribute to performance except on KeyCorridor-S4-R3 (any reason why?) and it is even disabled for Mujoco experiments.\nAnother issue is that contrary to some other algorithm like RIDE, RAPID does not easily extend to environments with large or continuous state action spaces because obtaining meaningful counts in these environments is difficult, to be fair this an issue still faced in non procedurally generated environments. It would also be nice to know how RAPID was applied to Mujoco experiments, was the observation space discretized?\n\nOverall I lean towards acceptance as I think that the paper presented a good idea with some solid experiments even though I have doubts regarding how applicable it can be to more complex environments.\n\nFew typos:\nthe methods based on intrinsic rewards -> methods based on intrinsic rewards\nmay take extremely long time -> an extremely long time\nexploration in sparse environments. -> environments with sparse rewards\ncould vary in diffident environments -> different environments\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I96W0Ucpy4C", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper tackles the problem of improving exploration in deep RL for procedurally-generated environments, where state-of-the-art exploration techniques typically fail. In the proposed approach, called RAPID, each agent-generated episode is evaluated with respect to its local exploration score (for the given episode), global exploration score (across all previous episodes), and extrinsic reward obtained. Episodes with high scores are stored in a replay buffer, and a policy is trained via behavioral cloning on batches of state-action pairs from this buffer. This policy is also used to produce the agent-generated episodes.\n\nRecommendation:\nThe approach is simple and intuitive, and empirically improves exploration in procedurally-generated environments. Training on procedurally-generated environments is becoming more common and useful, for instance in domain randomization for sim-to-real transfer in robotics, so this approach would be relevant for the ICLR audience. I have some concerns and questions (detailed below), but overall I recommend acceptance.\n\nPros:\n* The approach itself is simple and easy to implement.\n* The paper is clearly writtten and well-motivated.\n* The empirical evaluation is thorough: RAPID is compared against a suite of state-of-the-art baselines for exploration bonuses, exploration in procedurally-generated environments, and a self-imitation approach; there are also ablation studies and hyperparameter sensitivity studies. The ablation study comparing behavioral cloning versus exploration bonuses (where the per-episode exploration scores are given as part of the reward to the agent) is particularly interesting, given that existing approaches typically rely on the latter.\n* The empirical evaluation is on a variety of domains, including both discrete-action and continuous control tasks.\n\nCons:\n* I would like to see an empirical comparison against Never Give Up (NGU; Badia et al. 2020), which also uses episodic novelty and global novelty to guide exploration. Although NGU uses the same environment for training and testing, because it takes into account how controllable a state is, it wouldn't suffer from the limitation highlighted in Figure 1 (where a state is randomly generated, regardless of the agent's action), and may do well in procedurally-generated environments.\n* It seems strange to me that single state-action pairs are stored in the replay buffer, rather than keeping all state-action pairs from an entire episode together. It's possible that a particular state-action pair may only be good in terms of exploration, _in the context of_ the rest of the agent's trajectory in that episode.\n* The continuous control experiments for MuJoCo locomotion tasks are contrived, since the extrinsic reward for forward progress is summed and given at the end of the episode, which just doesn't make sense for locomotion tasks, and unnecessarily hampers the baseline approaches. Using a goal-reaching continuous control task would be more relevant: e.g., Swimmer in a procedurally-generated maze.\n* I would like to see a discussion of limitations and failure cases for RAPID.\n* There are minor grammatical errors and typos throughout the paper.\n\nQuestions:\n* Can the local score and global score be applied to observations (e.g., image observations) directly, instead of states?\n* Are the scores associated with state-action pairs in the replay buffer taken into account during sampling of batches, or ignored?\n* How were the weights chosen for the total score, that is a weighted sum of the extrinsic reward, local score, and global score? The weight for the global score is very small compared to the others (i.e., 0.001 versus 0.1 and 1). Why is this the case? What happens when this weight is higher (e.g., is there some type of suboptimal behavior that emerges)?\n* Why is it necessary to anneal the imitation learning to zero for some environments, but not others?\n* What is the state space for each of the tasks, that's used to compute the local and global scores? It would be useful to have a table in the Appendix that summarizes this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G2tZhCSlJx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an exploration method for procedurally-generated environments, RAPID, which imitates the past episodes that have a good exploration behavior. First, authors introduce exploration scores, local score for per-episode view of the exploration behavior, and global score for long-term and historical view of exploration. The authors use the weighted sum of these two exploration scores and extrinsic reward as a final episodic exploration score. They rank the state-action pairs based on episodic exploration score and train the agent to imitate behaviors with high score. In experiments, they show the results by comparing state-of-the-art algorithms in several procedurally-generated environments.\n\nThis paper is well written, but I have following concerns and questions about the paper.\n- MiniGrid seems to be a partially observable problem in which only 7x7 observation is given to the agent. What does the state used when calculating the exploration score in the paper mean?\n- Moreover, the paper also includes experiments on problems with continuous state space. In that case, I wonder the details of how the number of states and distinct states are handled.\n- When calculating the episodic exploration score (Equation (3)), it is thought that the results vary greatly depending on the weight $w_0$ - $w_2$ values. In the paper, it seems necessary to show the criteria for determining the weights and the difference in results according to the weights.\n- In order to show the effectiveness of the ranking buffer, in addition to show with/without results as shown in Table 1, the results when using the replay buffer without considering the ranking should be compared together.\n- It would be better if the MuJoCo tasks in Figure 7 could be explained more clearly. (ex. The existing MuJoCo would be a singleton, but what was changed to make it a procedurally-generated environment?)\n- In the experiment, the explanation is mainly based on KeyCorridor, but there is no explanation for what task it is. It would be better if a brief explanation of the KeyCorridor task is included in the paper (with the reason why this task is hard).\n- The author claims that the proposed method is effective in very sparse environments, but I am not sure exactly what makes it effective in very sparse environments. The justification for this claim needs to be explained more intuitively.\n\nThis paper contains experimental results for various tasks, but It is unclear the contribution of this paper and the reason why the proposed method is effective in very sparse and procedurally-generated environments. It seems necessary to explain intuitive motivation more clearly in the paper. Moreover, for the ablation study, It also needs precise verification that can show the effect of each component more intuitively, not ablation that simply separates and compares each component.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In order to learn good exploratory behaviors in settings where agents encounter diverse environments, the authors propose an approach which involves learning from episodes that exhibit good episode-level exploratory behaviors.  The innovation is in the scoring and learning from these episode-level behaviors rather than trying to come up with shorter timescale proxies of exploration.  In making this concrete, the authors propose to score trajectories based effectively on state coverage within an episode (i.e. good exploration corresponds to good state coverage) as well as by scoring episodes relative to one another and giving preference to episodes that explore less often encountered states.  To learn, the core algorithm interleaves standard RL updates with behavioral cloning updates using the best episodes of data, thereby training the policy to both solve the task and explore well at the episode level.

A weakness is that the paper uses low-level state in grid worlds and there is some ambiguity in applying this to settings with continuous states.  The authors discuss general strategies for dealing with these limitations as potential future work.

The reviewers were positive about the clarity of the text and felt the core idea that was proposed was simple and effective.  The authors put in solid effort to address reviewer concerns.  The most salient remaining concern, which I share, is that there will be challenges in scaling this approach to more complex environments with continuous state/observation spaces.

Overall, this paper had a consensus "accept" rating (7,7,7,6), and I endorse this as my decision.
Decision: Accept (Poster)