Revisiting Intrinsic Reward for Exploration in Procedurally Generated Environments
Authors: Kaixin Wang
Kuangqi Zhou
Bingyi Kang
Jiashi Feng
Shuicheng YAN
Venue: ICLR 2023 poster (2023)

Abstract: Exploration under sparse rewards remains a key challenge in deep reinforcement learning. Recently, studying exploration in procedurally-generated environments has drawn increasing attention. Existing works generally combine lifelong intrinsic rewards and episodic intrinsic rewards to encourage exploration. Though various lifelong and episodic intrinsic rewards have been proposed, the individual contributions of the two kinds of intrinsic rewards to improving exploration are barely investigated. To bridge this gap, we disentangle these two parts and conduct ablative experiments. We consider lifelong and episodic intrinsic rewards used in prior works, and compare the performance of all lifelong-episodic combinations on the commonly used MiniGrid benchmark. Experimental results show that only using episodic intrinsic rewards can match or surpass prior state-of-the-art methods. On the other hand, only using lifelong intrinsic rewards hardly makes progress in exploration. This demonstrates that episodic intrinsic reward is more crucial than lifelong one in boosting exploration. Moreover, we find through experimental analysis that the lifelong intrinsic reward does not accurately reflect the novelty of states, which explains why it does not help much in improving exploration.

Published as a conference paper at ICLR 2023
REVISITING INTRINSIC REWARD FOR EXPLORATION IN PROCEDURALLY GENERATED ENVIRONMENTS
Kaixin Wang
Kuangqi Zhou
National University of Singapore kaixin96. wang gma com
National University of Singapore kgzhou525@gman com
Bingyi Kang
Jiashi Feng
Shuicheng Yan
Sea AI Lab
ByteDance j jsh jshfeng@@mail.com com
Sea AI Lab yansc@sea, sea.cssee.comm com
bingykange com
ABSTRACT
Exploration under sparse rewards remains a key challenge deep reinforcement learning. Recently, studying exploration in pooedurally-geeneaaie environments has drawn increasing attention. Existing works generally combine lifelong in- trinsic rewards and episodic intrinsic rewards to encourage exploration. Though various lifelong and episodic intrinsic rewards have been proposed, the individual contributions of the two kinds of intrinsic rewards improving exploration are barely investigated. To bridge this gap, we disentangle these two parts and conduct ablative experiments. We consider lifelong and episodic intrinsic rewards used in combinations
prior works, and compare the performance of all lifelong-epison on the commonly used MiniGrid benchmark. Experimental results show that only using episodic intrinsic rewards can match surpass prior state-of-the of-the--rt art methods. On the other hand, only using lifelong intrinsic rewards hardly makes progress in exploration. This demonstrates that episodic intrinsic reward is more crucial than lifelong one in boosting exploration. Moreover, we find through experimental analysis that the lifelong intrinsic reward does not accurately reflect the novelty of states, which explains why it does not help much in improving exploration.
INTRODUCTION
How to encourage sufficient exploration in environments with sparse rewards one of the most actively studied challenges in deep reinforcement learning (RL) (Bellemare et al., 2016; Pathak et al., 2017: Ostrovski et al., 2017; Burda et al., 2019a;b; Osband et al., 2019; Ecoffet et al., 2019; Badia et al., 2019). In order to learn exploration behavior that generalize to similar environments, recent works (Raileanu& & Rocktäschel, 2020; Zhang et al., 2020; Campero et al., 2021; Flet-Berliac al., 2021; Zha et al., 2021) have paid increasing attention environments (Chevalier-Boiswar on intrinsic reward (Raileanu & Rocktäschel, 2020: Zhang et al., 2020) have proven be quite effective, which combine lifelong and episodic intrinsic rewards to encourage exploration. The lifelong intrinsic reward encourages visits to the novel states that less frequently experienced in the entire past, while the episodic intrinsic reward encourages the agent to visit states that are relatively novel within an episode. However, previous works (Raileanu & Rocktäschel, 2020; Zhang et al., 2020) mainly focus designing lifelong intrinsic reward while considering episodic one only as minor complement. The individual contributions of these two kinds of intrinsie rewards to improving exploration are barely investigated. To bridge this gap, we present in this work comprehensive empirical study to reveal their contributions.
procedurally-generaan et al., 2018; Küttler et al., 2020). Among them, approaches based
grid-like
Through extensive experiments on the commonly used MiniGrid benchmark (Chevalier-Boisvert et al., 2018), we ablatively study the effect of lifelong and episodic intrinsic rewards on boosting exploration. Specifically,
the lifelong and episodic intrinsic rewards considered in prior works (Pathak et al., 2017; Burda et al., 2019b; Raileanu & Rocktäschel, 2020; Zhang et al., 2020), and compare the performance of all lifelong-episodic combinations, in environments with and without extrinsic

---

Published as a conference paper at ICLR 2023
rewards. Surprisingly, find that only using episodic intrinsic rewards, the trained agent match or surpass the performance of the one trained by using other fielong--ppisodi combinations, in terms of the cumulative extrinsic rewards in sparse reward setting and the number of explored rooms in pure exploration setting. In contrast, only using lifelong intrinsic rewards makes little progress in exploration. Such observations suggest that the episodic intrinsic reward is the more crucial ingredient of the intrinsic reward for efficient exploration.
Furthermore, we experimentally analyze why lifelong intrinsic reward does offer much help in improving exploration. Specifically, we find that randomly permuting the lifelong intrinsic rewards within batch does not cause a significant drop the performance. This demonstrates that the lifelong intrinsic reward may not accurately reflect the novelty of states, explaining its ineffectivenesss.
We also compare the performance of only using episodic intrinsic rewards other state-of-the-an methods including RIDE (Raileanu & Rocktäschel, 2020), BeBold (Zhang et al., 2020), AGAC (Flet- Berliac et al., 2021) and RAPID (Zha et al., 2021). We find simply using episodic intrinsic rewards outperforms the others by large margins.
In summary, our work makes the following contributions:
We conduct a comprehensive study on the lifelong and episodic intrinsic rewards in explo- ration and find that the episodic intrinsic reward overlooked by previous works is actually the more important ingredient for efficient exploration in procedurally--enerate gridworld environments.
We analyze why lifelong intrinsic reward does not contribute much and find that it is unable to accurately reflect the novelty of states.
We show that simply using episodic intrinsic rewards can serve as very strong baseline outperforming current state-of-the-arr methods by large margins. This finding should inspire future works on designing and rethinking the intrinsic rewards.
2 BACKGROUND
2.1 NOTATION
We consider single agent Markov Decision Process (MDP) described by tuple (8, R,Y. denotes the state space and A denotes the action space. At time t the agent observes state 8 E S and takes action a, A by following policy π S - A. The environment then yields reward signal r, according to the reward function R(5t, oz). The next state observation 8++1 E S is sampled according the transition distribution function P(s |st, at). The goal of the agent is to learn optimal policy 'T" that maximizes the expected cumulative reward:
π arg Max ET.P TEll
Σ Cr'r
(=0
where IT denotes the policy space and E [o, 1) is the discount factor.
Following previous curiosity-driven approaches (Pathak al., 2017; Burda et al., 2019b; Raileanu & Rocktäschel, 2020; Zhang et al., 2020), we consider that at each timestep the agent receives some intrinsic reward rž in addition the extrinsic reward rf. The intrinsic reward designed to capture the agent's 's curiosity about the states, via quantifying how different the states are compared to those already visited. r can be computed for any transition tuple (st, at 8(41). The agent's goal then becomes maximizing the weighted sum of intrinsic and extrinsic rewards, i.e.,
Ti r + 3r
where 3 is hyperparameter for balancing the two rewards. We may drop the subscript to refer to the corresponding reward whole rather than the reward timestep t.
2.2 TWO PARTS OF THE INTRINSIC REWARD
For procedurally- senduurrll-ggeeerttedd enviromments, recent works (Raileanu & Rocktäschel, 2020; Zhang et al., 2020) form the intrinsic reward as the multiplication of parts:
lifelong
episodic
7( r
2

---

Published as a conference paper at ICLR 2023
lifelong intrinsic reward ylifelong that gradually discourages visits states that have been visited many times across episodes, and an episodic intrinsic reward priisodic that discourages revisiting the same states within episode. ,lifelong is learned and updated throughout the whole training process and piiiodoi is reset at the beginning of each episode.
However, these recent works (Raileanu & Rocktäschel, 2020; Zhang et al., 2020) mainly focus designing better lifelong intrinsic reward while considering the episodic one only as a part of minor importance. In this work, we find that the episodic intrinsic reward is actually more essential for encouraging exploration. For experiments and discussions in the following sections, we consider 4 choices of llfeeoong and 2 choices of episoddi used in prior works, and introduce them below.
ICM Pathak et al. (2017) introduce lifelong intrinsic reward based on learned state representation femb (s). Such representation ignores the aspects of the environment that have little impact upon the agent, and is obtained by jointly training a forward and an inverse dynamic model. The learned model makes a prediction of the next state representation femb((1+ i). and large prediction enror implies that the agent might not explore very well. For transition tuple (st, ats $141), r't
lifllon is:
rt (1lemmb(((t4)) Semb(
Built upon ICM, Raileanu & Rocktäschel (2020) propose novel intrinsic reward which encourages the agent to take actions that have a large impact on the state representation. Similarly, they train inverse and forward dynamic models. The difference is that, for transition tuple (8c, Qt, 8t+1), lifelong is now computed as the change in state representation:
RIDE
T
RIDE
[[.ent( - Semb (8++
(5)
RND Burda et al. (2019b) propose to train state embedding network (() to predict the output of another state embedding network [s) with fixed random initialization. The prediction error is expected to be higher for novel states dissimilar to the ones f(s) has been trained on. Therefore, for hiteoong.
transition tuple (st, dt2 81+1), the prediction error is used as r
RND (11(())1))) [[[(s+4)-f((++)]]]]] f(s++1)ll2.
BeBold Built upon RND, Zhang al. (2020) give a lifelong intrinsic reward that encourages the agent to explore beyond the boundary of explored regions. Intuitively, if the agent takes a step from an explored state to an unexplored state, then the RND error of the unexplored state will likely be lilekon computed as the larger than that of the explored state. For a transition tuple (st, ae, $141), rt clipped difference of RND prediction error between 8, and 81+1:
Bellold = max(0, (s(+1) fs ls((s)) f(s:) l12).
Episodic visitation count Raileanu & Rocktäschel (2020) discount their proposed lifelong intrinsic reward with episodic state visitation counts. Concretely, for a transition tuple (8t, Qtz $t41), r? is computed as
episndic
ep
(8)
V NNe(
where (s) denotes the number of visits state in the current episode. Nep(s) is fast-factum computed such that the denominator is always larger than O.
Episodic first visits Compared with the above episodic intrinsic reward, Zhang al. (2020) take more aggressive strategy: the agent only receives lifelong intrinsic reward when it visits the state for the first time in an episode. That is, for a transition tuple (St, 04: 81+1), episodic is computed as
sp.visit
(N (8t11) 11)
(9)
where I (.) denotes indicator function.
3

---

Published as a conference paper at ICLR 2023
KeyCorridor-SAie S4R3
KeyCoriidor-Son
MultiRoom N-BB8
MuttiRoom-N10S
ObstruciedMaze-10
ObstructedMoze
Agent
Door
Box
Key
Locked door
Obstruction
Goal position
Gool object
Figure 1: The hard-exploration environments in MiniGrid used for our experiments.
DISENTANGLING THE INTRINSIC REWARD
As discussed above, the intrinsic reward for procedurally--ecnraatee environments can be disentangled into lifelong term and episodic term. In this section, conduct extensive experiments MiniGrid benchmark (Chevalier- CheevelierBoiiveert et al., 2018) to study their individual contributions to improving exploration. To this end, ablatively compare different lifelong iffelong-episodic combinations under two settings (Sec. 3.2): (i) a sparse extrinsic reward setting where reward is only obtained upon accomplishing the task, and (ii) pure exploration setting without extrinsic rewards. We find that episodic intrinsic reward is more important for efficient exploration, while lifelong intrinsic reward does not help much. Then, in Sec. 3.3, we also compare the approach using only episodic intrinsic reward to other state-of-the-an methods. Finally, we analyze the reason why the lifelong intrinsic reward does not help improve exploration.
3.1 SETUP
Environments Following previous works (Raileanu Rocktäschel, 2020; Zhang et al., 2020; Campero et al., 2021; Zha et al., 2021; Flet-Berliac al., 2021) on exploration in procedurally- generated gridworld environments, we use the MiniGrid benchmark Sievvalir--Boiveer et al., 2018), which runs fast and hence is suitable for large-scale experiments. As shown in Fig. 1, we use commonly used hard-exploration tasks MiniGrid:
KeyCorriod dorr: The task to pick up an object hidden behind a locked door. The agent has to explore different rooms to find the key for opening the locked door.
Mult MultiRoom Room: The task is navigate from the first room, through sequence of rooms connected by doors, to a goal in the last room. The agent must open the door to enter the next room.
Obst
The task is to pick an object hidden behind locked door. The key is now hidden in box. The agent has to find the box and open get the key. Besides, the agent needs move the obstruction in front of the locked door.
The suffixes denote different configurations (e.g., N7S8 for Mult t i Room refers to 7 rooms of size no larger than 8). For each task, we choose 2 representative configurations from those used in previous works, in order to reduce the computational burden. Following prior works (Raileanu & Rocktäschel, 2020; Zhang al., 2020), we use the partial observation of the grid. The agent chooses from actions (see Appx. A. for details). In our sparse reward setting, the agent obtains positive reward only when reaches the goal position or picks up the goal object. Accomplishing the tasks requires sufficient exploration and completion of some nontrivial subroutines (e.g., pick up keys, open doors). More details about the environments and tasks are deferred to Appx. A.1.
Training details For RI. learning algorithm, we find that IMPALA (Espeholt al., 2018; Küttler et al., 2019) used in previous works takes too long to train and requires lot of CPU resources. Therefore, to make the computational cost manageable, we use Proximal Policy Optimization (PPO) (Schulman al., 2017) to train the policy, which is also high-performing actor-critic algorithm but easier and more efficient to run on a single machine than IMPALA.
Following previous works (Raileanu & Rocktäschel, 2020; Zhang al., 2020: Campero al., 2021; Flet-Berlice Berliac et al., 2021), we use convolutional neural networks to process the input observation (see Appx. A.2 for network architectures). The neural networks optimized with Adam (Kingma
4

---

Published as a conference paper at ICLR 2023
KeyCormdor-$4R3
KeyCorridor-S6k
MuttiRoom-N7Se
MultiRcom-MOOS
ObstructesMoze--U
ObstructedMaaze 2Q
10
ICM
Average
0.4
02-
0.
13
RIDE
06-
Average
04
0.0
0.
RND
06
Average
0.
0.
BeBold
Average
0.2
40 Fromes ( xtb?
0.5
10
20 Frames x1b7
60 Fromes (x107)
Fromes ( (XDD?)
Fromes ( xto7
Fromes (1b
ep_count lifelong
ep_visit lifelong
lifelong only
ep_count only
ep_vish only
Figure 2: Performance different combinations of lifelong and episodic intrinsic rewards on 6 hard exploration environments in MiniGrid. Note that simply using episodic intrinsic reward can achieve top performance among combinations. Best viewed in color.
Ba, 2015). The number of training frames are summarized in Fig. 11, which are kept the same as in (Raileanu & Rocktäschel, 2020; Zhang et al., 2020). All of our experiments can be run on single machine with 8 CPUs and a Titan X GPU. Details about the computation cost are included in Appx. A.3.
Hyperparameters and evaluation One critical hyperparameter in intrinsic-reward based ap- proaches is the coefficient / that balances intrinsic and extrinsic rewards (see Eqn. 2). Varying the value of 3 will have large impact on the performance. Thus for fair comparison, we search 3 for each experiment and present the results with best 3. The searched values of / well as other hyperparameters are summarized in Appx. A.5. As in prior works, we use training curves for performance comparison. The training curves averaged over 5 runs and the standard deviations are plotted as shaded areas. The curves are smoothed following the procedure used by Raileanu Rocktäschel (2020). We also include the testing results in Appx. A.6.1.
3.2 INDIVIDUAL CONTRIBUTIONS OF EPISODIC AND LIFELONG INTRINSIC REWARDS
In this section, we ablatively study how episodic and lifelong intrinsic rewards improve exploration with and without extrinsic reward (i.e., goal-reaching and pure exploration). On the MiniGrid environments, we compare the performance of 15 lifelong-episodic combinations: ICM y RIDE AND BeBold None} x spocontt visiitt None). Here "None" refers to not using lifelong or episodic intrinsic reward. If both are not used, then it reduces to vanilla PPO without intrinsic rewards. If only one not used, then the reward reduces to For the goal-reaching task, the extrinsic reward sparse and only provided when the agent achieves the goal. Exploration ability is measured by how fast the agent can achieve high average return on 6 environments introduced in Sec. 3.1. For the pure exploration setting, train agents Mult Room environments without providing any extrinsic reward when agents reach the goal. MultiRoom iRoom environments consist of several consecutive rooms connected by doors and the agent starts from the first (see Fig. 1). To explore the environment,
Moom
5

---

Published as a conference paper at ICLR 2023
ICM
RIDE
BeBoid
Multioom A7SS
Averogge Number of Rooms Explored
Mu Miliioomm AnoSIo
00
05
10
15
20
05
10
15
20
05
20
90
20
x12
(x007)
Frames (x007)
cp_ count lifelong
ep_ vist lifelong
lifllon only
ap_ count
ep visit
Figure 3: Average number of explored rooms without extrinsic rewards. Best viewed in color.
KeyCorridor-44R3
KeyCoridor-S6R3
MuttRcom-N7SB
MuttRoom-N10510
ObstructedMaze-1Q
ObstructedMaaze22
Average Return
08
0.4
02
0.0
50
6.0 0.0
15 Frames (x10")
2.0
6.0
6.0 Frames (x10'
Fromes (x10/)
Frames (x1o/)
Frames (x10/)
Frames (x10
ep_ count only
ep_ visit only
AGAC
RAPID
Figure 4: Comparison only using episodic intrinsic rewards and state-of-the-art methods. Note that the result of BeBold in AbstruccedMare 2Q is directly taken from the associated paper (Zhang et al., 2020).
the agent needs to open each door and navigate to different rooms. Thus, we can easily use the average number of explored rooms within an episode as a proxy for quantifying the exploration ability. Here we use respectively, and consider room is explored if the agent visits any tile within this room (excluding the connecting door).
N758 and MultiRoom iRoo AeoommNN000 N1 1S1000 which have 7 and 10 rooms
Fig. 2 shows the results of exploration with extrinsic rewards. We can that only using episodic intrinsic reward can match or surpass the performance of combining lifelong and episodic intrin- sic rewards. In contrast, only using lifelong intrinsic reward hardly makes any progress. These observations show that the episodic intrinsic reward is the key ingredient for boosting exploration. Moreover, comparing the performance of different lifelong intrinsic rewards under the same episodic intrinsic reward (i.e., dark red or dark blue curves in each column of Fig. 2), we find that recently proposed ones RIDE and BeBolld do not exhibit clear advantages over previous ones (T. and ,NDD). Additionally, when comparing two episodic intrinsic rewards, we can see they perform comparably well for most environments. An exception the Mult saltiRoon environments, where reivisit performs slightly better. The reason might be that y-Ppiisi is more aggressive than pp.couutt in discouraging revisits to visited states within an episode, pushing the agent to explore new rooms more quickly.
The results for the pure exploration setting are summarized in Fig. 3. Again, when only using lifelong intrinsic rewards (i.e., green curves), the agent fails to explore farther than the first 2 rooms. In contrast, it can quickly explore more rooms when augmented with episodic intrinsic rewards. This further demonstrates that episodic intrinsic reward more important than lifelong one for efficient exploration. Moreoever, comparing pp visit and rP faster. This difference exploration efficiency explains the performance difference between pivvist and pp.conttt in Mul t Room environments with extrinsic reward (see Fig. 2).
see rP visit helps the agent explore

---

Published as a conference paper at ICLR 2023
KeyCorridor-S4pa
KeyCorridor-Soff
MuttRooom-N/Se
Multiooom-NiSS
ObstructedMare-10
OostructedMare-
10
ICM
Average
0.
0.
RIDE
06
Average
0.0
RND
06
Average
00
08
BeBold
Average
0.
40
05
10
10
20
50
Fromes (x10²
Fromes ( x1b?)
Fromes (x10?
Fromes (X107
Frames (x107)
Frames (xl0?)
lifelong epucount
lifelong ep_visit
lifelong (permuted) ep_count
lifelong (permuted) ep
ep_jisit visit
Figure Performance of whether randomly permuting lifelong intrinsic rewards on hard exploration environments in MiniGrid. Best viewed in color.
3.3 COMPARISON WITH STATE-OF-THE-ARE METHODS
From the previous section, we can that only using episodic intrinsic rewards can achieve very competitive performance. In this section, we compare its performance with other state-of-the art approaches, including RIDE (Raileanu Rocktäschel, 2020), BeBold (Zhang et al., 2020), RAPID (Zha et al., 2021) and AGAC (Flet-Berliag et al., 2021), under the sparse extrinsic reward setting. Among the environments used, 1) for environments also used in these works, we contacted the authors and obtained the original results reported in their papers; 2) for other environments, train agents using their official implementations and the same hyperparameters reported in their papers. Please see Appx. A.4 for more details.
As Fig. 4 shows, only using oP visit or y. sount clearly outperforms other state-of-the-ard methods, achieving high average return with fewer training frames. Specifically, in KeyCorridor-Se Obstruct e edMaze-10 and Obstruce woeettdMaae22220 it is almost an order of magnitude more efficient than previous methods. Note that the performance of BeBold in directly taken from their paper but we cannot reproduce it. Compared to these sophisticated methods, simply using episodic intrinsic rewards serves very strong baseline. In Appx. A.6.2, we also include the comparison results on other environments in MiniGrid benchmark.
ze z2-2Q is
3.4
WHY DOESN'T TT LIFELONG INTRINSIC REWARD HELP?
As shown Sec. 3.2, lifelong intrinsic reward contributes little to improving exploration. In this section, we make preliminary attempts to analyze the reason for its ineffectiveness.
To begin with, we examine the learned behavior of an agent trained with the lifelong intrinsic rewards only (see Fig. 2), by rolling out the leamned policies in the environment. We find that the trained agent often stuck in the first room and oscillating between two states. An example, as shown in Fig. 6, is that the agent keeps opening and closing the first door. In comparison, the agent trained with episodic intrinsic rewards only can explore all rooms without oscillating (see videos in the supplementary material). A possible reason for this failure could be that the lifelong intrinsic reward
7

---

Published as a conference paper at ICLR 2023
open
close
open
Agent
Closed door
Opened door
Figure 6: The agent gets stuck, keeping opening and closing the door.
is not able to accurately reflect the novelty of states. During training, the agent may randomly sample a moving-fowe forward action after opens the door, but the lifelong intrinsic reward it receives from this step is lower than the previous opening-door step. Thus, as the training progresses, the agent gets stuck in the opening-closing behavior since this gives higher intrinsic rewards than moving forward to explore new rooms.
To verify this, we randomly permute the lifelong intrinsic rewards within a batch, and evaluate the resulting performance under sparse extrinsic rewards. Specifically, for transition batch {(s). Oy. s})] and the corresponding intrinsic reward batch {r,}ā=ı+ we randomly permute {ry} such that each transition (sj, 9j, si)) might be associated with the wrong intrinsic reward тиK (where & * j) rather than the correct one. If this random permutation does not lead significant performance drop, then it indicates that the computed lifelong intrinsic reward may not accurately measure the novelty states. From results in Fig. 5, we can see that permutation does not have much negative impact the performance for most cases, especially when using y episodic intrinsic reward. This shows that the computed lifelong intrinsic reward may not accurately reflect the novelty of states, otherwise such permutation would lead to obvious performance drop. We note the performance impact is not negligible in one case (e.g., BeBold ep count in MultiRoom environments), but we believe it will not alter the message that the lifelong intrinsic reward is not critical to performance.
4 DISCUSSIONS
In the above experiments, show that episodic intrinsic reward solely is able to boost exploration in mrcceuralll--geneaal grid-world environments, while lifelong intrinsic reward struggles to make progress. Such intriguing observations provide new perspectives to think about intrinsic reward design as below.
Episodic intrinsic reward in latent spaces Our experiments use grid-world environments as testbeds, which have discrete state spaces, so it is easy to obtain visitation counts for each state. Therefore, one may wonder whether episodic intrinsic reward is able generalize continuous state spaces, where it is infeasible to calculate SP visit and 7P count One possible solution to compute the episodic intrinsic reward in latent space (Badia et al., 2019). However, we empirically find that directly borrowing this solution from singleton environments to MiniGrid ineffective, as latent embedding learning is much more challenging environments. For future research, may consider designing more powerful embedding models.
high-dimensional
poocedurally-generaae
Lifelong intrinsic reward in procedurally-generatee like RND (Burda et al., 2019b) and ICM (Pathak et al., 2017) has shown promising performance in singleton environments (e.g., Atari (Bellemare et al., 2013)), but performs poorly in procedurally- generated environments (e.g., MiniGrid). This quality of the representation plays an important role in the success of curiosity- make lifelong intrinsic reward work again, one has think about more generalizable representation learning for RL.. One possible direction is to design proper auxiliary or pretext tasks like self- supervised learning instead of just memorizing the environment.
environments Lifelong intrinsic reward
intuuiive phenomenon further shows that methods. To
RELATED WORKS
Exploration and intrinsic rewards In RL, the agent improves its policy via trial and error. Thus a steady stream of reward signals is critical for efficient learning. Since many practical scenarios come
8

---

Published as a conference paper at ICLR 2023
with sparse rewards, studying how to encourage exploration behavior in such environments attracts continuing attention. One of the most popular techniques is to provide intrinsic rewards (Schmidhuber, 2010), which can viewed as the agent's curiosity about particular states. In the absence of extrinsic rewards from the environment, the agent's behavior will be driven by the intrinsic rewards. If we expect the agent explore novel states, the intrinsic reward can be designed reflect how different the states are from those already visited. Such difference can be measured by the pseudo-counts of states derived from density model (Bellemare et al., 2016; Ostrovski al., 2017). or errors of predicting the next state (Pathak et al., 2017) or a random target (Burda et al., 2019b).
Episodic and lifelong intrinsic reward Prior works (Pathak et al., 2017: Burda et al., 201 90:b))) typically model a lifelong intrinsic reward, which is updated throughout the whole learning process. Recent works focusing on procedurally-genrrate
environments (Raileanu & Rocktäschel, 2020; Zhang et al., 2020) additionally use episodic intrinsic rewards to modulate the lifelong ones. However, they mainly focus on designing the lifelong intrinsic rewards. In this work we show that the episodic intrinsic reward is actually more important in encouraging exploration. Concurrent our work, Henaff et al. (2022) also finds that the episodic intrinsic reward is essential to good performance.
Our work is also related to recent progress on Atari (Badia et al., 2019; 2020), where more attention is paid to episodic intrinsic rewards and the lifelong intrinsic reward is only considered as an optional modulator. They focus on pushing performance limits Atari, while our work aims to compare the individual contributions of episodic and lifelong intrinsic rewards in procedurally-generaae gridworld environments. One interesting connection note is that the episodic visitation count bonus considered in our paper is essentially an MBIE-EB-style bonus (Strehl & Littman, 2008).
Generalization and procedurally--eneeee Zhang et al., 201 Sa;b;;; Machado et al., 2018: Song et al., 2020) find that deep RL is susceptible to overfitting to training environments and suffers poor generalization. Using procedurally gegerratd environments Sehevalier-Boisveee et al., 2018; Chevalier- Boiisver,,, 2018; Juliani et al., 2019; Cobbe al., 2020), where each episode is randomly constructed but corresponds to the same task. To make progress in such environments, an agent must learn generic policy that can generalize to novel episodes. Most prior works exploration are also susceptible to overfitting issues, since they train and test on the same environment, such as Montezuma's 's Revenge (Bellemare et al., 2013) or VizDoom (Wydmuch et al., 2018). To achieve generalizable exploration, recent works (Raileanu & Rocktäschel, 2020; Campero et al., 2021; Zhang et al., 2020: Zha al., 2021; Hit-Berliac et al., 2021) often use ceneeddurall-ggneertte gridworld environments to benchmark their methods.
environments Recent papers (Rajeswaran et al., 2017;
6 CONCLUSIONS
Exploration is a long-standing and important problem in reinforcement learning, especially when the environments have sparse rewards. To encourage exploration in procedurally-generaa environments, recent works combine lifelong and episodic intrinsic rewards. However, how these two kinds of intrinsic rewards actually contribute to exploration is seldom comprehensively investigated. To answer this question, we conducted the first systematic empirical study by making exhaustive combinations of lifelong and episodic intrinsic rewards in environments with and without extrinsic reward signals. and intriguing: episodic intrinsic reward alone is able to boost
The results exploration and set new state-of-the-art. Our further investigation shows that the poor performance of lifelong intrinsic reward roots in its ineffectiveness in distinguishing whether state is novel not. We believe our findings are inspiring for future work to better design intrinsic reward functions. We would like to note that our discovery focuses more on hard exploration tasks MiniGrid, and whether it applies to general procedurally generated environments requires further investigation.
7 REPRODUCIBILITY STATEMENT
We describe the training details (e.g., environment configurations, network architectures, hyper- parameters) in Sec. 3.1 and the Appendix. To reproduce the results, we also include the source code in the supplementary material. The computation cost is detailed in Appx A.3.

---

Published as a conference paper at ICLR 2023
REFERENCES
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi
Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1479-1487, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large- scale study of curiosity-driven learning. In International Conference on Learning Representations, 2019a. URL https: wpwpr/veereeeeeeeeeeeeeeeeeeeeee net / Eorum? r JAqUx
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019b. URL https: //openreview..o.mm/// net htt::/ooooddddliuummaa
Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with famig]o: Adversarially motivated intrinsic goals. In International Conference Learning Representations, 2021. URL httpps////peeeeeee....
net. faat/forum?iie Beeddd=TT__MMMMMP
Maxime Boiiseier-Boisvertt. gym-miniworld environment for openai gym. https://githii com/maximecb//yee wwwp://wwwe.mm.miiiiiiiiaememmme miniworld, 2018.
Maxime Chevaller--oiiveer Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https:://///. / htt.ttttthh..g.gg com/ cetmm/maximecbb htw.mmmme//gmmmmmeeeeee gymm www.ee///ggiiiiiiii....... 2018.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp- 2048-2056. PMIR, 2020.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for exploratoorti problems. arXiv preprint arXiv: 1901. 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. IMPALA: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. In International Conference on Learning Representations, 2021. URL https: //openreview..o net forum? iidiidddmme @ast=__@pcr__iii _iy.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. arXiv preprint arXiv: Arxiv-2210.0502 2022.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019.
Planning
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http::///// //t///axxx /tarxxivv.. org.iiioorr///oooooo orgg /abs///////// 1112. 6980.
10

---

Published as a conference paper at ICLR 2023
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL. arXiv preprint arXiv:1910.03552, 2019. URL https:://// https://githuss htt.://aaaabbboeeeeea....o torchbeast.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61 61:523-562 2018.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62, 2019.
Georg Ostrovski, Mare G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference machine learning, pp. 2721-2730. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning, pp. 2778-2787. PMLR, 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference on Learning Representations, 2020. URL htttps://ppenrvviie net (ttppppmrreeeeeeeeeoooioiiiiiiie Forum?id=rkggTeei
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade. Towards generalization and simplicity in continuous control. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3563-6564, 2017.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-22) 247, 2010.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /ttp:///axvvv.// http:/aawvvi........../////....t 15066 02438.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (too::777.0444 2017.
Xingyou Song. Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Learning Representations, 2020. URL ht httpp::/ /ttp:////eeeeeeeeeeeemmmmmeeeeee net/ wwwp:/eweweeeeeennneeeeeeee
Alexander L. Strehl and Michael L. Littman. analysis of model-basee interval estimation for markov decision processes. J. Comput. Syst. Sci., 74(9):1309-1331 2008. doi: 10.1016/j.jcss. 2007.08.009. URL https: //tttp::://o.... / .org/ /100 1 01 j/j. jcsss 2007. 00.0008... 009.
Marek Wydmuch, Michat Kempka, and Wojciech Jaškowski. Vizdoom competitions: Playing doom from pixels. IEEE Transactions on Games, 2018.
Daochen Zha, Wenye Ma, Lei Yuan, Xia Hu, and Ji Liu. Rank the episodes: A simple approach for environments. In International Conference on Learning exploration in procedurally-geneeee Representations, 2021. URL https: wwwp//wienreeeeeeeo/ee net/ forum Mttrrummii-MMMMttCcCC
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprins arXiv:/806.07937, 2018a.
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. arXiv preprint arXiv: 1.06022, 2018b.

---

Published as a conference paper at ICLR 2023
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint 1e04.80044229 06893, 2018c.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. BeBold: Exploration beyond the boundary of explored regions. arXiv preprint arXiv:2012.08621, 2020.
12

---

Published as a conference paper at ICLR 2023
Environment
Observation
1st channel
2nd channel
3rd channel
0
D
Q
1
Q
0
5
0
0
0
0
1
0
0
Unssenn
Non-cbject
Opon
Empty
6
Ball
Red
Closed
Lacked
Floor
Gool
Blue
Door
Lava
Purple
Agent
Grey
Figure 7: Observation encoding in MiniGrid environment. See text for details. Best viewed in color.
A APPENDIX
A.1 MINIGRID ENVIRONMENT
MiniGrid CChevalier-Boisver et al., 2018) benchmark consists suite of procedurally-genenen gridworld environments. The gridworld is built with N x N tiles, where N varies across tasks. Each tile contains one no object. The possible object types include wall, floor, lava, door, key, ball, box and goal.
By default, the observation an egocentric view the environment, as shown in Fig. 7. The agent not the world behind walls closed doors. Specifically, the observation is compactly encoded as a x 3 array. The first channel encodes the object type, egg., wall, door. The second channel encodes the object color, e.g., red, green. The third channel encodes the state, e.g., door open, door closed.
Observation
Action In MiniGrid, there are actions available for the agent: turn left, turn right, move forward, pick up an object, drop an object, toggle and done. The agent can change the direction it is currently facing by turning left or right. Taking the move forward action will move the agent one tile forward along its current direction. If moving forward causes collision (e.g., running into walls or closed doors), the agent will remain the previous location. The agent can use the toggle action open closed door in front of it (or a locked one if the agent has the corresponding key). The loggle action can also be used open a box. The done action is not necessary for the environments considered in this paper, but we still keep it in the action space following previous works.
Task As briefly introduced in Sec. 3.1, we consider 3 tasks MiniGrid: KeyCorridor, MultiRoom ObstructedMaze Here provide more details about these 3 tasks.
KeyCorridor (Fig. 8) The environment consists of a corridor and several rooms connecting to the corridor. The agent starts in the corridor and aims pick up a goal object placed in of the rooms. The door to this room (i.e., the room with the goal) is locked. To open the locked door, the agent has find the key that is placed in one of the rooms except the locked one. The configuration suffix is specified as SxRy. where denotes the room size and denotes the number of rows. Each row has rooms, placed on each side of the corridor.
MultiRoom (Fig. 9) The environment consists of sequence of rooms connected by closed doors. The agent starts in the first room and aims to reach the goal position in the last room. The configuration suffix is specified as NxSy, where x denotes the number of rooms and denotes the maximum room size.
Obst ruct edaaze (Fig. 10) The environment consists of 3 grid of rooms. In these rooms, I is the center room, are corner rooms and 4 are non-corner rooms. The doors connecting the center room and non-comner rooms are closed. The doors connecting corner rooms and
13

---

Published as a conference paper at ICLR 2023
KeyCorridor-Soe KeyCorridor-S3R) KeyCorridor-S4R)
KeyCorridor-S5R3
KeyCorridor-Sóhe
Figure KeyCorridor environments.
MultiRoom-N750
MultiRcom-N1054
Multiccom-N10500
MultiRoom-N1233
Figure 9; MultiRoom environments.
ObstructedMoze-2DMM
ObstructesMazee-DDhbb
ObstructedMoze-12
ObstructedMMze-220
ObstructedMozz--U
Figure 10: ObstructedMazz environments.
non-corner rooms are locked. The keys are placed in non-corner rooms. Depending on the configuration, the agent starts in the center room or non-corner rooms, and aims to pick up an goal object hidden in one of the corner rooms. The configuration suffix is more complex, so we only introduce commonly used ones and refer interested readers the official GitHub repository!.
2D1h The goal object is always hidden in the top right room. The keys to the comeer rooms are hidden in boxes. The agent always starts in the middle right room.
2D hb Compared to h, the difference is that locked doors are blocked by obstructions.
The agent has to move these obstructions before unlocking the doors with keys.
1Q Compared to 2D1hb, the difference is that the agent starts in the center room instead of the middle right one.
2Q Compared to 12, the difference is that the goal object hidden either the top right room the bottom right room.
Full Compared to 10, the difference is that the goal object is hidden in one of the corner
rooms.
Episode length In these tasks, there is limit on the maximum number of steps allowed in each episode, denoted as fmax. varies from task task, as summarized in Fig. 11.
Reward When the agent achieves the goal, it receives an extrinsic reward rf. This reward is computed based the number of steps taken and the maximum number of steps allowed tmax:
14

---

Published as a conference paper at ICLR 2023
Task
MultiRoom
KeyCorridor
ObstructedMaze
Suffix
N6S10
N758
N10S4
N10S10
N12S10
S3R2
S3R3
S4R3
2Dth
2DIhb
1Q
fmax
120
140
140
200
200
240
270
270
480
750
1080
576
576
720
1384
3600
Fromes
267
2e7
2e7
2e?
2e7
2e7
2e7
267
4e7
4e7
6e7
4e7
6e7
6e7
9e7
1eB
RIDE
v
v
x
X
x
BeBold
V
x
v
x
x
V
x
v
v
v
V
V
V
RAFID
v
v
v
x
x
AGAC
x
x
x
Figure 11: Different MiniGrid task configurations used in previous works.
rf 1 0.9t 0.9t/fmax. The agent is encouraged to reach the goal as fast as possible. Only the optimal path gives the highest reward.
A.2 NETWORK ARCHITECTURES
For the policy and value network in PPO, we use shared feature extractor and two separate heads (one policy head and one value head), as shown in Fig. (a). The feature extractor uses convolutional neural network, similar the ones used in prior works (Railcanu & Rocktäschel, 2020; Zhang et al., 2020; Flet-Berliac et al., 2021). ). For RND and Baooii,, the target and predictor networks the architecture in Figure 12 (b). For JCM and RIDE the network architecture of the state embedding function femb shown in Figure 12 (b). The network architectures of the forward and inverse dynamic model shown in Figure 12 (c) and Figure 12 (d) respectively.
A.3 COMPUTATION COST
single machine with 8 CPUs and a Titan X GPU. Specifically, one run an experiment (e.g. training with JCM /ep.counl) takes less than 600M concurrently several experiments single GPU. On average, training for 1e7 frames takes about 1.5 hours. The total wall-clock training time for all conducted experiments is roughly 4000 GPU hours.
As mentioned in Sec. 3.1, our experiments
GPU memory Titan X GPU. So,
In comparison, the IMPALA used in the released code of RIDE (Raileanu & Rocktäschel, 2020) is typically 10-20x slower than PPO for our hardware. The low efficiency is mainly because it uses CPU to do the forward pass of the policy during experience collection and it is non-trivial for us to migrate it to GPU. This also means that we need large number CPUs (typically 100+) to collect samples in parallel to reach the full speed. Such computation requirement (high CPU but low GPU demand) does not our computation resources (where each workstation often less than 32 CPUs). Therefore, we choose PPO the base algorithm to fully utilize our resources.
A.4 TRAINING DETAILS
We provide more training details for the comparison with other state-of-the-ar methods in Sec. 3.3. For RIDE (Raileanu & Rocktäschel, 2020), we obtained results on MultiRoom N758, MultiRoom-M1OSL1. from the authors, and rerun their released code² for other environ- ments. For KeyCorridor-S4R3 and KeyCorridor-SGRR rameters KeyCorridor -333 provided in their paper. For ObstructedMaze-10 and ObstructedMaze -Q, we use the same hyperparameters as rasttractedMaee2000
we use the same hyperpa-
For AGAC (Flet-Berliac et al., 2021), we obtained results on Multi t Room-N 0S10 0, obstrucctdMaze. and ObstructedMaze- 2Q from the authors, and rerun their released code for other environments. For hyperparametes, we use their default configurations.
15

---

Published as a conference paper at ICLR 2023
(a) Policy and value network architecture
(b) State embedding network architecture
Policy output
Value output
State embedding
Fully Connected (Units: 7)
Fully Connected (Units: 1)
ELU
Convolution
ReLU
(Stride: 2, Paddiigg Channell
ELU
Fully Connected S12)
3x3 Convolution (Strice: 2, Padding:: 1, Channel 32)
ELU
3x3 Convolution (Stride: 2, Padding: Channel: 32)
ELU
3x3 Convolutiin (Stricer Poddling: Channet:
FLU
Input ooservation (7x7x3)
5x5 Convolution (Stride: Padding: Channel: 32)
ELU
3x3 Convolution
(Stride: 2, Padding: Channel: 32)
inverse dynamic network architecture
Input observation (7x73)
Predicted action
(c) Forward dynamic network architecture Predicted next state embedding
Fully Connected (Units:
ReLU
Fully Connected (Units: 128)
Fully Connected (Units: 256)
ReLU
Concotenate
Fully Connected (Units: 256)
State embedding
State embedding
Next state embedding
Figure 12: Network architectures.
For RAPID (Flet-Berliac et al.. 2021), we obtained results on MultiRoom-NN1SSO from the authors, and rerun their released code for other environments. For hyperparameters, we use their default configurations.
and
For BeBold (Zhang et al., 2020). we obtained the results on KeyCorridor-S4R3 MoltiRoom-N758,
KeyCorriddr-s6R3, Obstructeddaze--22 2Q from the authors. and rerun their released code for MultiRoom-NI N1 0. For hyperparameters, we use their default configurations.
ObstructedMazz-19 and
16

---

Published as a conference paper at ICLR 2023
A.5 HYPERPARAMETERS
For training the policy, we use the hyperparameters listed Tab. 1. Generalized Advantage Estimation (GAE) (Schulman al., 2016) is used estimate the advantage. As mentioned in Sec. 3.1, we search the intrinsic reward coefficient / for each experiment. We start with a search range [5e-2, le-2, 5e-3, 1e-3). and expand it until finding a locally best value or reaching 5el le-5. The best searched values for 3 are summarized in Fig. 13, 14, 15, 16, 17. Here we showcase in Fig. 18 that choosing an appropriate value for critical for the performance. Other results in the search (i.e., using different (3) will be made publicly available.
Table I: Hyperparameters.
Number of parallel environments
128
Number of timesteps per rollout
128
PPO clip range Discount factor 7 GAE A
0.2
.99
.95
Number of epochs
4
Number of minibatches per epoch
8
Entropy bonus coefficient
0.01
Value loss coefficient
0.5
Advantage normalization
Yes
Gradient clipping (&2 norm)
0.5
Learning rate Total timesteps
10-1
See Fig.
Forward dynamic loss coefficient Inverse dynamic loss coefficient
10
0.1
RND loss coefficient
0.1
Environments
KeyComddor
MultiRoom
ObstructedMoaze
Experiments
S4R3
S6R3
N7S8
N10S10
1Q
2Q
PCM only
1e-1
1e-1
1e-1
Te-1
1e-1
1e-1
PM prpronn
5e:1
5e-1
1e-1
le:1
1e
1e-1
JM pP
1o-1
5c-1
1o-1
1o-1
1o-1
1c-1
JCM
(permuted) re count
5e-1
1
5e-1
5e-1
Te-1
1e-1
IMM (permuted) + pauviet
5e-1
Figure 13: Best searched |3 values for experiments with 1CM
Environments
KeyCorrider
MultiRoom
ObstrucceedMaze
Experiments
S4R3
56R3
N758
N10510
10
2Q
only
1e-1
1e-1
1e-1
le-1
1e-1
1e-1
RUDD pfuntond
Se-1
5e-1
RIDE
pipviss
5e-1
Se-1
1e-1
1e-1
Se.2
Se-2
RIDE (permuled) yrP
1
Se-1
5e-1
16-1
1e-1
RIDE
(oermuted), pipsi
5e-1
5e-1
5e-1
5e-1
5e-2
Te-1
Figure 14: Best searched 3 values for experiments with RIDE
17

---

Published as a conference paper at ICLR 2023
Environments
KeyCoriior
MultiRoom
ObstructedMoaze
Experiments
S4133
S6R3
N758
NI0S10
1Q
2Q
MIND only
Te-5
1e-5
1e-5
1e-5
le-3
1e-5
5e-4
1e-3
5e-4
5e-4
1e:5
5e-5
RND pPP visit
5c-3
5c-3
5e-5
RND
(permuted) pfp_ocunt
1e-3
1o-3
1o-4
MNND
(permuted) pip.vist
5e-3
5e-3
5e-3
5e-3
1e-4
1e-4
Figure 15: Best searched /8 values for experiments with RND
Environments
KeyCorndor
MultiRoom
ObstructedMaaz
Experiments
S4R3
S6R3
N7S8
N10S10
1Q
2Q
B.Bsll only
5e-3
5e-3
5e-3
5e-3
5e-3
5e-3
BeBall
yop
1e-2
1e-2
5e-2
1e-2
1e-3
5e-3
Belill
1e-1
5e-2
5e-2
5e-2
5e-3
5e-3
(permuted) pfp_rtuus
fe-1
(permuted)
1e-1
1e-1
1e-1
Te-1
Se-3
1e-2
Figure 16: Best searched 3 values for experiments with BeBoold
Environments
KeyCorddo
MultRRom
ObstructedMaze
Experiments
S3R3
S4R3
S5R3
S6R3
N7S8
M054
MOS10
N12S10
2Dth
10
2Q
pruuenn only
1e-3
1e-3
1e-3
1e-3
1e-3
1e-3
1e-3
1e-3
5e-5
5e-4
5e-4
crly
50-3
56-3
5e-3
5e-3
5e-3
56-5
50-5
Figure 17: Best searched |3 values for experiments with episodic intrinsic rewards only.
1.0
0.8
Average Return
0.6
5e-2
1e-2
5e-3
20
1e-3.
5e-4
1e-4
0.0
Frames (x107)
Figure 18: Performance of training with opavisit only KeyCorridors4n3, using different 8.
18

---

Published as a conference paper at ICLR 2023
A.6
ADDITIONAL RESULTS
A. FINAL TESTING PERFORMANCE OF TRAINED AGENTS
For completeness, we test the trained agent for 100 episodes and show the results in Tab. 2.
Table 2: Final testing performance of trained agents.
KeyCorridor
MultiRocm.
ObstructedWaze
S4R3
S6R3
N758
M10810
1Q
2Q
mucomm only
0.93::0.026
0.95=00034
0.68+000006
0.69::0.005
0.85+102241
0.71+003888
ppp
only
0.93:100020
0.95=0.026
0.68:10.001
0.69:10.009
0.83-10,280
0.66:10.391
ICM only
0.0+0..0
0.0:0..0
0.0:0..0
0.0=0.00
0.0++000
0.0+00,0
JCM PoLmm
0.93=0.020
0.94=0.046
0.67++0.008
0.69::0.003
0.84+:0228
0.73++0.351
JCM FP
0.93:10025
0.95=0.027
0.63:10.054
0.69110002
0.85+10.253
0.64110.40
RIDE only
0.0+00.0
0.0:00.0
0.0:+0.0
0.0=0.00
0.0:
0.0+00,0
RIDEE SP coun
0.93:100023
0.12
0.66:10.023
0.69:000006
0.86+:02241
0.611:0.417
.IIDE sp-visit
0.93+00002
0.95=00023
0.43++0.022
0.69
0.79-10.222
0,0:0.00
p RND only RND
0.0+00.0
0.0:00.0
0.0+0.00
0.0=0.0
0.0+0000
0,0+00.0
0.93100032
0.93
0.56:10.100
0.69
0.831102227
0.61110.408
RRND Sp.iii
0.92++0.028
0.93==0097
0.67::0.008
0.68
0,54+00420
0.72+:0362
ReBold only
0.0+10.0
0.010..0
0.0+1..0
0.0-0.00
0.0:10.0
0.0:10.0
DeBellld pP count
0,92:000068
0.93=0.047
0,68+00009
0.69
0.0+0.00
0.0+0000
BeRold
0.93+000024
0.94=0.033
0.66+0.022
0.69::0.015
0.73+50.309
0.60:+0.395
A.6.2 COMPARISON TO STATE-OF-THE-ART METHODS OTHER ENVIRONMENTS MINIGRID
In addition to the comparison in Sec. 3.3, we provide results on following environments: KeyCorridors3re
MultiRoomNI 054, Obst wedddaae2Diii The results in Fig. 19 further demonstrates that pop count or y visit can match or surpass other state-of-the-an methods.
DeyCorridors555
2510 O,
KeyComdor-SS3B KeyCorridor-S5i
MuttiRoom-N104
Mult/Room-N12510
ObstructedMaze-2D0
Average Return
04
0.2
0.3
0.0
05
0.
4.
0.0
0.5
10
15
20
0.0
0.5
1.0
40
(xto7)
(x10²
(x107
(x107)
(x107
ep_count only
ep_visit only
AGAC
BeBold
RAPID
RIDE
Figure 19: Comparison of only using episodic intrinsic rewards and state-of-the-art methods on additional environments. Best viewed in color.
A.6.3 COMPARISON OF LIFELONG COUNT- BASED BONUS AND EPISODIC VISITATION BONUS
During the discussion period, we conduct experiments only using lifelong count-based intrinsic
1
rewards
(instead of episodic ones). The results are shown in Fig. 20. We can see the
(afe(St41)
performance is close to that of only using sp.visi One possible reason is that lifelong counts behave quite similarly to episodic counts since every episode is new in procedurally seredurrlly--geerratedd environments. It is also possible that the performance difference between episodic and lifelong intrinsic rewards on MiniGrid actually results from mooutt-based intrinsic rewards" v.s. "prediction- intrinsic rewards".
19

---

Published as a conference paper at ICLR 2023
keyCorrdor-S4RS
KeyCorridon-SRRS
MuttRoom-N7ss
Mutifoom-Niosio
ObstructedMoze-1Q
ObstructedMoze-22
10
Average Return
0.6
00
0.0
10
20
50 (x10)
4.
0.0
20
4.0
6.0
0.0
0.5
10
15
20
0.
05
10
15
20
00
20
4.0
6.
0.0
30
60
80
(x107)
(x10²)
(x10²)
(x1o²)
ep_visit
lifelong count
Figure 20: Comparison to lifelong count-based bonus
6 hard exploration environ-
(icf((s+11
ments in MiniGrid, averaged from 5 runs. Best viewed in color.
A.6.4 MAIN RESULTS AVERAGED FROM 10 RUNS
During the discussion period, we conduct the main experiments (Fig. 2) with additional runs and plot the results Fig. 21.
KeyContdor-54RS
KnyCoriidor-See MultHooomNNSS
Multiccom-Nias00
ObstructeddMaze-10
ObstructedMazz-20
08
ICM
0-
02
RIDE
08
RND
BeBold
04
02
60
0.5 10
15
Framnss (x10²)
Fromes (x10²)
Fromes (x107)
Framess (xIO²)
Frames (x107)
Fromes (x10²)
60_count lifelong
60_visit lifelong
lifelong only
only
ep_vist only
Intrinsic reward
Figure 21: Performance of different combinations of lifelong and episodic intrinsic rewards hard exploration environments in MiniGrid, averaged from 10 runs. Note that simply using episodic intrinsic rewards can achieve top performance among all combinations. Best viewed in color.
20


Reviews:
[{"review_id": "yQI9TnAipY", "reviewer": "Reviewer_pHUd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the how two types of curiosity, lifelong and episodic, contribute to performance and exploration when training an agent using PPO within three different hard exploration tasks from the MiniGrid evironment: KeyCorridor, MultiRoom, and ObstructedMaze. The authors claim that previous work in this setting typically involve designing for lifelong curiosity and leaving episodic curiosity as a minor complement. The authors proceed to show that the contribution of lifelong curiosity in minimal or non-existent when combined with episodic curiosity and as such does not improve performance of the agents within these tasks.  The authors then continue to demonstrate that the reason for this is likely due to lifelong curiosity not assigning higher value to novel states.", "summary_of_the_review": "Overall, I would recommend acceptance of this paper. I found the work to be a good empirical study with results that have the potential to shape future work in the space of lifelong curiosity or other intrinsic reward design. I do think that it would do well for the authors to make clear that rocedurally generated actually maps to hard exploration task, since the former is a bit vague in definition.", "strength_and_weaknesses": "Strengths:\n* The authors seem to have done a good job of ablating curiosity configurations from recent work in this space, really trying to tease apart the contributions of both lifelong and episodic curiosity used in this prior work, particularly with respect to showing all combinations of configurations across all of the tasks they evaluated with.\n* The authors do a good job of obtaining original source code from previous works, helping to eliminate potential sources of variation within their study.\n* The authors presented their work in a clear and concise way that was easy to read and understand.\n\nWeaknesses: \n* The authors tend to use procedurally generated as a proxy for hard exploration task. I think this can cause some confusion with respect to their claims. I would prefer them to either be more explicit that their discovery does not necessarily apply to all procedurally generated environments, and rather to hard exploration tasks.\n* I am a bit concerned that it appears as though there is little to no variance in the performance of some of the runs, even over 5 seeds. Perhaps I am missing something in these graphs?\n* I would have liked to see more than 5 seeds run for a paper where the main contribution is empirical results.\n* I would like to have seen mentioned or analysis the sensitivity to hyper parameter selection from the various tasks they trained on, even when those selections came from other works.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xe1kEmkuki", "reviewer": "Reviewer_GnK2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In some papers using intrinsic reward methods, the intrinsic reward is composed of two components: lifelong intrinsic reward (where the \"novelty\" of each state is tracked across multiple episodes) and episodic intrinsic reward (where the \"novelty\" of each state is tracked only across the current episode). The core idea of this paper is that, within the MiniGrid benchmark task, lifelong intrinsic rewards contribute little to agent performance, while episodic intrinsic rewards provide the bulk of performance improvements. The authors provide evidence for these differences in contributions via a battery of experiments on MiniGrid, replicating and ablating those components of several recent intrinsic reward proposals.", "summary_of_the_review": "I am leaning to reject this paper because it is missing too many details, to the point that it is somewhat difficult to follow and assess. I think that the primary message of the paper is sufficiently well-supported for acceptance, but the paper as a whole is not in acceptable form. I recommend the authors clear up the errors and missing details in the paper to bring it to the acceptance threshold.", "strength_and_weaknesses": "This paper is well-organized. The experiments seem to provide a reasonable demonstration of the key point: in MiniGrid, episodic intrinsic reward bonuses appear provide more benefit in terms of extrinsic performance and exploration coverage than lifelong intrinsic reward bonuses.\n\nHowever, the descriptions of the experiments and results are not completely clear and seem to be missing some key details (as detailed in my answer on Clarity and Reproducibility). \n\nThere are a few errors/inaccuracies that don't seem to form a larger pattern, but should be addressed in revision of this paper:\n1. \"This demonstrates that the learned lifelong curiosity does not accurately assign high values to novel states and low values to states the agent is familiar with.\" (p. 7) ← I don't think this conclusion logically follows from the evidence provided.\n2. \"A possible reason could be that the learned lifelong curiosity is not able to accurately reflect the novelty of states, making the agent receive higher cumulative (intrinsic) rewards from the opening-closing behavior than from exploring new rooms.\" (p. 7) ← I found this statement concerning, because it appears to reflect either a misunderstanding about intrinsic reward methods or a miscommunication where I am misunderstanding what the authors are trying to say. Whether an agent would receive higher rewards by exploring new rooms than opening and closing the door is sort of beside the point: the agent has never experienced the rooms it has never visited before, so it doesn't have any estimate for the value of new rooms, so there is nothing to direct its behaviour towards those rooms. The only thing that matters is whether the opening-closing states have higher value than the states the agent nearby that the agent has visited before. This is a known problem with exploration reward bonuses. This is related to what Shyam et al. (2019) refer to as reactivity or what Ecoffet et al. (2021) refer to as detachment.  Intrinsic rewards only influence behaviour with respect to previously visited states, so methods that rely on them also have to rely on stochastic behaviour or careful search control on exploration frontiers.\n\nSome typos, grammar, and concision suggestions (just to help with editing, no need to respond to these):\n- \"environments draws increasing\" (p. 1) → \"environments has drawn increasing\"\n- \"This finding would be inspiring for\" (p. 2) → consider something like \"This finding should inspire\"\n- \"to refer to the corresponding reward as a whole rather than the reward at timestep t.\" (p. 2) ← This choice may leave open some confusion, given that r is also defined as the reward function itself in Section 2.1.\n- \"Raileanu et al. (Raileanu & Rocktäschel, 2020)\" (p. 3, appears in both RIDE and Episodic visitation count sections) → \"Raileanu & Rocktäschel (2020)\"\n- \"Zhang et al. (Zhang et al., 2020)\" (p. 3, appears in both BeBold and Episodic first visits sections) → \"Zhang et al. (2020)\" \n- \"encourages the agent to explore beyond the boundary of explored regions.\" (p. 3) → without all of the intuition provided by Zhang et al., it's not very clear what is means to explore beyond the boundary of explored regions, so you might do better to provide intuition about the structure of the reward.\n- Once the lifelong component of RIDE is removed, the episodic visitation count bonus reduces to an MBIE-EB-style bonus, which is quite well-studied, so it would be good to include a reference to Strehl & Littman (2008) to help the reader find more information.\n- \"sparse extrinsic reward on accomplishing the task and pure exploration without extrinsic reward (Section 3.2).\" (p. 4) ← This phrase is quite difficult to parse, so I would recommend splitting it up into multiple parts, something like \"(a) a sparse intrinsic reward setting where reward is only obtained upon accomplishing the task, and (b) a pure exploration setting without extrinsic rewards.\"\n- \"door in order to enter next room.\" (p. 4) --> \"door to enter the next room.\" While \"in order\" is not incorrect, it lengthens the sentence unnecessarily.\n- \"grid as observation and choose from 7 actions\" --> \"grid as the observation and chooses from 7 actions\" (p. 4)\n- It would be helpful to have some explanation of what the non-obvious 'done' action does. (p. 4)\n- \"The agent obtains a non-zero reward\" (p. 4) ← \"In our sparse reward setting, the agent obtains a non-zero reward\" because I assume that's not true in the pure exploration setting of your experiments.\n- \"deferred to Appendix.\" --> \"deferred to Appendix A.1.\" (p. 4)\n- \"are included in Appendix.\" → \"are included in Appendix A.4.\" (p. 4)\n- \"(see Eqn. equation 2).\" → \"(see Equation 2).\" (p. 5)\n- \"have large impact\" → \"have a large impact\" (p. 3, p. 5)\n- \"Detailed configurations about hyperparameters are included in Appendix.\" (p. 5) ← I'm not sure what configurations about hyperparameters are; it might be more useful to say that a detailed explanation of how the hyperparameter search was performed and demonstrations of sensitivity to different values of beta can be found in Appendix A.5.\n- \"Here “None” refers to not using lifelong or episodic curiosity. If both are not used, then it reduces to vanilla PPO without intrinsic rewards.\" (p. 5) ← And if only one is used, I assume the other reduces to 1, because they are combined multiplicatively?\n- \"chieves\" (p. 5) → \"achieves\"\n- \"when agents reaching the goal.\" → \"when agents reach the goal\" or \"when an agent reaches the goal\" (p. 5)\n- \"the agent starts from the first room\" (p. 5) ← Is there some way of picking out the first room besides it being where the agent starts? This statement seems tautological.\n- In the caption to Figure 4, it would be great to reiterate \"the performance of BeBold in ObstructedMaze-2Q is directly taken from their paper but we cannot reproduce it,\" because it sticks out as an outlier amongst your results. \n- \"recent proposed ones\" (p. 6) → \"recently proposed ones\"\n- \"Such gap\" (p. 6) → \"This difference in exploration efficiency\"\n- \"included in Appendix\" (p. 7) → \"included in Appendix A.4\"\n- \"In Appendix, we also include the comparison results on other environments in MiniGrid benchmark.\" (p. 7) → \"In Appendix A.6, we also include the comparison results on other environments in MiniGrid.\" \n- \"3.4 Why lifelong curiosity does not help?\" (p. 7) → \"3.4 Why doesn't lifelong curiosity help?\"\n- \"as shown in Figure 6 is\" (p. 7) → \"as shown in Figure 6, is\"\n- \"Bebold\" (p. 12) → \"BeBold\"\n- \"comarison\" (p. 19) → \"comparison\"\n\nEcoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021). First return, then explore. Nature, 590(7847), 580-586.\n\nShyam, P., Jaśkowski, W., & Gomez, F. (2019, May). Model-based active exploration. In International Conference on Machine Learning (pp. 5779-5788). PMLR.\n\nStrehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309 – 1331.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WkMYyrpuc2s", "reviewer": "Reviewer_3Vou", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper evaluates several intrinsic rewards (lifelong and episodic) on some of the difficult Minigrid environments.\nThe results are that simple episodic rewards are better than the established lifelong once in these environments.\n", "summary_of_the_review": "The paper is interesting and studies a relevant aspect of RL.\nHowever, I have the suspicion that the paper is flawed and that the results are due to the lack of memory in the architectures. \n\nPost-rebuttal:\nThanks for the clarification, see my answer below. I increased my score to 5. ", "strength_and_weaknesses": "Strengths:\n - relevant survey \n - several ablations and analysis performed \n \nWeaknesses:\n - the policy and value function seem to be non-recurrent. The Minigrid is a POMDP environment, so some form of memory is required. This might explain why episodic returns are so successful here.\nThe RND reward, for instance, should explicitly make the policy find all states in the state-space. The fact that it does not, shows that the policy cannot distinguish the states (or there is a bug somewhere else).\n - The statements are too strong: 1. There are only a few fixed environments used (really procedurally-generated is e.g. in procgen where every level is different)\n - The ablation with the randomization of the lifelong reward hints to me that there is either a bug or that, without memory, there is no to determine the reward properly from the observation alone. \n \nDetails:\n- Intro p1: \"gradually discourages\": from my understanding the intrinsic rewards are positive, so they are not discouraging but instead are encouraging the visitation of certain states (as long as \"intrinsically interesting\") and then the reward for these states goes eventually to 0 (no extra incentive to go there)\n- contribution 3: this finding would be ... why conjuctive?\n- EQ 8: the denominator needs an additional constant to be not 1/0 (or is it post-factum computed, such that all counts are at least 1?)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DvAOGxsuAf", "reviewer": "Reviewer_e8Yz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts a thorough experimental study of episodic and lifelong curiosity bonuses for procedurally-generated (PG) environments. While several prior works have proposed different lifelong novelty bonuses with various motivations (RIDE, BeBold/NovelD, AGAC), and include episodic bonuses as a detail, this work instead shows that the episodic bonus is the main driver of performance, and that the lifelong novelty bonuses have little effect. This is an interesting insight which suggests rethinking exploration in procgen environments is needed. ", "summary_of_the_review": "Overall, I recommend this paper for acceptance because I believe the paper provides important insights for exploration in procgen environments. While there has been lots of work on exploration in singleton MDPs in the past, there has been increasing interest in procgen environments (and contextual MDPs more generally), and this paper shows that fundamentally different kinds of bonuses are needed for this setting (i.e. episodic rather than lifelong). For this reason, I think it's important to make the insights of this paper more widely known. \n\nThe paper does have some things that need fixing in the writing (as per my comments above), and a few changes experiment wise (i.e. rerunning BeBold experiments with offiical codebase which is now released), but I think these are relatively minor and can be done for the camera ready. ", "strength_and_weaknesses": "Strengths:\n- The paper's experimental study is very thorough. They examine a large number of combinations of episodic and lifelong novelty bonuses, and tune the intrinsic reward coefficient well, which can have a large impact on performance. They show that when this is well-tuned, the episodic bonus alone is sufficient to get state of the art performance. \n- Although the paper does not propose a novel method, the main insight is novel and in my opinion high impact: it shows that to some extent, we have been thinking about exploration in procgen environments wrong, that lifelong novelty bonuses used for singleton RL environments do not automatically transfer over to procgen settings, and that episodic bonuses are more promising. \n\nWeaknesses:\n- While the paper shows that episodic bonuses are indeed helpful, the episodic bonuses it considers are all count-based which will only work in fairly simple settings like MiniGrid where there are not many distractors (which would break count-based bonuses). So although the paper does give compelling evidence that episodic bonuses deserve more investigation, it doesn't suggest a good or general solution based on episodic bonuses. \n\n\nAs a side note, this paper was posted after the ICLR submission deadline so does not affect the novelty of the present paper, but the authors might be interested in it since it is quite related: https://arxiv.org/abs/2210.05805. They also find that episodic bonuses are very important and propose a way generalize beyond count-based episodic bonuses. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CIMNEV930b", "reviewer": "Reviewer_bLVL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes various intrinsic reward generation algorithms in the MiniGrid benchmark and presents that episodic curiosity is helpful for training while lifelong curiosity is not.\n\nThe authors compare four different curiosity algorithms (ICM, RIDE, RND, BeBold) with and without two episodic curiosity (episodic visitation count from RIDE and episodic first visits from BeBold). The experimental results show that episodic curiosity-only models generally have the best performance while models with the lifelong curiosity-only have the worst performance and models using both curiosities have similar or worse performance than the lifelong curiosity-only model.\n\nThe authors supposed that this might be because lifelong curiosity cannot capture the correct novelty of the state presenting that randomly shuffled reward within each batch did not hugely deter the performance.\n", "summary_of_the_review": "Although analyzing the effectiveness of lifelong curiosity and episodic curiosity is interesting, as mentioned in the weakness section, not compared with [Ref. 1] and the method is only applicable to discrete environments, not the general procedurally-generated environments, I gave Reject as the result of the initial review.", "strength_and_weaknesses": "**[Strength]**\n\n1. The paper is well written and easy to understand, especially the difference between each existing method.\n\n2. Moreover, the authors presented that a simple count-based method in each episode is already enough for training MiniGrid.\n\n\n**[Weakness]**\n\n1. The authors omitted a highly relevant paper [Ref 1]. The author should compare with [Ref 1] in terms of performance and conceptual difference. Moreover, It could be the answer to the first question in the Discussion section.\n\n2. As already mentioned in the Discussion section, the authors only used MiniGrid which is a discrete and almost static environment. It seems that the reason why episodic curiosity was effective is that the action and the position of agents are discrete and the number of possible positions is relatively small compared to other environments such as ProcGen (Cobbe 2020). Consequently, explicitly using the state visitation information is effective and not surprising. In other words, from the current version of the paper, it is difficult to the convenience that episodic curiosity is really effective in other procedurally generated environments.\n\n\n[Ref. 1] Savinov, Nikolay, et al. \"Episodic curiosity through reachability.\" ICLR. 2019.\n\n\n**[Minor]**\n\nOn page 7, the author mentioned that the official implementation of BeBold is not open-sourced. However, the first author of that paper released the code in November 2021 (tianjunz/NovelD).\n\n\"Leveraging procedural generation to benchmark reinforcement learning\" has two BibTeX entries (Cobbe 2019, Cobbe 2020). Please merge them together.\n\nIn README.md, \"NeurIPS 2021\" -> \"ICLR 2023\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster