Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores the how two types of curiosity, lifelong and episodic, contribute to performance and exploration when training an agent using PPO within three different hard exploration tasks from the MiniGrid evironment: KeyCorridor, MultiRoom, and ObstructedMaze. The authors claim that previous work in this setting typically involve designing for lifelong curiosity and leaving episodic curiosity as a minor complement. The authors proceed to show that the contribution of lifelong curiosity in minimal or non-existent when combined with episodic curiosity and as such does not improve performance of the agents within these tasks.  The authors then continue to demonstrate that the reason for this is likely due to lifelong curiosity not assigning higher value to novel states.
strength_and_weaknesses: Strengths:
* The authors seem to have done a good job of ablating curiosity configurations from recent work in this space, really trying to tease apart the contributions of both lifelong and episodic curiosity used in this prior work, particularly with respect to showing all combinations of configurations across all of the tasks they evaluated with.
* The authors do a good job of obtaining original source code from previous works, helping to eliminate potential sources of variation within their study.
* The authors presented their work in a clear and concise way that was easy to read and understand.

Weaknesses: 
* The authors tend to use procedurally generated as a proxy for hard exploration task. I think this can cause some confusion with respect to their claims. I would prefer them to either be more explicit that their discovery does not necessarily apply to all procedurally generated environments, and rather to hard exploration tasks.
* I am a bit concerned that it appears as though there is little to no variance in the performance of some of the runs, even over 5 seeds. Perhaps I am missing something in these graphs?
* I would have liked to see more than 5 seeds run for a paper where the main contribution is empirical results.
* I would like to have seen mentioned or analysis the sensitivity to hyper parameter selection from the various tasks they trained on, even when those selections came from other works.

clarity,_quality,_novelty_and_reproducibility: The paper was very well written. Each section was presented in a clear and concise way. The authors did a good job of getting original source code from prior work, making the reproducibility of their work very high. Hyperparameter selections were well documented as well as the experimental procedure. The results appear to show a novel discovery with respect to the effects of lifelong and episodic curiosity within MiniGrid.
Minor typo in Section 3.2, the word "achieves" is misspelled in the sentence "For the goal-reaching task, the extrinsic reward ..."
summary_of_the_review: Overall, I would recommend acceptance of this paper. I found the work to be a good empirical study with results that have the potential to shape future work in the space of lifelong curiosity or other intrinsic reward design. I do think that it would do well for the authors to make clear that rocedurally generated actually maps to hard exploration task, since the former is a bit vague in definition.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper evaluates several intrinsic rewards (lifelong and episodic) on some of the difficult Minigrid environments.
The results are that simple episodic rewards are better than the established lifelong once in these environments.

strength_and_weaknesses: Strengths:
 - relevant survey 
 - several ablations and analysis performed 
 
Weaknesses:
 - the policy and value function seem to be non-recurrent. The Minigrid is a POMDP environment, so some form of memory is required. This might explain why episodic returns are so successful here.
The RND reward, for instance, should explicitly make the policy find all states in the state-space. The fact that it does not, shows that the policy cannot distinguish the states (or there is a bug somewhere else).
 - The statements are too strong: 1. There are only a few fixed environments used (really procedurally-generated is e.g. in procgen where every level is different)
 - The ablation with the randomization of the lifelong reward hints to me that there is either a bug or that, without memory, there is no to determine the reward properly from the observation alone. 
 
Details:
- Intro p1: "gradually discourages": from my understanding the intrinsic rewards are positive, so they are not discouraging but instead are encouraging the visitation of certain states (as long as "intrinsically interesting") and then the reward for these states goes eventually to 0 (no extra incentive to go there)
- contribution 3: this finding would be ... why conjuctive?
- EQ 8: the denominator needs an additional constant to be not 1/0 (or is it post-factum computed, such that all counts are at least 1?)

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. The graphics are to the point, nice and clean. Code and details are provided.
summary_of_the_review: The paper is interesting and studies a relevant aspect of RL.
However, I have the suspicion that the paper is flawed and that the results are due to the lack of memory in the architectures. 

Post-rebuttal:
Thanks for the clarification, see my answer below. I increased my score to 5.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In some papers using intrinsic reward methods, the intrinsic reward is composed of two components: lifelong intrinsic reward (where the "novelty" of each state is tracked across multiple episodes) and episodic intrinsic reward (where the "novelty" of each state is tracked only across the current episode). The core idea of this paper is that, within the MiniGrid benchmark task, lifelong intrinsic rewards contribute little to agent performance, while episodic intrinsic rewards provide the bulk of performance improvements. The authors provide evidence for these differences in contributions via a battery of experiments on MiniGrid, replicating and ablating those components of several recent intrinsic reward proposals.
strength_and_weaknesses: This paper is well-organized. The experiments seem to provide a reasonable demonstration of the key point: in MiniGrid, episodic intrinsic reward bonuses appear provide more benefit in terms of extrinsic performance and exploration coverage than lifelong intrinsic reward bonuses.

However, the descriptions of the experiments and results are not completely clear and seem to be missing some key details (as detailed in my answer on Clarity and Reproducibility). 

There are a few errors/inaccuracies that don't seem to form a larger pattern, but should be addressed in revision of this paper:
1. "This demonstrates that the learned lifelong curiosity does not accurately assign high values to novel states and low values to states the agent is familiar with." (p. 7) ← I don't think this conclusion logically follows from the evidence provided.
2. "A possible reason could be that the learned lifelong curiosity is not able to accurately reflect the novelty of states, making the agent receive higher cumulative (intrinsic) rewards from the opening-closing behavior than from exploring new rooms." (p. 7) ← I found this statement concerning, because it appears to reflect either a misunderstanding about intrinsic reward methods or a miscommunication where I am misunderstanding what the authors are trying to say. Whether an agent would receive higher rewards by exploring new rooms than opening and closing the door is sort of beside the point: the agent has never experienced the rooms it has never visited before, so it doesn't have any estimate for the value of new rooms, so there is nothing to direct its behaviour towards those rooms. The only thing that matters is whether the opening-closing states have higher value than the states the agent nearby that the agent has visited before. This is a known problem with exploration reward bonuses. This is related to what Shyam et al. (2019) refer to as reactivity or what Ecoffet et al. (2021) refer to as detachment.  Intrinsic rewards only influence behaviour with respect to previously visited states, so methods that rely on them also have to rely on stochastic behaviour or careful search control on exploration frontiers.

Some typos, grammar, and concision suggestions (just to help with editing, no need to respond to these):
- "environments draws increasing" (p. 1) → "environments has drawn increasing"
- "This finding would be inspiring for" (p. 2) → consider something like "This finding should inspire"
- "to refer to the corresponding reward as a whole rather than the reward at timestep t." (p. 2) ← This choice may leave open some confusion, given that r is also defined as the reward function itself in Section 2.1.
- "Raileanu et al. (Raileanu & Rocktäschel, 2020)" (p. 3, appears in both RIDE and Episodic visitation count sections) → "Raileanu & Rocktäschel (2020)"
- "Zhang et al. (Zhang et al., 2020)" (p. 3, appears in both BeBold and Episodic first visits sections) → "Zhang et al. (2020)" 
- "encourages the agent to explore beyond the boundary of explored regions." (p. 3) → without all of the intuition provided by Zhang et al., it's not very clear what is means to explore beyond the boundary of explored regions, so you might do better to provide intuition about the structure of the reward.
- Once the lifelong component of RIDE is removed, the episodic visitation count bonus reduces to an MBIE-EB-style bonus, which is quite well-studied, so it would be good to include a reference to Strehl & Littman (2008) to help the reader find more information.
- "sparse extrinsic reward on accomplishing the task and pure exploration without extrinsic reward (Section 3.2)." (p. 4) ← This phrase is quite difficult to parse, so I would recommend splitting it up into multiple parts, something like "(a) a sparse intrinsic reward setting where reward is only obtained upon accomplishing the task, and (b) a pure exploration setting without extrinsic rewards."
- "door in order to enter next room." (p. 4) --> "door to enter the next room." While "in order" is not incorrect, it lengthens the sentence unnecessarily.
- "grid as observation and choose from 7 actions" --> "grid as the observation and chooses from 7 actions" (p. 4)
- It would be helpful to have some explanation of what the non-obvious 'done' action does. (p. 4)
- "The agent obtains a non-zero reward" (p. 4) ← "In our sparse reward setting, the agent obtains a non-zero reward" because I assume that's not true in the pure exploration setting of your experiments.
- "deferred to Appendix." --> "deferred to Appendix A.1." (p. 4)
- "are included in Appendix." → "are included in Appendix A.4." (p. 4)
- "(see Eqn. equation 2)." → "(see Equation 2)." (p. 5)
- "have large impact" → "have a large impact" (p. 3, p. 5)
- "Detailed configurations about hyperparameters are included in Appendix." (p. 5) ← I'm not sure what configurations about hyperparameters are; it might be more useful to say that a detailed explanation of how the hyperparameter search was performed and demonstrations of sensitivity to different values of beta can be found in Appendix A.5.
- "Here “None” refers to not using lifelong or episodic curiosity. If both are not used, then it reduces to vanilla PPO without intrinsic rewards." (p. 5) ← And if only one is used, I assume the other reduces to 1, because they are combined multiplicatively?
- "chieves" (p. 5) → "achieves"
- "when agents reaching the goal." → "when agents reach the goal" or "when an agent reaches the goal" (p. 5)
- "the agent starts from the first room" (p. 5) ← Is there some way of picking out the first room besides it being where the agent starts? This statement seems tautological.
- In the caption to Figure 4, it would be great to reiterate "the performance of BeBold in ObstructedMaze-2Q is directly taken from their paper but we cannot reproduce it," because it sticks out as an outlier amongst your results. 
- "recent proposed ones" (p. 6) → "recently proposed ones"
- "Such gap" (p. 6) → "This difference in exploration efficiency"
- "included in Appendix" (p. 7) → "included in Appendix A.4"
- "In Appendix, we also include the comparison results on other environments in MiniGrid benchmark." (p. 7) → "In Appendix A.6, we also include the comparison results on other environments in MiniGrid." 
- "3.4 Why lifelong curiosity does not help?" (p. 7) → "3.4 Why doesn't lifelong curiosity help?"
- "as shown in Figure 6 is" (p. 7) → "as shown in Figure 6, is"
- "Bebold" (p. 12) → "BeBold"
- "comarison" (p. 19) → "comparison"

Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021). First return, then explore. Nature, 590(7847), 580-586.

Shyam, P., Jaśkowski, W., & Gomez, F. (2019, May). Model-based active exploration. In International Conference on Machine Learning (pp. 5779-5788). PMLR.

Strehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for Markov decision processes. Journal of Computer and System Sciences, 74(8):1309 – 1331.
clarity,_quality,_novelty_and_reproducibility: Novelty: My understanding is that each episode is a new layout of the task. In this way, each episode presents a new exploration problem to tackle: the keys, doors, obstructions, and goals have all moved around. To be successful, the agent needs to explore thoroughly once again in every episode and cannot rely much on what it has previously learned about the value of each state. For this reason, it does not seem especially novel to note that an intrinsic reward that encourages complete re-exploration of the environment in each episode would perform better than an intrinsic reward that assumes that parts of the world that the agent visited in previous episodes are no longer novel enough to be worth exploring—the agent needs to re-explore in each episode!

That said, as the authors of this paper have noted, prior work, even those working with MiniGrid, has sometimes focused on these lifelong intrinsic reward designs. This suggests that the core idea of this work is not completely obvious, so seems worth sharing. 

Clarity: Much of the paper is quite clear, but there are some areas where the writing is vague or nonspecific on details to the point that the paper becomes difficult to follow.
- As you've written the section on ICM (p. 3), it sounds like the inverse model has no effect on the ICM intrinsic reward, so it sounds like extraneous information. It should be made clear how the inverse model plays into the ICM intrinsic reward.
- "is too time-consuming" (p. 4) ← Do you mean in terms of ease of implementation, training time, or something else?
- "curiosity about the states" (p. 2) ← given that the intrinsic reward is computed from a transition tuple, does the intrinsic reward sometimes capture something about actions too (not just states)?

Reproducibility: Section 3.4 is especially problematic for reproducibility because it lacks many details that are important for understanding, nevermind attempts to replicate the experiments. 
1. "after training it" (p. 7) ← How much training has the agent received? Have you frozen the intrinsic reward or the value function, or is the agent still learning?
2. "randomly permuting the lifelong curiosity within a batch" (p. 2), "we randomly permute the lifelong curiosity within a batch" (p. 7) ← It is unclear how this permutation is performed.
3. "We first generate trajectories that accomplish the tasks and cover all rooms." (p. 7) ← How are these trajectories generated?

There are also a number of other details missing:
1. "Surprisingly, we find that only using episodic curiosity, the trained agent can match or surpass the performance of the one trained by using lifelong-episodic combination; in contrast, only using lifelong curiosity makes little progress in exploration." (p. 1) ← You refer to "performance" generally here, but at this point you have also implied two different performance metrics: one being extrinsic return and the other being some kind of exploration metric when no extrinsic reward is provided to the agent. Can you be a little more explicit? (The same general use of the word "performance" also continues onto the next page.)
2. "the learned lifelong curiosity assigns similar values to both novel and familiar states." (p. 2) You say "learned" here; is this based on a measurement after some number of episodes? How are novel and familiar defined here?
3. "When applicable" (p. 5) → Can you be more specific about when this statement is and is not applicable? Noting that some of the results are taken from other papers, not being explicit calls into question whether or not the comparisons in the figure are reasonable.
4. Please define what a run is. ("When applicable, the training curves are averaged over 5 runs and the standard deviations are plotted as shaded areas." p. 5) 
5. "the average number of explored rooms within an episode" (p. 5) Please define what an "explored room" is (does the agent have to visit every grid square in the room and toggle all the objects? Visit the doorway? Visit any grid square of the room? Can a room be partially explored or is this a discrete metric?
summary_of_the_review: I am leaning to reject this paper because it is missing too many details, to the point that it is somewhat difficult to follow and assess. I think that the primary message of the paper is sufficiently well-supported for acceptance, but the paper as a whole is not in acceptable form. I recommend the authors clear up the errors and missing details in the paper to bring it to the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper conducts a thorough experimental study of episodic and lifelong curiosity bonuses for procedurally-generated (PG) environments. While several prior works have proposed different lifelong novelty bonuses with various motivations (RIDE, BeBold/NovelD, AGAC), and include episodic bonuses as a detail, this work instead shows that the episodic bonus is the main driver of performance, and that the lifelong novelty bonuses have little effect. This is an interesting insight which suggests rethinking exploration in procgen environments is needed. 
strength_and_weaknesses: Strengths:
- The paper's experimental study is very thorough. They examine a large number of combinations of episodic and lifelong novelty bonuses, and tune the intrinsic reward coefficient well, which can have a large impact on performance. They show that when this is well-tuned, the episodic bonus alone is sufficient to get state of the art performance. 
- Although the paper does not propose a novel method, the main insight is novel and in my opinion high impact: it shows that to some extent, we have been thinking about exploration in procgen environments wrong, that lifelong novelty bonuses used for singleton RL environments do not automatically transfer over to procgen settings, and that episodic bonuses are more promising. 

Weaknesses:
- While the paper shows that episodic bonuses are indeed helpful, the episodic bonuses it considers are all count-based which will only work in fairly simple settings like MiniGrid where there are not many distractors (which would break count-based bonuses). So although the paper does give compelling evidence that episodic bonuses deserve more investigation, it doesn't suggest a good or general solution based on episodic bonuses. 


As a side note, this paper was posted after the ICLR submission deadline so does not affect the novelty of the present paper, but the authors might be interested in it since it is quite related: https://arxiv.org/abs/2210.05805. They also find that episodic bonuses are very important and propose a way generalize beyond count-based episodic bonuses. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
- Medium-high. The paper is very clearly written and easy to understand. The main message is simple, but in my opinion important. I have some comments below that would improve the clarity. 

Quality:
- Medium-high. The paper's experimental investigations are quite thorough, with many combinations of episodic and lifelong bonuses investigated. This shows that their conclusions are robust and not an artifact of just one setting. 

Novelty:
- Medium-high. The methods are not very novel, but the insights are. I think this is important. 

Reproducibility:
- High. The authors provide code to reproduce their experiments. 


## Detailed comments:


- Section 2.2: When describing RIDE, it says: "Raileanu et al propose a novel lifelong curiosity". RIDE is actually not really a lifelong curiosity bonus - it measures the distance between consecutive embeddings, which will not disappear with sufficient exploration the same way RND, NovelD or ICM would (ICM will only vanish if the env is deterministic though). I would fix this in the text. 
- Equation 7: NovelD/BeBold has a scaling constant in front of the second RND error, please add
- Section 3.1, in training details: it is mentioned that IMPALA is too slow and that PPO is faster, which is interesting since most works use IMPALA on MiniGrid. Please include a comparison of speed between the two, this could be useful to others. 
- Throughout the paper, there are references to the Appendix but not specific parts. Please organize the appendix into sections and include references to the different parts, like (Appendix A.1, Appendix B.3, etc so people can easily find which what you're referring to in the main text). 
- In Figures 2 and 3, which lifelong curiosity bonus is being used? Currently it only says "lifelong"
- In Section 3.3, it says the BeBold code is not open sourced, but it is now: https://github.com/tianjunz/NovelD. Although I'm guessing running the official code vs. the reimplemented code won't qualitatively change the results, please rerun using the official code for the camera ready. 
- In Section 3.3, is the same underlying RL algorithm being used? I.e. RIDE, BeBold/NovelD use IMPALA but the count-based implementation uses PPO it seems. Please make sure the same underlying RL algorithm is the same, since this can make a big difference in sample complexity (IMPALA is in general less sample efficient than PPO since it only uses each sample once). 
- In several places throughout the paper, there is both a \citet{} ref and a \citep{} ref - please only use one. 

Minor grammar/language comments:
- Abstract: "only using lifelong curiosity can hardly make progress" -> "...hardly makes progress..."
- Intro: "recent works...pay increasing attention to" -> "recent works have paid increasing attention to"
- "does not cause significant performance drop" -> "does not cause a significant performance drop"
- "our work makes following contributions" -> "our work makes the following contributions"
- Section 2.1: ". Next state observation" -> ". The next state observation"
- Section 3.2: "agent chieves" -> "agent achieves"
- Section 3.2: "By contrast" -> "In contrast"
- Title 3.4: "Why lifelong curiosity does not help?" -> either "Why lifelong curiosity does not help" or "Why does lifelong curiosity not help?"
- Section 5: "skeptical to overfitting" -> "susceptible to overfitting"
summary_of_the_review: Overall, I recommend this paper for acceptance because I believe the paper provides important insights for exploration in procgen environments. While there has been lots of work on exploration in singleton MDPs in the past, there has been increasing interest in procgen environments (and contextual MDPs more generally), and this paper shows that fundamentally different kinds of bonuses are needed for this setting (i.e. episodic rather than lifelong). For this reason, I think it's important to make the insights of this paper more widely known. 

The paper does have some things that need fixing in the writing (as per my comments above), and a few changes experiment wise (i.e. rerunning BeBold experiments with offiical codebase which is now released), but I think these are relatively minor and can be done for the camera ready.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper analyzes various intrinsic reward generation algorithms in the MiniGrid benchmark and presents that episodic curiosity is helpful for training while lifelong curiosity is not.

The authors compare four different curiosity algorithms (ICM, RIDE, RND, BeBold) with and without two episodic curiosity (episodic visitation count from RIDE and episodic first visits from BeBold). The experimental results show that episodic curiosity-only models generally have the best performance while models with the lifelong curiosity-only have the worst performance and models using both curiosities have similar or worse performance than the lifelong curiosity-only model.

The authors supposed that this might be because lifelong curiosity cannot capture the correct novelty of the state presenting that randomly shuffled reward within each batch did not hugely deter the performance.

strength_and_weaknesses: **[Strength]**

1. The paper is well written and easy to understand, especially the difference between each existing method.

2. Moreover, the authors presented that a simple count-based method in each episode is already enough for training MiniGrid.


**[Weakness]**

1. The authors omitted a highly relevant paper [Ref 1]. The author should compare with [Ref 1] in terms of performance and conceptual difference. Moreover, It could be the answer to the first question in the Discussion section.

2. As already mentioned in the Discussion section, the authors only used MiniGrid which is a discrete and almost static environment. It seems that the reason why episodic curiosity was effective is that the action and the position of agents are discrete and the number of possible positions is relatively small compared to other environments such as ProcGen (Cobbe 2020). Consequently, explicitly using the state visitation information is effective and not surprising. In other words, from the current version of the paper, it is difficult to the convenience that episodic curiosity is really effective in other procedurally generated environments.


[Ref. 1] Savinov, Nikolay, et al. "Episodic curiosity through reachability." ICLR. 2019.


**[Minor]**

On page 7, the author mentioned that the official implementation of BeBold is not open-sourced. However, the first author of that paper released the code in November 2021 (tianjunz/NovelD).

"Leveraging procedural generation to benchmark reinforcement learning" has two BibTeX entries (Cobbe 2019, Cobbe 2020). Please merge them together.

In README.md, "NeurIPS 2021" -> "ICLR 2023"
clarity,_quality,_novelty_and_reproducibility: The arguments of the paper are clearly stated with various experiments. The originality of the paper is weak since the paper just extracted episodic curiosity term from RIDE and BeBold and apply it to other curiosity models or alone.
 

summary_of_the_review: Although analyzing the effectiveness of lifelong curiosity and episodic curiosity is interesting, as mentioned in the weakness section, not compared with [Ref. 1] and the method is only applicable to discrete environments, not the general procedurally-generated environments, I gave Reject as the result of the initial review.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: When solving exploration problems procedurally generated environments the field often does so through a combination of two types of bonus that are multipled together: a lifelong one, which takes into consideration the agent’s experience throughout its entire lifetime, and an episodic one, which considers only the agent’s experience in that particular episode/problem instance.

This paper performs a thorough analysis of the impact of each of these bonuses (lifelong and episodic) in the agent’s performance, and it shows that, in procedurally generated MiniGrid environments, the types of lifelong bonuses they investigated have little to no effect on the agent’s performance, while the episodic bonuses they investigated explain pretty much all the gain in performance when using these bonuses. This paper is an interesting paper to the community of exploration in procedurally generated environments because it suggests the community should maybe rethink how exploration should be done.

Importantly, this paper led to a lot of discussion between the reviewers because while they all agree about its potential to shape how the community thinks about generalization, there were also several concerns about presentation and the claims being made. The authors did address several of these concerns during the discussion phase, but I’d encourage the authors to read all reviews carefully again and make sure other comments are addressed as well. Of particular note are the concerns around Section 3.4, such as the static policy rollout experiment, the permutation experiment, and Figure 5. The authors said they will remove Figure 5 and that they will make Section 3.4 much more speculative, and I recommend they do so, while carefully following the reviewers’ suggestions.

Personally, I am concerned about how the episodic bonus leverages so much domain knowledge while the lifelong one is very general. In that sense, it is not surprising that the episodic bonus carries so much weight. I would have found more interesting an experiment where the same function is used for the lifelong and episodic bonuses, allowing us to better understand if the observed phenomenon is due to the particular choice of bonuses chosen or indeed because a reset at each episode is the more effective way of doing so. 

Nevertheless, at the end we concluded that the benefits outweighs the downsides of the paper and as long as the paper addresses the reviewers concerns it will be sharing an important message to the PCG-exploration community.

justification_for_why_not_higher_score: The paper is interesting to a subcommunity of exploration in RL which focuses on procedurally-generated environments. The paper is not necessarily interesting for the RL community thinking about exploration outside these environments, so it is not something for a large audience and thus I don’t see the need for a spotlight/oral slot; not to mention it was a borderline paper.
justification_for_why_not_lower_score: The main message of the paper could make the community it is targeted to to rethink exploration. This is a good reason to accept the paper.
Decision: Accept: poster