Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigated the benefits of global vs episodic exploration bonuses, and provided interesting intuition as to when each would help learning and when it would not. Moreover, this paper studies setting of contextual MDPs, which is relatively understudied compared to single MDPs. It proposes a simple way of combining global and episodic bonuses that work nicely in the tasks that they investigate. 
strength_and_weaknesses: Strengths: this study provided nice toy tasks that gave good intuition about when global vs episodic exploration bonuses would be beneficial and when they would not. Their multiplicative combination solution at the end of the paper (combining the 2) was a simple solution that can be easily implemented. They hypothesized why it works (the relative scaling of gobal vs episodic bonuses over  episodes). Finally, the fact that they tried it both for count-based bonuses as well as the more scalable E3B bonus was commendable. 

Weaknesses: The tasks tried remain toyish and are limited in scope (confined to the minihack suite). While I really liked the demonstrations they provided, and think that minihack is a good start, it would have been nice to see at least a hint of either 1) how global vs episodic exploration bonuses behave in a slightly more scaled up task (e.g. procgen or anything with nontrivial pixels), or 2) how their multiplicative solution behaves in such a scaled up task and whether it still works. 

Finally, their investigation of their multiplicative solution ends on a hypothesis: that it works because of the relative scaling of global vs episodic bonuses over  episodes. It would have been great if they could provide some more comprehensive empirical evidence for this hypothesis. The question of how to combine global & episodic exploration bonuses is a good question (and more generally, understanding  the effects of exploration durign learning/generalization is a very important question in RL). The authors did a cool experiment investigating multiplicative vs additive combination.  It is a very simple manipulation, but simple manipulations often have great merit. They should have dissected this experiment  for a more comprehensive understanding of this result. 
clarity,_quality,_novelty_and_reproducibility: The paper was clear and well written, on in a setting (contextual MDPs) that is important yet understudied, and it provides novel results concerning the use of exploration bonuses. I have not seen other investigations on combining global & episodic explorations besides this one, which is cool. However,  their investigations on combining exploration bonuses in this paper is a little scant even though it is one of their main novelties. This can be improved. 
summary_of_the_review: This paper was an interesting and easy read and contains useful and novel results and intuitions for the community concerning the use of exploration bonuses, especially their combination. However, as it stands, there is not yet a hint of how their study would behave in slightly more scaled up tasks, and their analysis on combining exploration bonuses is a little too scant. If these factors can be rectified, I would definitely move my score up on what I think is an interesting paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of exploration in RL. This has long been a challenging problem in RL, the paper talks about the episodic novelty and global novelty in the exploration. The paper studies the two novelty and draws the conclusion that they are effective in different settings. Episodic novelty is most effective when there are structures shared while global novelty is most effective when there are fewer structures shared. The paper conducts experiments on 18 MiniHack tasks. 
strength_and_weaknesses: Strength: 
1. The paper tackles the problem of episodic and global novelty, which is a very interesting setting. 
2. The paper is well written. 
3. The baselines are compared thoroughly. 
4. The experiments is conducted over 18 tasks.

Weakness: 
1. I feel the paper may dismiss some baselines like Never Give Up, MADE. 
2. I feel like the experiments conducted are toy and it is hard for me to judge whether this is a valuable contribution in general. I would suggest conducting more experiments in different domains to support the conclusion. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
summary_of_the_review: Overall I appreciate the authors' effort in tackling the exploration problem. However, it is a little hard for me to judge whether the conclusion can be drawn only from the MiniHack experiments. Please see the strengths and weaknesses above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors evaluate the relation between episodic and global novelty bonus with regard to contextual MDPs. They show two simplified examples for demonstrating the different advantages of the two novelty bonus types. Additionally, they explore 6 different combinations of the episodic E3B bonus with global novelty bonus definitions, proposed in the literature. The methods are evaluated on 18 MiniHack domains.
strength_and_weaknesses: The authors try to tackle an interesting problem in a clear to understand way. The evaluation is reasonable diverse, but could be improvement by adding different domains (not just variants of MiniHack). However, the paper suffers from flaws in the reasoning, combined with a lack of a crucial detail (the used feature space, see next section). The claims from Sections 3.1 do only hold under the used state feature spaces, because it does not generalize due to the context dependent value of the feature space. It is likely, that a context independent feature space - like the message features from Section 3.2 - will not be affected by the mentioned issue. This is also supported by Figure 5b, showing competitive results for the episodic variant. Furthermore, the results from Sec. 3.3 and 4 are not clearly showing the assumed benefit, as explained in the next Section.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the authors introduce a reasoning for each of their steps, that is clear to follow. The notation cloud use some slight improvements by clearly stating which methods (Sec. 2.2) are working on a featurized representation of the state ($\psi(s)$), but this is not required to fully understand the paper. The result section is missing some details, that the read can only try to infer. Namely, the state feature space used for the experiments in Sec. 3.3 and 4. Furthermore, Figure 6 lacks a frame of reference - meaning results without a combination of bonus terms. The reviewer assumes that the experiments shown in Figure 6 are based on the same feature space as Fig 4 & 5. Under that assumption, it seems like the global-only variant is outperforming the combined method in Corridors and KeyRoom, putting the claims into question. Something comparable is true for Section 4: Assuming that the message feature space was also used where applicable, some improvements are expected by combining E3B with a global method, because the global method should be capable of generalizing across contexts (compare Fig 5b). A comparison with a pure global method is required for substantiating the claims. It is still interesting that only the multiplicative method is able to show these improvements, but this cannot overcome the mentioned issues.

The analysis itself is novel, but the proposed method (multiplicative bonus) is a straight forward combination of existing ideas. However, additional insight into exploration strategies is a relevant topic in this domain, as this is important for reducing sample requirements and improving applicability. Reproducibility is assumed, given the source code should be available and architectural details as well as hyperparameters are given.  
summary_of_the_review: The paper is well written, but lacks a crucial detail and contains some questionable argumentations. Namely, the effect of the feature space is not sufficiently evaluated and it is not clear if a global-only bonus would be sufficient, given a context independent state feature space. In conclusion, the quality of the proposed method is not substantiated. The authors should clarify the mentioned feature space issues and add additional experiments with a global-only baseline and evaluate different feature spaces, if possible.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to analyze the use of two exploration novelty bonuses - global bonuses and episodic bonuses - in the problem of exploration in environments. Previous works show that global novelty is computed using agent’s entire training experience while episodic novelty is constrained within the specific episode the agent is in. This paper delves into the different behaviors of these two bonuses. When there is little shared structure between environments, episodic bonuses are useful; when there is more structure shared between environments, global bonuses are useful. This leads to the combination of global and episodic bonuses, and evaluation on various settings. 
strength_and_weaknesses: Strengths 
- This work is based on the contextual MDP (CMDP) setting, where C is the context space in addition to the MDP setting. Per each episode, the context c is sampled from the context distribution. 
- Nice and detailed explanations on background on exploration bonuses (RND, novelD, AGAC, RIDE) in section 2.2. 
- I think the main part of this paper is the section 3, raising the questions of when are episodic bonuses and global novelty bonuses are useful, respectively. 
- There are some example of CMDPs where global novelty bonuses fail, and advantage of the episodic bonuses succeed.(MiniHack, for example). Where the environment changes each episode, global bonuses fail (Figure 2). 
- There are other cases of CMDPs where global novelty bonuses are preferred. This is useful (1) singleton MDP case (2) although with large |C|, the structures are shared across contexts. 
- I think the arguments made by the authors are compelling so far. 
- The experiments in the section 3 are valid
- Evaluations are valid. 

Weakness
- Although the authors showed the insights on when to prefer global bonuses, and when to prefer episodic bonuses, and its relationship with level of structures being shared across the context. I think the contribution is not largely significant. This paper does not introduce a new algorithm,  although it shows some design choices for episodic and global novelty bonuses. 
- Although the authors argue that general CMDPs with large |C| with shared structures across contexts are good fit for global bonuses, they didn’t add theoretical insights in it; only tried to prove it via one example. 
clarity,_quality,_novelty_and_reproducibility: Clarity: high. well written.
Quality: medium. meaningful questions, but the contributions are very marginal. 
Novelty: not high
Reproducibility: medium. I couldn't find any information on tuning \beta parameters for 18 different tasks. 
summary_of_the_review: The idea is straightforward and convincing, but I think theoretical contributions / novelty / experimental evaluations are limited. I give the marginally below the threshold score, but I'm happy to raise the scores after the discussion with other reviewers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper examines the use of global versus episodic novelty bonuses for exploration in multi-task RL, and identifies that these two types of bonuses are useful in different contexts. Controlled experiments are performed in a few carefully selected MiniHack levels to demonstrate this effect, as well as in a more realistic domain (Habitat). The paper then proposes a multiplicative combination of these bonuses, and shows that this performs comparably well across multiple different types of environments in MiniHack, outperforming each individual bonus type on average.

The reviewers agreed that the paper studies an important and interesting problem, that the experimental evaluation on MiniHack is extensive, and that the results are insightful. In the initial reviews, the reviewers expressed some reservations that the evaluation was limited to the MiniHack domain, and that some potential baselines were missing. The authors' rebuttal partially addressed these concerns; in particular, during the discussion the reviewers said that they appreciated the experiments with Habitat and NGU and found that they strengthened the paper. However, there was also a lack of confidence that the results of the paper would necessarily generalize beyond the MiniHack domain. While the Habitat experiment helps, it would be better if the paper also had results across more domains in Section 4 demonstrating the same effect (that the different bonuses individually do well on different types of tasks, and that their combination is robust) in different settings.

While I agree the paper is insightful, the big question I find myself asking is: does this insight---that global and episodic bonuses should be combined to achieve the best exploration performance in CMDPs---hold in other CMDPs besides MiniHack? The Habitat experiment provides support for part of the hypothesis regarding episodic bonuses, but there is otherwise no evidence to suggest that the insight will hold more generally. While this might be less of a concern with a more canonical environment, MiniHack has a somewhat unusual observation space (most RL environments don't contain text observations, for example), which makes me less sure about the robustness of the results. As such, I think the paper falls (slightly) short of the bar for publication at ICLR. It is a very promising hypothesis, though, and I encourage the authors to continue working on this and to submit a revision to a future venue.
justification_for_why_not_higher_score: Please see my comments in the "Summary of AC-reviewer meeting".
Decision: Reject