Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: * This paper challenges the contemporary experimental setup of exploration algorithms, and suggests that we should not start exploring in new environments tabula-rasa, and instead use knowledge derived from prior experience to bootstrap exploration. 
* The paper also identifies issues with using existing tabula-rasa exploration algorithms for the new setting, and propose a new method C-BET to overcome those shortcomings. 
* C-BET and a bunch of existing exploration algorithms are benchmarked on newly constructed transfer exploration setups in the MiniGrid and Habitat environments. 
main_review: This paper offers a fresh perspective on training and evaluation exploration algorithms for RL environments. The current paradigm in training exploration methods is to start "tabula-rasa" in each environment, with no pre-conceived knowledge about the world. This means that the agent has to blindly try-out and explore a combinatorically large state-action space which then leads to requiring billions of frames or years of experience to solve sparse reward problems, something that's impractical for real-world usage. Not to mention the inherent safety issues involved with tabula-rasa exploration. The paper proposes a new experimental setup to evaluate exploration algorithms: the algorithms are first pre-trained in a task-agnostic manner on some environments and then transferred to unseen environments. 

The writing in the paper is clear throughout and it does good job of explaining why this different experimental setup makes sense. I also appreciate the generous use of figures to make the exposition clear. One of the most important contributions of this paper is the construction of new experimental setups for exploration transfer, and benchmarking existing methods on them. These experimental setups cover MiniGrid (a suite of GridWorld environments) and Habitat (a suite of visual 3D environments), thus covering a wide spectrum of scenarios. 

Its interesting to see curiosity like methods fail during transfer, but somewhat obvious in hindsight. After learning a predictive model of the world, similar states are not so surprising anymore even though they might still be inherently interesting in the new environment. C-BET tries to fix this by using a pseudo count of interesting changes along with count resets every episode. The results on pre-training and task-transfer are quite compelling. The analysis in the supplementary material is quite informative and its interesting to see that a simple trick like count resets has such a large impact. 

Overall, I really like this paper, the ideas and perspective on exploration are original, presented well and are of great significance. I would have recommended acceptance even without the introduction of a new method, but that is an added bonus. I do have some comments / questions though and it would great if the authors can expand on it: 
* Can you benchmark / ablate the impact of *change* pseudo counts? Specifically, running your "Count" baseline along with episode resets, and comparing with C-BET (count + change + episode resets) would clarify this. 
* In the Habitat experiments, the use of panaromic images seems like leveraging access to privileged informtation that other methods don't have. Have you tried experimenting with using egocentric images for pseduo-counts and the change encoding? Does it lead to a significant performance drop? 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper focuses on task-agnostic exploration in RL, with two key ideas:
- introducing a new evaluation method for task-agnostic exploration, by measuring "how well" the learnt exploration policy behaves in new tasks; "how well" can either be measured based on extrinsic return in the withheld task, or on exploration-related quantities like coverage
- proposing a new count-based intrinsic reward that is shown to outperform other baselines in the transfer task in procedurally-generated gridworlds, and in the coverage task in 3D scenes from Habitat.
main_review: The article is well written with a wealth of experimental results, as well as thorough in the Related work section.

Originality: 
- The idea of measuring how well a policy transfers is not new, but I have not seen it applied specifically to evaluate exploration policies.
- As far as I know, combining state visitation count (already established in the literature) with change count has never been published.

Contribution/significance: I think good enough, with one "medium" and one minor contribution.
- The first problem addressed by the paper -- evaluating task-agnostic exploration -- is an important one in the community, and I think the proposed protocol has merits. There are a few limitations that I think currently limit the applicability of the method, but they seem somewhat easy to overcome and are clearly identified in the paper. To me this is a significant enough contribution.
- The proposed C-BET exploration algorithm seems to achieve good performance according to that metric. I have to say I'm always a bit wary of papers that both introduce a new benchmark and a new algorithm that outperforms its competitors on that benchmark, but in this instance the relatively thorough set of baselines offsets this issue. However I'm not sure how generally applicable C-BET is (both in terms of scalability and applicability).

Questions:
- How scalable is C-BET when the state-action space grows? In the Habitat experiments, the authors had to use a very coarse action space with 3 actions (thus drastically reducing the complexity from a 3D env to a gridworld with rich observations). Does C-BET still perform reasonably well with, eg, 30° vertical and horizontal rotations?
- How does C-BET perform in environments with intra-episode stochasticity (eg, noisy tv)? The 1/sqrt(N(c)) will ultimately compensate for this if the episode is long enough, but because counts are reset every episode, the additional |A| factor (|S||A| state-action pairs to visit instead of just |S| states) seems like it can significantly hurt performance in practical setups.
- Figure 7: Shouldn't the results on MultiRoom-N12-S10 ("no transfer" row) match those of Fig 3h in https://arxiv.org/pdf/2002.12292.pdf? RIDE should get to ~0.6 in 25M frames.
- Figure 9: it's not clear to me how to interpreter error bars. Is the colored bar showing the average and the top of the error bar the maximum over 5 seeds? If so, what about the minimum?
- Section 5, "advantage" paragraph (and scattered throughout the rest of the paper): I'm not sure I see how C-BET accounts for "rarity" of change, since eg the agent's position is presumably part of the state, and C-BET doesn't have any form of (action, state) embedding that would be able to learn translational (or other) invariances? Presented with an environment with, eg, N on/off switches that are part of the state but do nothing interesting, won't C-BET be tempted to explore all 2^N combinations?

Extra comment: unless negative rewards exist in MiniGrid, something seems off with the smoothing of curves (eg Fig7)
limitations_and_societal_impact: There is a clear and fair acknowledgement of limitations in an explicit section, which I really appreciated.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a new paradigm for exploration based on transfer. They argue that we should not focus on exploration in isolated environments (it is not realistic for humans either), but instead focus on a sequence of tasks. More interestingly, they propose that we need to disentangle the exploration signal, into one agent-centric one based on visitation counts, but also a environment-centric one that identifies states where we get a change in the environment, and are thereby inherently interesting over tasks. They propose a new intrinsic reward that combines both properties, and test these in gridworld experiments and first-person navigation tasks, where they perform on par with or better than baseline exploration methods. 
main_review: Strong:
* I really like the main idea: the fact that certain objects are interesting by definition, not in a single task, but over repeated tasks. This is indeed a novel perspective that I have not seen before, but makes a lot of sense.
* Extensive experimentation, interesting observations. For example that transfer using model characteristics is not efficient at all for transfer. 

Weaker:
* I would like a more detailed comparison to Raileanu and Rocktäschel [36]. Since the claims in the introduction are quite strong (a completely new paradigm for exploration), you should motivate this carefully compared to related work (I think the statement is a bit strong).
* I am a bit in doubt about the “count resets”. I understand the reasoning, but it may actually be harmfull within a task (if there is one state we have not visited yet, then keeping the counts this may be rewarded highly, while with a reset we have to start from scratch again). I think the paper at least needs to discuss this, and a more elegant solution (which allows optimal exploration within the task as well as for transfer) would be preferable.
* I am a bit confused between c(s, a, s) (e.g. L215) which is a vector, and N( c) in Eq. 1. Is N( c) the sum over the vector? 
* I am also confused about the panoramic images (L283-288). How do you align these for the equation in L 290 (otherwise the element-wise difference does not make sense right, if you have rotated your view every element will change).
* Fig 10: I find the Habitat interpretation really qualitative, the main result being “that C-Bet is the only policy visiting the right-hand corner”. Sure, but RIDE is the only one to visit the top-middle corner? 

Summary:
I really like the idea of this paper, that task-agnostic exploration should be separated into generic environment-centric properties (objects that are always interesting) and agent-specific properties depending on the interaction history (such as novelty). I think this is an important insight, especially that one can estimate the former property during training. There are some smaller issue with the paper as mentioned above, but none of them is really big. The experiments are sounds as well, and indeed show that the proposed method works. I recommend this paper for acceptance, since I believe it will definitely interest the broader RL exploration community.  

limitations_and_societal_impact: Adequately addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a novel problem setting within the subfield of exploration in reinforcement learning, and a novel method aimed at solving this problem setting. Specifically, in the new problem setting the agent first gathers experience in several environment with no access to extrinsic reward. The agent is then transferred to similar environments where it must perform the task at hand. What makes this interesting from an exploration perspective is the fact that all the environments are hard exploration problems, so one possible good approach is to learn a good exploration policy in the initial environments, such that when the agent transfers to the new environments it can quickly optimise the extrinsic reward. The new method (CBET) is based on combining standard state-visitation counts with a novel state-change-count, and episode resetting of the counts. The method gets better performance than existing methods (which are targeted at the tabula rasa exploration setting) in terms of exploration in the initial environments and speed of learning in the new environments, with experiments on MiniGrid and Habitat.
main_review: ## Overview
Overall, I believe the paper is just about worthy of acceptance. I think the novel problem setting of transfer-exploration is a useful problem setting that hasn't been studied before, and is more relavant to real-world settings than pure tabula rasa exploration. The benchmarks provided for this problem setting aren't the most thorough or rigorous (another benchmark in which transfer-performance can be measured on a down-stream task would be beneficial), but do still match the capability the problem setting aims to capture. The extensive experiments show that existing methods focused on tabular rasa exploration fail to perform as well in this new setting. The novel method (CBET) proposed for this setting contains some interesting insights about the difference between this setting and tabula rasa exploration, and shows better performance in this setting. I believe the paper could be improved with more experiments ablating different parts of CBET, to see which components are most useful for learning an exploration policy which transfers well to down-stream tasks. Further, a variety of clarifications are needed to ensure that the methodology described in the paper is clear and unambiguous.

## Strengths

As described above, I think the novel problem is interesting and useful, and more relevant to real-world problems than tabular-rasa exploration. The proposed benchmarks adaquately assess the capabilities required in the problem setting. The thorough experiments on existing methods shows that they fail in this problem setting (which isn't surprising, but is useful information), and further experiments in the appendix suggest why this is (due to model learning and hence intrinsic reward collapsing relatively early for these methods). Their novel method is well motivated to solve these problems, and does demonstrate better performance in that benchmarks that existing methods. In general the paper is well written and easy to read.

## Weaknesses and suggestions for improvements
There are several ways in which the paper isn't as strong as it could be

  - While the problem setting is defined informally, it would be useful to have a formal definition, which corresponding terminology. This would enable discussion of the various parts of the problem setting more easily, and make it clear exactly the kind of problem you're trying to present and subsequently solve.
  - While CBET is demonstrated to perform better in the benchmark, and some analysis of the method of resetting is done in the appendix, I think further analysis, and specifically ablations, would be useful to understand why CBET performs better than a simple count-based method. Specifically, the two adjustments CBET makes to a count-based method is the (multiplicative) inclusion of the state-change-count as part of the intrinsic reward, and the use of count resetting after every episode during pretraining. Ablations which remove these two components would be useful to see which of them contribute to performance, and in what way.
  - While I don't think it's necessary for acceptance, another benchmark which had a down-stream extrinsic reward to optimise for (or adjusting the Habitat benchmark to have a downstream task which the policy is transferred to) would improve the robustness of the results and ensure that the capability being measured matches the capability the problem setting desires.
  - Again, I don't believe this is necessary for acceptance, but the current SOTAs on MiniGrid is (I believe) [BeBold: Exploration Beyond the Boundary of Explored Regions](https://arxiv.org/abs/2012.08621) and [Adversarially Guided Actor-Critic](https://arxiv.org/abs/2102.04376), neither of which are compared against in this paper. If these comparisons could be added, it would improve the results in the paper and give evidence that CBET is a valuable new contribution. In particular AGAC doesn't suffer from intrinsic reward disappearing during training, and BeBold uses episode count resets as part of it's intrinsic reward calculation.

### Smaller points

  - Related to Figure 7: It would be beneficial to have an average task-specific reward (or solve rate) for CBET and each of the baselines in MiniGrid, averaged across all transfer tasks, to have a number to compare between the methods. This could be a number related to sample efficiency as well as or instead of task reward.
  - As part of improving our understanding of CBET's performance, It would be great to see the values of the state-count and change-count plotted for different environemnts and different settings. For example, my intuition is that the change counts (and the state counts) for the Habitat environment, given the observation is so rich, should never be the same. Is this the case, or movement in the state-space discrete in this environment, and hence states are often repeated?
  - Related to the above point on BeBold and AGAC, it would be informative to see results of CBET on the standard "No Transfer" setting in MiniGrid across a wider range of environments, such that it can be compared to other exploration methods which are targetted at that setting. While it isn't relevant to the main concern of your paper, it would be informative to see whether CBET's performance on the transfer-exploration setting comes from just better exploration in general, or is specifically most useful in this setting.
  - L23: If the rewards are unknown then the goal isn't defined, so this isn't an entirely realistic setting - there's nothing to achieve. I think you should stick to sparse rewards as a motivating example for exploration.
  - L66: Which approaches? This sentence is unclear to me.
  - L112: "In feature transfer, during..." you've missed a word in this sentence I think.
  - L115: I think it might be worth citing work such as [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://arxiv.org/abs/2008.02790), which is a meta-RL work which aims to learn an exploration policy which is useful in down-stream tasks, and is hence similar in some sense to your problem setting. In general, I think it could be worth contrasting with this kind of Meta RL setting in the related work setting. In L117 you claim that task-specific policies are transferred, but this generally isn't the case in the Meta-RL setting.
  - L139: "[...]and the changes that are rare": add "that"
  - L158: At this point it would be worth discussing the limitations of count-resets - each episode the agent could explore the same parts of the environment, effectively memorising a policy which hits all the novel states and interesting changes it can, but never exploring the environment beyond that point.
  - L226: "...ten new environments, seven of which are new." - This should be just "...ten environments, seven of which are new..." I think.
  - L240: That's quite a lot of smoothing - are evaluations very high variance? What do less smoothed curves look like?

### Clarification Questions
  - What's the difference between CBET (Ours) and Our Reward in Figure 7?
  - In Figure 7 no-transfer MultiRoom-N12-S10, RIDE gets 0 reward after 100M time-steps, but in the original RIDE paper, RIDE achieves 0.6 extrinsic reward after ~20M time-steps (see Figure 3 (h) in that paper). Do you have an idea why there's that discrepancy in results? Other results from the RIDE paper roughly match the corresponding no-transfer experiments in your paper.
  - In your experiments do you continue training with the intrinsic reward combined with the extrinsic reward after transfering to the new environments, or do you solely train with extrinsic reward after transfer and intrinsic reward before transfer? If you're only training the pre-trained policy with extrinsic reward in the new environment that might explain the discrepancy in scores pointed out in the bullet point above, but I also think that it's not a valid evaluation methodology - there's no reason to disallow intrinsic rewards and further exploration bonuses being used after transfer to a new environment.
  - L178: Your method is unclear to me here: IMPALA learns a state-value function $V(s)$, and a policy $\pi(a|s)$, but doesn't learn a Q-function $Q_i(s,a)$ as you've written it. How do you adapt IMPALA to learn a Q-function?

## Conclusion
I believe the paper is marginally above the acceptance threshold, but I think if the adjustments and additional experiments described above are added (especially the ablations) then I'd be willing to raise my score further.

## Edit:
Given the comments in this response and your response to other authors, I believe my confusions have been clarified, and I believe the paper will be much improved with the improvements you've described here and elsewhere. Hence, I'm raising my score to an 8 - I believe the paper is an interesting, novel and relevant problem setting and presents a novel approach to solving it, while also bein rigorous in the evaluation of their and previous works to ensure that their contribution is robustly useful.

limitations_and_societal_impact: Comments on limiations have been adaquately addressed in the paper, with any additional concerns raised in the review above.
review_title: null

Meta-review: This paper introduces a new problem setup for exploration in RL, which is to allow the agent to learn an exploratory policy in the absence of reward/task and evaluate how quickly the agent learns when the task is given in the transfer phase. All of the reviewers highly appreciated this problem setup and agreed that this setup is reasonable and deserves more attention, though a similar setup has been considered in the past [1], which should be cited and discussed in the paper. In addition, the paper proposes a new count-based exploration method that encourages rare state changes and proposes to reset count at every episode to overcome the vanishing intrinsic reward issue specifically for the new proposed problem setup. The results on MiniGrid and Habitats environments show that the proposed method performs better than baselines in both the exploration phase and the transfer phase. The authors promised to address several minor questions around count reset and ablation in the revision. Overall, the new problem setup and the proposed method would be quite interesting to the research community. Thus, I recommend acceptance for this paper.

[1] Rajendran et al., How Should an Agent Practice?
Decision: Accept (Oral)