Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a method that alternates between learning the feature mapping via the inverse dynamics model and using the learned feature to construct the elliptical bonus for exploration. The paper also analyzes several heuristic exploration bonus method by removing the count-based component in them and shows that they fail under such modification. The paper claims that the proposed algorithm contributes the most to the contextual markov decision processes (CMDP) setting, and shows its empirical competitiveness in the MiniHack benchmark. 
strengths_and_weaknesses: ## Strength

1. The proposed method, E3B, when evaluated in the MiniHack benchmark, which is a reasonable benchmark for CMDP, shows better performance than the previous empirical methods with heuristic exploration bonuses. The evaluation also contains some variants of the baselines to provide more insights into the results. 

2. The paper is easy to follow with several informative visualizations. 

## Weakness

1. It is very surprising to see that the proposed algorithm, which resets the empirical covariance matrix at the beginning of the episode (according to Algorithm.1), can actually work. First, this should not work in theory, where your covariance matrix $C$ is constructed using the policy cover defined on all previous iterations. Intuitively, it is also not very clear why this can work effectively. Resetting $C$ at the beginning of the episode means the bonus construction has no memory of what states have already been visited before. Combined with that the algorithm performs policy optimization based on reward + bonus, consider the setting if we have sparse reward and short horizon: for example, if one wants to escape a maze in small time budget, the algorithm could first visit some dead end, but receives high bonuses due to resetting $C$, and update to visit the dead end with higher probability after the PG update, and visit the dead end again but still receives high bonus, and thus eventually stuck?

2. The contribution of this work does not seem obvious, either. Using elliptical bonus in Deep RL or with neural network function approximation has already studied by some previous work. For example, PC-PG [1] (which this paper also mentions) actually uses PPO+elliptical bonus in their experiment. [2] even has a specific Deep RL version that uses deep RL + elliptical bonus and is evaluated in the deep RL benchmarks. If we also consider the representation learning part, [3] provides both theoretical and empirical results with neural network function approximation. In addition, [4] also justifies that a variant of [2] is actually performing feature learning in a noisy system. There seems no comparison against any of these works in this paper, given that the approaches are similar and the above baselines actually also show strong performance in practice.

### references

[1] Agarwal, Alekh, et al. "Pc-pg: Policy cover directed exploration for provable policy gradient learning." Advances in neural information processing systems 33 (2020): 13399-13412.

[2] Song, Yuda, and Wen Sun. "Pc-mlp: Model-based reinforcement learning with policy cover guided exploration." International Conference on Machine Learning. PMLR, 2021.

[3] Xu, Pan, et al. "Neural contextual bandits with deep representation and shallow exploration." arXiv preprint arXiv:2012.01780 (2020).

[4] Ren, Tongzheng, et al. "A free lunch from the noise: Provable and practical exploration for representation learning." arXiv preprint arXiv:2111.11485 (2021).
questions: 1. In figure 1, why are the baselines not performing well even with their original constructions (with counts)?

2. The motivation against using counting based method seems not very fair? "if each state is unique" (line 102) almost implies that the state space is continuous, then one obviously should never use naive count-based bonus, since count-based bonus only works for tabular MDP. 

3. In the feature learning part, it is known that the inverse dynamics model is valid only if it is conditioned on some conditional distributions of actions (for example, policies). In the proposed algorithm, is the conditioned policy being a mixture of all previous policies? Would it providing conflicting information, for example, at some state policy a goes left and policy b goes right? Since the algorithm aims to perform exploration.

4. In result shown in Fig.6, is there any hypothesis why RND completely fails on all tasks?

5. The ablation shown in section 5.1.3 is not using very strong baselines. For example, in [1,2,4], those method are using RFF kernels and random networks like RND as feature mappings, which show convincing empirical performance. 

6. In Fig.8, is there any possible explanation for why E3B is worse than the baselines in some task? 
limitations: 1. As mentioned in the previous section, resetting the covariance matrix seems a limitation of the proposed work. 

2. The results shown in Section 5.2 shows that the proposed algorithm achieves very limited improvement or even no improvement than previous algorithms on regular MDPs with large state space.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper gives evidence of the importance of the pseudo count quantity used in existing exploration algorithms in RL, and proposes a new algorithm to extend the pseudo count idea to continuous state spaces by also proposing a method to learn a feature encoder. They give this evidence through various empirical experiments.

Thanks to the authors for putting in the effort in doing this work!
strengths_and_weaknesses: Strengths:
- I appreciate works that take a hard look at existing methods in the manner that is done in this paper and try to understand the fundamental flaws.
- I like table 1 of distilling the exploration methods to their essence, and also I like figure 3.
- The time-counter example of how states can appear unique is a nice one for intuition.
- The paper is generally well-written and easy to follow.

Weaknesses:
- I view the paper as trying to solve two independent problems, but the paper makes it seem they are intrinsically tied together. The two problems are: 1) learning a feature encoder and 2) exploration. While the paper does include an ablation study showing how important both these parts are, it would be nice to get a better understanding of these two independent components. For example, how does using the learned encoder improve other existing count-based methods, or how does using off-the-shelf feature encoders affect the performance of the elliptical exploration algorithm?

questions: Questions/Suggestions:
- How are differences between contexts defined? I see that there is a distribution over contexts but how extreme can differences between two contexts be? It doesn't appear to be the case here, but generally does the reward function change between contexts too?
How many samples are actually needed for fitting in the ellipse? 
- The paper also notes that “any feature learning method” could be used, but it proposes to use the inverse dynamics model. I am curious how off-the-shelf feature extractors perform with the elliptical exploration algorithm? It seems like this problem is an independent problem itself, and it would make sense to leverage existing works instead of re-inventing something new.
- How sensitive are results to the regularization term in Equation (1)? In my experience, I have found this term to be sensitive.
- I know the goal isn't explicitly lifelong learning, but I am curious what the authors think about getting E3B (non-episodic) from Figure 7 to work in that setting? In a lifelong setting where the agent is constantly interacting with no clear start and end to different contexts, I wonder how the proposed method would work then and how it compares to existing methods?
- So we can see a clear performance improvement in Figure 6. But I am curious why ICM performs similarly to E3B on the high-dimensional Vizdoom domain, especially given that the motivation was current methods don’t fare well in high dimensional/continuous state spaces?
- How will the count-based methods in Table 1 improve or change if you use the proposed feature encoder (inverse dynamics model) as input into these existing exploration methods instead of using the proposed elliptical count-based algorithm?

limitations: Yes, they have.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work highlights an important weakness for existing exploration methods relying on intrinsic motivation in CMDPS, namely their over reliance on a count-based episodic term. Since this count-based term is computed on hand-engineered features, or on the entire observation, such methods underperform in noisy or more realistic environments. The authors propose to replace this term with an elliptical episodic bonus, computed on learned features extracted by inverse dynamic modeling. This simple modification significantly improves the performance of intrinsically motivated agents in complex environment which differ across episodes. This is shown through experiments on the MiniHack suite, and potential for scaling to image-based tasks is presented on VizDoom. The authors also report some interesting nuances in environments, and clarify the importance of heuristic choices in existing methods.
strengths_and_weaknesses: **Strengths**
* The claims made through the paper are modest, but well argued and sufficiently supported by experimental results.
* The paper is well written and easy to follow. The motivating weakness of existing methods is outlined well, and the idea of elliptical bonuses is clearly presented, and motivated from different point of views.
* The experimental section is convincing: the metrics reported follow a statistically principled evaluation approach, baselines are satisfactory and their behavior is well explained.

**Weaknesses**
* While the motivation and the intuition of the method are clear, the main body of the paper does not explicitly describe the algorithm itself, which is relegated to the Appendix. As a result, this part of the paper is lacking in clarity.
* The method is not particularly original, as it can be seen as an incremental generalization of count-based exploration bonuses to continuous settings.
questions: **Main comments**
* The method's clarity can be improved. While the computation of the elliptical bonus is clear, its integration in the reward signal is not. This should be clearly discussed in the main body of the paper. Furthermore, to the best of my undestanding, E3B's reward only includes an episodic term, while RIDE, AGAC and NovelD also include a global term. While the lack of robustness in the episodic term is understandable, it is not clear why E3B also removes the global term.
* I would suggest author to ablate the method by replacing the elliptical terms with a hash-based count term [1], which can be computed over the representations learned through inverse modeling.
* The choice of MiniHack tasks is not well motivated. I am not fully informed on common evaluation protocols on MiniHack, but to my understanding the authors performed a selection of tasks. The reasoning behind this selection should be motivated, particularly in relation to prior works.
* Figure 6 shows how RIDE does not achieve any reward signal in any of the tasks considered. Can the authors provide an intuitive explaination for this result?

**Minor comments**
* line 50: Prior works on elliptical bonuses should be cited before the Related Works section on the last page. I would suggest to add them here, in order to clarify that this idea is not novel.
* lines 83-85: I would recommend adding references to NovelD and AGAC to better assist the reader.
* Figure 1: What is the behavior of each agent when disabling the count-based episodic bonus? Is their explorative behavior less extended, or do they fail to move at all?
* line 123: This experiment is only reported for NovelD, but the reasoning for this choice is only provided on line 243. I would recommend to move this explanation here.
* line 176: Why is $n$ not fixed across all experiments? How is its value selected?

**References**

[1] Haoran et al. "#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning.", 2017
limitations: The authors do not overstate their contribution, and the limitations of their work are properly presented.
review_title: null

Meta-review: metareview: The reviewers appreciated the fundamental questions the paper was asking, clear writing and argumentation of the paper and convincing empirical experiments. While there were concerns about the theoretical rationale of resetting covariance matrix, the empirical results show it is indeed important. For these reasons, I recommend acceptance.
confidence: Certain
Decision: Accept