Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new design of intrinsic reward, as a variant of the count-based reward, to tackle the sparse-reward tasks in reinforcement learning. 

Prior works related to count-based reward, especially the ones focusing on procedurally generated environments, widely use the episodic count, which means resetting the state visitation count for each episode, instead of recording the global visitation count throughout the entire training with multiple episodes.

This work proposes LECO, i.e. to get the state visitation count via learning the VQ-VAE model and regulate the count-based intrinsic reward via the LIRPG framework. Specifically, VQ-VAE encodes the state into a latent feature and maps the latent feature to the nearest embedding in the code book. The hash code is a sequence of indices of the matched embeddings and the intrinsic reward is based on the visitation count of each hash code in an episode. Also, the count-based reward is combined with the task-specific modulation, which is learned through the LIRPG framework.

The experiment is conducted on procedurally generated environments in MiniGrid benchmark and Deepmind Lab. LEGO is compared with recent works NovelD and AGAC and some variants as ablation study. It turns out that LEGO mostly outperforms these baselines.

strengths_and_weaknesses: Pros:
It is relatively novel to learn VQ-VAE and then use the discrete hash code from the codebook for state visitation count.
The method is naturally motivated and technically reasonable. Based on prior methods of locality-sensitive hashing with auto-encoder (VE), it is a natural idea to modify AE to VAE. Because we need discrete state representation for counting, it is natural to try VQ-VAE.
The experiment result is positive, especially on hard exploration tasks ObstructedMaze in MiniGrid.

Cons:
The novelty is a bit limited. LECO seems simply a combination of two prior works: VQ-VAE and LIRPG. It will be great if you can clarify more technical contributions beyond the combination.
The method is not clearly explained in Section 3.3 and 3.4. Ideally, the method section should be self-contained enough for readers, even though they are not familiar with VQ-VAE or LIRPG.
The experiment part to compare with baselines can be improved.  The comparison with the most recent SOTA on MiniGrid (MADE) is missed. The performance of baseline NovelD in Figure 5 is worse than the results reported in NovelD paper. 

questions: In Section 3, it seems that some details are missed. For example, in Section 3.3, what’s the dimension of $z^e$? What’s the relation between k and $k_i$? Although we can infer these details from the context, it will be better if they are crystal clear. In Equation (2), $log$ is a function, not a probability distribution. What’s the meaning of $log(s_t|z^q)$? Conditional probability distribution? In Section 3.4, $r^i_t$ does not include $\lambda$, which is different from Equation (1). Is it a typo?

Is LECO sensitive to hyper-parameters? such as the size of the codebook in VQ-VAE, the weight of task-specific modulation, the weight of intrinsic reward, etc. Could you please analyze the effect of each critical hyper-parameter? How and why does the higher or lower value choice of each hyper-parameter influence the final performance?

In Figure 5, why does LECO perform worse than NovelD in MultiRoom and KeyCorridor environments, but outperform it in ObstructedMaze? Any intuitive explanation?

In Figure 5, NovelD is worse than the results reported in NovelD paper. Is it because you use VQ-VAE to count instead of using their naive approach to count? Why not use their original setting?

VQ-only in Figure 5 and AE-LSH in Figure 6 shows a weird performance pattern. They can learn reasonably well on easy task MultiRoom and hard task ObstructedMaze, but fail on the medium task KeyCorridor. Why? Have the hyper-parameters (e.g. weight of the intrinsic reward when added upon extrinsic reward) in these two baselines been searched extensively?

In Figure 6, could you explain why LECO is less sample efficient than NovelD for the noisy TV problem? The difference is kind of significant, that is, LECO costs around 4e7 more frames than NovelD to converge.

In Figure 7, it will be more impressive if LECO is compared with more classical and strong methods in Deepmind Lab, such as RND ('Exploration by Random Network Distillation') and ICM (“Curiosity-driven Exploration by Self-supervised Prediction”). NovelD mainly showed advantages in procedurally generated environments, instead of general sparse-reward tasks. Maybe NovelD baseline is not strong enough compared with other possible baselines on Deepmind Lab.

About the ablative study, could you please add the comparison to AE-LASH + LIRPG? I interpreted LECO as VQ-VAE + LIRPG (please correct me if I misunderstand it), so I’m curious about the effect of VQ-VAE in comparison with the widely used counting method AE-LASH.

What has been learned as state features $z^e$? Could you qualitatively or quantitatively analyze it? How about showing some examples of states assigned to the same hash code, and states assigned to different hash codes?

Minor points:
I doubt it is proper to name the memory buffer for hash code counts as ‘episodic memory’ (see Figure 1). In the literature of psychology, this term means recalling particular and subjective life experiences. In the RL area, this term is extended to refer to the state-action pairs with corresponding Q values in a buffer (e.g. ‘Model-Free Episodic Control’, ‘Episodic Memory Deep Q-networks’). The meaning of ‘episodic memory’ in this work seems to deviate from the literature.

The figures look preliminary without axis labels or meaningful sub-captions.

limitations: The authors mentioned the limitations of LECO on tasks that require inter-episode novelty, or difficult bi-level optimization. The issues are not fully addressed, but it is a natural idea to combine the episodic novelty in LECO with inter-episode novelty.

It will be interesting if the authors study VQ-VAE to count inter-episode state visitation. Especially on hard-exploration tasks on Atari games, the inter-episode novelty is necessary. Could the count learned with VQ-VAE outperform the previous approaches with pseudo-count, e.g. AE-SimHash ('#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning'), CoEX ('Contingency-Aware Exploration in Reinforcement Learning'), NGU ('Never Give Up: Learning Directed Exploration Strategies')?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a method for count-based intrinsic motivation in Reinforcement Learning. Considering the issue that the counting mechanism may drive the exploration towards states that are known to be useless for maximizing the extrinsic reward, the introduced method LECO modulates the intrinsic reward such that exploration of useless agents is not encouraged. The proposed method uses a neural network for computing the counting (using a variant of a variational autoencoder) and meta-gradient for the training of the modulator. Experiments on meaningful problems are provided.
strengths_and_weaknesses: Strengths
-------------
- The paper deals with the interesting field of intrinsic motivation, and studies an important issue affecting methods in literature;
- The proposed method is sound and it seems easy to implement;
- The empirical evaluation provides good evidence of the advantages of the proposed method over several baselines.

Weaknesses
-----------------
- The paper does not provide theoretical results. This is typical for intrinsic motivation method, but it would be nice to see some theoretical guarantees about the convergence to 0 (or at least a very small value) of the intrinsic reward for useless states. Probably this would need a different analysis and implementation allowing closed-form solution, perhaps replacing the MLP of the modulator with a linear approximation;
- It is not clear how the $\lambda$ parameter is computed, although this is not an important weakness and I'm sure it is quite straightforward. Still, I'd like clarifications;
- Most of the equations are inline. This makes it difficult to read sometimes. For example, from line 197 to line 203;
- If the method is similar to LIRPG, why there are no comparisons with it? Am I missing something?

Minor comments
----------------------
- In the list of contributions, the last two items are not contributions; I'd remove them, or rephrase;
- line 121: adapted -> adopted
- Figure 3: visitaion -> visitation
questions: - I'm sure I'm missing something, but why the modulation reward is a tanh? In this way, it is unclear to me what the effect on the total reward will be when the modulator reward is > 0. In this way, it seems like the modulator can add an exploratory bonus to some states if they are found to be interesting, is this wanted? It seems more reasonable to me to use a sigmoid and subtract it from the episodic count reward. I'm probably missing something here, so I'd be glad if the authors can clarify this doubt;
- How is the $\lambda$ parameter selected? It seems to me that it should be directly related to the magnitude of the episodic count reward;
- In the appendix, it is stated that the proposed method is similar to LIRPG when the normalized modulation coefficient $\tilde{\lambda} = 1$. Can you clarify what these similarities are?
- Is the need for a vector-quantize VAE crucial? Why not use a regular VAE? Why not a regular AE?
limitations: The limitations of the proposed work are very briefly explained, from an intuitive point of view, in the conclusion. I think the authors could add more information, for example, describing under which conditions tasks may require intensive leveraging of inter-episode novelty. I also think that the similarity with LIRPG is a limitation that should be addressed properly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies a learnable hash function for count-based exploration. The hash function is learned through a vector quantized variational auto-encoder. The associated intrinsic reward is further modulated by a second reward that learns to focus on task relevant extrinsic reward, which is learned with a meta-learning approach. Both signals are combined through addition, hoping the task modulation will gradually learn to dominate the novelty, and the agent will gradually switch from exploration to exploitation. Experiments in MiniGrid environments and two DMLab tasks shows that their method generally outperforms or is on par with several baselines. 
strengths_and_weaknesses: Strong:
* Relevant topic (intrinsic motivation). 
* Decent experiments with better results than the baselines. 
* Well-written paper. 

Weak:
* The idea is a bit incremental, since it combines two known ideas: 1) count-based novelty (which was also already tried with VQ-VAE), and 2) meta-optimization of a task-specific intrinsic reward.
* I find it a bit hard to interpret the DMLab results, since there is no figure of the actual task, nor a description how the authors ended up with these two specific tasks. The learning curve of their method does look better. 
* I also slightly miss the motivation why these baselines are chosen (DSC, AS-LSH, NovelID, AGAC). 

questions: -

Conclusion: This is a decent submission: it is a relevant topic, the paper is well written, and their approach outperforms the baseline. My biggest worry is the slightly incremental character of the approach: they essentially glue together two known ideas, which is still a relevant contribution, but not extemely novel. The better performance compared to the baselines is promising, although I slightly miss a description of 1) how baselines were chosen and 2) how test tasks were chosen. Otherwise, this is a good paper. 
limitations: -
review_title: null

Meta-review: metareview: Reviews were mixed here and all quite borderline. There are legitimate points raised for why this is being consistently given borderline ratings, with two in particular resonating with my own reading (novelty and comparisons with other methods). However, despite these issues the paper itself is a solid contribution, and I think could easily lead to others building on the core ideas and approach. The paper has also improved notably during revisions and at present does not have any fundamental flaws that should preclude publication. Therefore, I recommend acceptance.
confidence: Less certain
Decision: Accept