Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method for training RL agents in settings where the environment distribution is long-tailed, i.e. an agent must solve N tasks, but some of them occurs much more frequently than others. Examples of such a setting include the Zipfian RL environments introduced by [1]. This paper proposes a method that uses contrastive learning to get a familiarity score based on the contrastive loss (smoothed using momentum). This familiarity score is then combined with an episodic memory to provide additional learning for trajectories that belong to rare tasks. 

The method is evaluated on a single environment, which is a modified version of the Zipfian gridworld introduced in [1]. It shows improvement over standard IMPALA, as well as different ablated versions of the proposed method, when the distribution of tasks is uniform rather than long-tailed, and the improvement is especially pronounced for rare tasks. 








[1] Chan et al., "Zipfian environments for Reinforcement Learning". 
strength_and_weaknesses: Strengths:
- The setting is interesting and the general idea of considering heavily skewed task distributions is well-motivated
- The method does seem to show improvement, although it's only evaluated on a single task

Weaknesses:
- The main weakness of this paper is the experimental evaluation. First, only a single environment is used - this is insufficient to showcase the generality of the method. It could be that this particular approach is overfit to this task. I would suggest the authors test their methods on several more environments. 
- For some reason, the authors do not use the original version of the Zipfian Gridworld task from [1] and make their own modified version of it (which is also easier, 10 objects/tasks instead of 20). They claim in their paper that this is because the code of [1] is not available, but it is: https://github.com/deepmind/zipfian_environments. It has been available long before the ICLR submission deadline. Since the environments from [1] are open source, I think they should be used instead to facilitate consistency across publications and to ensure comparability of results. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 
- Medium. The main idea/algorithm is fairly clear when described in words, but the figures do not do a very good job of illustrating it. In particular, my suggestions for Figure 3 are to illustrate the merge/fetch operations better, as well as the "transfer tk rarest states" step. 

Quality:
- Low. As noted above, the experimental evaluation is not sufficient. My suggestions would be to use the open-source Zipfian Gridworld task as described in the original paper, and add the other two tasks from [1] as well. 

Novelty:
- Medium-High. The method appears novel and I think does hold promise, it just needs to be better demonstrated through the experimental evaluation. 

Reproducibility:
- Low. The authors do not mention making their code public and do not include it. 
summary_of_the_review: Overall, I think this paper proposes a promising idea, but in its current form it is not ready for publication at ICLR. I would highly encourage the authors to extend the empirical evaluation as described in my comments. If the improvements reported here hold up to a stronger empirical evaluation, I think this would make a nice paper at a top conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a mechanism to keep long-tail events in episodic memory to improve RL performance in Zipfian-like environments. Through contrastive learning, observations' rarity score is estimated and stored in a familiarity buffer. After certain training epochs, only the top rarest events from the buffer transit to the episodic memory, providing long-term context to the policy network following IMPALA architecture. The proposed method is tested in Zipf's Gridworld task and outperforms several variants of the IMPALA baseline, especially in handling long-tail maps. 
strength_and_weaknesses: ### Strength: 
Encouraging rare events in episodic memory is a reasonable idea. It fits well with the problem the paper aims to solve: Zipfian-like environments. Using contrastive learning to discover rare samples is well-motivated by long-tail learning literature. The experimental results show apparent performance gain in the 20% rarity case. Some baselines are wisely chosen to verify the contribution of the proposed idea.  

### Weakness:
The method is relatively incremental. It merely combines components such as IMPALA, episodic memory and contrastive learning. The new module is the familiarity buffer, whose design is not surprising given the paper's main idea (storing the rarest events). This idea is not new either and can date back to this paper [1]. 

Another problem is the writing and presentation. The introduction provides little information on the operation of the proposed method, which creates a gap in understanding the claimed contributions. The notations are confusing and sometimes overlapping. The algorithm is hard to read; one may not implement this algorithm to reproduce the method. 

Last but not least, the experiment is weak. Good results in a toy task and for only one setting (10 maps+10 objects) are not enough to validate the method. There are many additional hyperparameters. Since the performance changes as they vary (Table 2-5), it may require a lot of tuning to make this method work. 

### Detailed comments and questions:

- The second paragraph in the introduction seems to be about other papers (Sutton & Barto, 2018b; Espeholt et al., 2018). Please rewrite it to clarify your contribution. 
- Symbols like $tk$ can be written as $t_k$ for readability. T is used in both Eq. 2 and 3; are they different? What is N in Eq.4? What is v in Eq. 7?
- Sec. 4.2: IMPALA+MEM part should be described in detail in the background.
- The literature review is missing. Your references (e.g. Pritzel et al. (2017); Blundell et al. (2016)) are not up-to-date. It is better to have a separate section covering recent advances in episodic reinforcement learning. 
- Algorithm: Please link the functions to the section you describe them. 
- Fig. 1:  There should be k_t in the KNN part
- Fig. 4: There are some unexplainable cases. d) Contrastive learning can solve the rarest one (9-9) while your method fails. All methods fail in (5-1), which is far from the hardest. Why?
- Table 1: Why is your method better in the Uniform case? Is there any advantage? 
- Hyperparameter: Does changing the memory/familiarity buffer size matter?
- To strengthen the experiment, please consider the following:

   * Test in other environments that also have rare events or sparse rewards. Many Atari and Minigrid environments exhibit these properties. Although they may not be precisely Zipfian, the need for long-tail discovery remains.  
   * Test in different configurations of your toy environment by varying the environment's hyperparameters
   * Compare your contrastive learning with simple long-tail discovery methods, such as using VAE for novelty/outliner detection. 
   * Consider some other episodic control methods as the baselines.

[1] Frank, Jordan & Mannor, Shie & Precup, Doina. (2008). Reinforcement learning in the presence of rare events.  
clarity,_quality,_novelty_and_reproducibility: The writing is not clear. The method is incremental and lacks details to reproduce (see details in weakness)

summary_of_the_review: Although the motivation of the paper is good, its method is incremental and the result is insufficient. The writing needs major improvement.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This works presents a memory based method to deal with learning in Zipfian distributions, i.e., when some of the training data is rarely visited. To do so, authors start from an existing IMPALA + an Episodic Memory (MEM) method, and introduce a "familiarity buffer" that acts as a long-term memory and selects the top k rarest events to send them to the MEM, that acts like a shorter-spam or working memory.

Experiments shows that adding this familiarity buffer does indeed boost performance specially with rare events
strength_and_weaknesses: Authors make a great work introducing the topic of research motivations and goals. Empirical results evidence that the presented method boosts the ability of agents to track long-tailed events.

I believe there are two major weaknesses in this work. 

First, is the "method" section (Sec. 4), which is sometimes disjointed and could also improve with giving some intuition at some points. It also presents plenty of hyperparameters for the method, without detailing much about how sensible the proposed solution is to the hyperparameters. I'll detail further in the box below

Second, the experiment section is too narrow. In the sense that:
* It only contrast their extension of IMPALA+MEM against IMPALA and IMPALA+MEM, it would have been a great addition to compare with a completely different kind of memory-based solution like HCAM [1] to contrast. It would also greatly impact the significance of this work if authors demonstrated that the familiarity buffer can be integrated with other methods than MEM
* It would really benefit from ablation experiments. This would let the community know how influential is each element of your proposed solution, also it would clarify how sensible it is to the different hyperparameters.

[1] Lampinen, Andrew, et al. "Towards mental time travel: a hierarchical memory for reinforcement learning agents." Advances in Neural Information Processing Systems 34 (2021): 28182-28195.
clarity,_quality,_novelty_and_reproducibility: The paper is clear for the most part, although Section 4 needs some further work:
* It should clarify how sensible is the mthod to the different hyperparameters
*End of page 3, it should give a reference to the contrastive loss since it hasn't been introduced yet
* Page 4 line 3, "the augmented image" hasn't been explained yet
* It would be good to give some intuition there on why are you augmenting the image by adding noise and cutting a region
* In section 4.1 you explain the momentum Eq. 4 and the contrastive loss (Eq.5 and 6). But not how they are related, which makes it confusing
*Eq.7, you never explain what is v there
*Some intuition on equation 8 is missing too.
* Additionally, why using KNN and not the classic query-key match with hard attention? Hard attention is classically used in that context so would be good to motivate why authors did differently

Quality is a fair for the basic points, but I have major concerns due to the points I raised above about the method explanation and the lack of ablations and at least one baseline that is not the vanilla method authors are working with.

The novelty of the contributions feel incremental but fair. 

Authors give plenty of details for reproducibility in the appendix although there is no mention for the code 


summary_of_the_review: This work tackles a complex problem and a very interesting research line. The method seems promising but the works is still incomplete with several points to be tackled for the community understand the solution presented in this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for tackling RL with long-tailed distributions. It uses a self-supervised contrastive method to estimate state familiarity, and then adds the rare states to an episodic memory module based on prior work. The paper considers one task domain, in which it demonstrates compelling improvements from the proposed approach.
strength_and_weaknesses: Strengths:
The idea is interesting and compelling.
The results on the environment considered are impressive.

Weaknesses:
Using only a single environment makes the results somewhat limited. It would be nice to see experiments on another domain, to verify whether the results generalize. 
It would be very useful to consider more ablations to the procedure. 
Is the momentum in the familiarity necessary, or would using a simple average, or even just a single value of the loss give similar performance? While Zhou et al. presumably explored this in the image setting, it’s not clear that the results would be the same in RL.
Why these particular data augmentations, as opposed to others e.g. the others used in methods like SimCLR [Chen et al., 2020]?

Comments / notes:
The idea of identifying state familiarity through loss on an unsupervised method on those states is related to ideas that have been used to measure state novelty in exploration, such as in RND [Burda et al., 2018]. It might be worth making this link in the paper, because it suggests the possibility of more fertile overlap between these seemingly-disparate research streams in the future.
“Due to the unavailability of the code for the environment …” — The code for Zipf’s Gridworld task has been released (at least now) at https://github.com/deepmind/zipfian_environments/tree/main/gridworld — in fact, I’m using it in some present work. Might be worth updating this comment. The other environments could potentially be used to address the weakness noted above. 

References
------ 

Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018, September). Exploration by random network distillation. In International Conference on Learning Representations.

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1597-1607, 2020.

clarity,_quality,_novelty_and_reproducibility: The paper is moderately clear; I think the presentation could be improved in some places by adding more signposting to guide the reader. For example, at the beginning of section 4, adding a short paragraph to describe the pieces of the method and how they fit together would help frame the rest of the section.

The quality seems decent, though there are weaknesses as noted above.

The paper seems fairly novel—I have not seen similar approaches applied to long-tailed problems in RL.
summary_of_the_review: 
I think this paper offers an interesting idea and some compelling preliminary results. I think it would be a great workshop paper now, but I feel it would need experiments on one more domain and some more ablations to accept as a main conference paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors tackle the challenge of learning in RL environments with long-tailed distributions (tasks are not sampled iid but according to a Zipfian distribution). They propose a method which uses contrastive learning to determine how rare a state is in order to determine whether to store it in episodic memory. They tested on a variant of Chan et al’s Zipfian gridworld tasks and found improvement over standard IMPALA. 

Reviewers found it a well-motivated and reasoned idea, and the results on the presented environment are generally impressive. However many expressed major concerns that only a single environment was used, which limits the generalizability of their findings. Other issues brought up were that only a single type of memory agent was compared, and that the methods and overall presentation are a bit unclear. 

During the rebuttal period, the authors did update their paper to test on the original Zipfian gridworld environments, which had since been open-sourced, but this unfortunately didn’t satisfactorily address reviewers’ concerns about the limited nature of their tasks. One reviewer suggested converting existing environments such as the Atari suite, ProcGen, or Minihack into Zipfian ones by sampling from them under a long-tailed distribution, which I think is a great idea that would significantly enhance the quality of their experiments. 

Reviewers were relatively engaged with discussions, which I think were overall fruitful and helped to improve the paper’s clarity. While I cannot recommend acceptance at this time, I hope their comments will be helpful for future iterations of this work and I look forward to future iterations of it.

justification_for_why_not_higher_score: Reviewers all agreed that the experiments were too limited by being focused on only one environment. Lack of clarity was also a common issue brought up, indicating that this work in its current form is not ready for publication at ICLR.
Decision: Reject