Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a model-based approach to leverage episodic memory in reinforcement learning problems.
It aims to tackle three fundamental issues of memory-based methods, i.e., the near-deterministic assumption, sample inefficiency of memory construction, and the inflexible connection between parametric and non-parametric modules.
The experiments show that the proposed method outperforms baselines over a variety of environments.
main_review: **Originality**: To my knowledge, the proposed method is novel and the motivation is clear and convincing. I agree this paper aims to tackle some fundamental problems in EC-based approaches. The proposed method has good contributions for this track of research.

**Quality and Clarity**: The structure of this paper is well-organized. The proposed method is well-motivated and self-contained. The description of implementation details is acceptable.

**Significance**: This paper aims to address some critical issues in episodic control algorithms. The design of the proposed approach is reasonable. The experiment results are strong and significant. My concerns are discussed in the next section.
limitations_and_societal_impact: **Main Concerns**:

(1) Given the trajectory representation is changing during learning, how to ensure the key stored in the memory is consistent with the current trajectory encoding? What will happen if we find incorrect neighboring records?

(2) In some Atari games, the trajectory length might be very long, e.g., more than $10^4$ steps in a single trajectory. In this case, does the claim *"from that memory, we must be able to recall any past event"* still hold? How about the relation between model accuracy and learning performance in Atari games?

(3) If the above two problems matter, I am curious about to what extent the error will lead the memory to find incorrect neighbors.

**Minor Questions**: (I did not penalize my evaluation according to these points)

(1) Maybe there are some typos in Algorithm 1. I guess that line 12-16 should be included in the *for*-loop at line 4. In the current version, these two *if*-blocks are executed after the *for*-loop.

(2) A concurrent work [1] aims to address the second issue considered by this paper in continuous control settings. It is not required to compare performance with very recent papers, but a short discussion would be helpful.

Reference:

[1] Hu et al., Generalizable Episodic Memory for Deep Reinforcement Learning, ICML 2021.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims to combine episodic memory (where we store previous RL traces to act in the environment) with model-free/model-based RL (where we learn a global policy or value function to act in the environment). Their main contributions are that they 1) learn encodings for the trajectories, and 2) use the trajectories within a model-based planning procedure, and 3) adaptively integrate episodic memory with model-free RL based on a learned, tunable parameter. The paper has extensive experiments in small tasks, Atari 2600 and a first-person view POMDP, showing better performance than previous episodic control methods. 
main_review: Update after rebuttal:
I thank the authors for their rebuttal, but my main problem, about the clarity of the paper, unfortunately remains. I will therefore stick to my vote. 

Strong:
* Interesting and promising direction to combine episodic control with common model-free/model-based RL techniques.
* Extensive experiments. I also like the small-scale experiments with interpretability. Authors also include several ablation experiments. 
* I like the dynamic integration of model-free and episodic memory, i.e., learning the trade-off in a separate neural network. It indeed makes sense to adapt the contribution of episodic memory over training, since it can likely be high initially, but be decreased near convergence of the solution approximation.

Weak/possible improvements: 
* Clarity: I think your writing could be much clearer. I will give some example from the introduction:
    *You repeatedly mention ‘habitual’, but it took me some time to understand that you mean ‘model-free’ here, or ‘fast prediction of policy/value with function approximation. 
    * L30: “storing state-action value which demands experiencing all actions” → This is only true for tabular representations right, not if you profit from generalization?
    * L36 “the model” → What model? The method you introduce? I also do not really see in you explanation how this solves the stochasticity issue, you do not get back to it all? 
    * In L39: what is the difference between ‘the world’ and ‘the trajectories’. I can sample trajectories from a world model right? Explain such things.
    * L40-47: Again, I have a lot of trouble understand why this solves problem b, and what you are exactly proposing.


* Related work: Your related work is good when it comes to memory for partial observability. But, as you indicate as well, episodic memory is different. RNN’s for partial observability try to propagate/memorize relevant state representations, while episodic memory tries to also mimic previous actions, i.e., it is not just a state estimation technique. You mention this in line 99, but you do not explain the difference, nor give some related work in the other direction (although it is present in the introduction).
* Methods (Sec 3): At some points I have trouble following what happens in Sec 3. For example, section 3.2 introduces memory read, write and update procedures, but it is not clear too me when and why they are necessary. Overall, I often miss motivation why what you propose is necessary. I like Figure 1, but I think it needs more motivation, and then in your text you could get back to this figure and explain where each step happens, why it is relevant, and what considerations were important for your solution.
* Noisy transition (L232 and onwards): I am not convinced of your choice of a noisy transition, where you with random probability do not execute the action. This means that you still stay on the exact same trace, and you can simply try the action again at the next step. Thereby, it is arguably the minimal and simplest form of stochasticity. These experiments would be much stronger with a truly stochastic domain, not a deterministic one you slightly modified.
* Sec 4.4.: When you discuss results in your main paper, the results should also be visible there. Otherwise, state something like: “see appendix .. for additional experiments on a partially observable first-person view environment”. I think these experiments are really interesting and valuable, but then you should reserve the required space for them in the main paper. 
* Discussion: I miss a discussion. There is a one sentence remark about future work in the Conclusion, but no discussion of limitations of this work at all.  

Conclusion: I like the novel idea of this paper, which I think is relevant to the community. The experiments are also good and extensive. However, on the downside I do not think the paper is very clear (see the comments above). I especially sometimes miss a clear story line, why certain choices are made, how they solve existing problems, etc. This especially applies to the introduction and methodology section. I do think the paper could be improved a lot in this direction, and I think the paper deserves a discussion. I will therefore vote borderline here, although I do think the idea has much potential. 

limitations_and_societal_impact: I could not find a section on this.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a novel method for episodic control (EC) in reinforcment learning, combining several improvements aimed at targeting weakensses of current EC methods. These improvements are a trajectory-based (rather than state-based) representation for storage in memory, a model-based 1-step planning component for decicion making (including learning a reward model), and a dynamic combination of the non-parametric (memory-based) and parametric value estimate based on the current trajectory representation. Combined, these methods result in MBEC++. This method is compared against previous EC methods from the literature. First they evaluate methods in a variety of toy environments with specific properties to demonstrate that MBEC handles noise in the reward and transition function better that previous work and that the planning component is useful. They then test on the ALE, training for 10M steps as is standard practice in EC works, and show better results in terms of both median and mean performance, outperforming previous EC methods that were trained for 40M steps. Several ablations show the importance of the different components of MBEC++.
main_review: ## Overview

Overall, I believe the paper is a novel combination of new techniques for improving episodic control work. The problems with existing methods are presented and tackled by the new techniques, motivating them well. The experiments empirically demonstrate the superior performance of MBEC, and the ablations (including those in the appendix) provide insight into the importance of the different components. The main weaknesses of the paper are the writing clarity (there are numerous small grammatical errors, and some unclear writing and structure); the multitude of hyperparameters for their method, with relatively little intuition given about what good settings these might be when tackling new problems; and the lack of a focused related work section on previous EC methods.

## Strengths

Originality: I believe the paper presents an original method, combining original architectural components in a way that enables a novel planning algorithm. Particularly the formulation of the trajectory representation model to allow for 1-step lookahead planning for value estimation is novel and interesting. While dynamic modulation between parametric and non-parametric components has been done before (for example see [Adaptive Semiparametric Language Models](https://arxiv.org/abs/2102.02557)), it hasn't been done in the context of EC methods.

Quality: The method presented seems to be technically sound and is described in sufficient detail in the appendix and the main paper to be reproduced. The performance claims are empirically validated. 

Clarity: The method presented is complicated, which makes it difficult to present a simple explanation. I believe the explanation of the method presented in the paper is good but not great. Further details in the appendix clarified any confusions or uncertainties I had. There are multiple small grammatical errors which I haven't listed here, but which could probably be fixed by passing the paper through a rigorous online grammar checker (e.g. Google Docs, Grammarly).

Significance: I think the demonstration of the combination of model-based planning with model-free value estimation in an episodic control setting is useful and important, and it seems likely that future work in episodic control will use ideas from this paper or build upon it's method, possibly to integrate further model-based components or methods. I think episodic control in model-based RL is underexplored, and this paper is a first step in that direction.

## Weaknesses

As described, the main weaknesses I see with the paper are (1) writing clarity, (2) many hyperparameters, (3) lack of EC related work section.

I've made several suggestions for improvements below regarding (1). I also believe many grammatical errors could be fixed by passing the text through a grammar-checking software.

(2) is a more difficult weakness to address. I believe the paper is still sufficiently useful despite the large amount of additional hyperparameters, but the paper would be improved if there was more intuition and justification for how to choose these hyperparameters in different settings based on characteristics of the environment. For example, if I was to try this method in a novel environment, what heuristics could I use to pick good initial values for these hyperparameters? Which hyperparameters do you expect to be agnostic to the environment, and which should be changed per environment? Perhaps linking certain hyperparameters together (e.g. given a chosen K or L, what would be reasonable choices for N). These insights could probably be drawn from the existing experiments and ablations in the paper and appendix, or new experiments could be run to support any claims made.

For (3), I think an additional subsection in the Background section focusing on previous EC works would better situate the work in the existing literature. I think less time could be spend discussing DQN methods, given it's of relative little relevance to the rest of the paper, compared with previous EC methods. This section could also stress the weaknesses described in the paragraph starting on L28.

### Smaller Suggestions for improvement

  - I think it would be beneficial to clarify your usage of "episodic memory" as not a memory that's within an episode, but rather memory that's maintained across a lifetime. There's confusion in the literature on this terminology, as there is prior work that uses "episodic memory" to refer to within-episode memory (for example [Generalization of Reinforcement Learners with Working and Episodic Memory](https://arxiv.org/abs/1910.13406) and [Rapid Task-Solving in Novel Environments](https://arxiv.org/abs/2006.03662) both use Episodic Memory to refer to a within-episode memory component), and also the papers cited in your work which use "episodic memory" or "episodic control" to refer to an across-lifetime memory. Perhaps in one of the first few paragraphs this could be clarified with a short sentence, so future readers are clear from the outset what you mean by "episodic memory".
  - L28-34: It would motivate your argument more if you could cite or reference experimental evidence of these weaknesses in a more explicit way. For (a) I believe your own experiments support this claim, and possibly the same is true for (b) and (c). If that's the case I think it's worth rephrasing this paragraph to say that you demonstrate these are problems with current EC methods, which your method then addresses. If evidence is in prior literature then explaining how that is demonstrated, rather than just citing papers where the problems have been demonstrated, would be beneficial.
  - L130: It would clarify the algorithm explanation to describe what estimations of the true trajectories values are used here. (It is explained later on as well, but mentioning you used Monte Carlo estimations of the value here would improve the ease of understanding your method).
  - L139: I think it would be clearer to denote the estimated value by $\hat V(\vec\tau)$ (i.e. V rather than R). $R$ is often used as a reward function, and $R(\vec\tau)$ I'd intuitively read as the reward from the trajectory $\vec\tau$, whereas in the paper it's the remaining reward after completing trajectory $\vec\tau$
  - L148: The sentence starting "Following [22]..." is slightly confusing. I think it would be better as: "Following [22], we choose to write the trajectory representation of every L'th time-step (rather than every time-step) to save compuation while still maintaining good memorizaton." Otherwise it implies that you write to the memory every L time-steps, but you could still be writing every trajectory representation, just saving them up for L time-steps (as is done for example in Neural Episodic Control, where every time-step is written, but only at the end of the episode)
  - L151: The name "update" for the **memory update** function isn't the most informative, as it does a memory write just the same as the "write" function. I'd recommend changing it to "bootstrap_write", as the value you're writing is calculated with the 1-step bootstrap value estimate (in equation (3)).
  - L153: "As such, at an arbitrary timestep...". At this point I think it would be worth clarifying how do you choose the time-step? I believe the answer is you randomly decide to update with probability $p_u = 0.1 $, from the appendix section B.1 (L513)
  - Eq. 6 and 3 are basically identical - is there a way you could unify them to make it clear you're doing the same operation in both cases? I think this would clarify the algorithm further
  - L193: Without an explanation of the reward function for this environment (which is in the appendix but not the main body of the paper), understanding the optimal behaviour isn't possible. For example it's unclear why staying in the maze is beneficial (see L214) unless you know that touchingthe walls leads to negative reward. In the interest of making the paper self-contained, I think a brief description of the standard reward function for the maze environments in the main body of the paper would be beneficial.
  - L196: It's needlessly confusing to define the variable $n_e$ if the only value it takes in your paper is 3. I think you should just say you work in 3x3 mazes.
  - L275: It would further justify your method if you highlighted results on atari games that (a) were stochastic (and hence MFEC/NEC doesn't cope well but MBEC++ does), (b) where you believe the state-action-value storage is slowing does learning, (c) where the dynamic combination of parametric and non-parametric value estimates is useful and (d) where you think the model-based planning component is providing improvements.
  - L284: I believe you meant "state-action-based value estimations fall short..."?
  - L291, 293, Table 1: Ephemeral Value Adjustment is EVA, not EEVA.
  - Algorithm 1
      - L12: this and L13 should be indented to match the scope of the **for** loop.
      - L14: this line could be removed and lines 15-16 unindented for clarity (as it only happens at the end of each episode).
  - Figure 2
      - (b, upper): The logarithmic scales makes it unclear the actual improvement between MBEC and DQN here, which seems minimal visually but from the text of the paper and the graph below is substantial. Perhaps a different scale should be used.
      - (c): Why is the x-axis only a fragment of the full training run? This should be mentioned in the caption (to avoid confusion).

### Clarifications

  - L137: Instead of alternating between these reading at random, did you try mixing them? If not, do you have justification for why randomly alternating between them is the best choice?
  - L209: Does falling into the trap terminate the episode?
  - L216: If you're using a much larger memory for MFEC than you do for MBEC, then it's possible this late-stage degradation in performance is only due to this hyperparameter change, rather than your method. Also, do you have evidence that this is the cause of the worse performance for MFEC? For example, the average reward plateus at 45k steps for MFEC in Fig 2 (b, upper) - is this when the memory fills up?
  - On Atari results, you say you train for 10M frames. Do you mean 10M timesteps (which is 40M in-game frames, due to frame stacking?) If so, I think you should use "timesteps" rather than "frames"

## Conclusion

Overall, I believe as it stands the paper is marginally below acceptance (5). Making improvements on clarity or related work would move my score to marginally above acceptance (6). Further making improvements on intution and heuristics behind choosing hyperparameters, to compensate for the large amount of additional hyperparameters, would move my score to accept (7).

### Edit

Overall, I appreciate the authors efforts to improve the clarity of the paper, and working on the assumption that many of the comments or confusions in mine and other reviews will be addressed, I will raise my score to a 6. I think the current heurstics for picking hyperparameters aren't sufficiently justified to raise my score to a 7.

limitations_and_societal_impact: Societal impact is addressed (if only briefly). My concerns on the limitations of the paper are mainly centred on the large number of hyperparameters, as discussed in the main review above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to leverage episodic memory for improving the performance of control tasks. The proposed method uses estimated trajectory values for efficient policy learning. On top of the episodic memory control, they built a hybrid control system to combine the current learning algorithms. The experimental results are better than a few baselines in traditional control tasks and some of the atari benchmarks. 
main_review: The originality of this paper is good. This paper clearly clarifies the difference between their method with previous methods in this domain. 
The paper is well written. 
The significance of this paper is reasonable. Previous methods, even those value-based RL algorithms, might ignore the episodic nature of most of the benchmarks. In this paper, they utilize this fact and improve upon previous methods with an augmented trajectory memory.
While the experiments still need a bit of improvement, I would say the idea itself is valuable and inspiring for the field. So I would recommend for acceptance if a bit more evidence are provided to show the effectiveness of the proposed method.
limitations_and_societal_impact: The experimental section missed some strong model-based baselines  --- dreamer v1/v2,  SIMPLE and strong model-free baselines such as rainbow.  While this method might be developed from DQN and it already shows improvement upon it, it is still important to test the same idea on sota or more advanced methods since many ideas only work on some specific backbone algorithms.
review_title: null

Meta-review: I recommend accepting this paper.

Both the pre-discussion scores (5, 5, 6, 6) and post-discussion scores (5, 6, 6, 6) are borderline.

Reviewers agreed that there was a clear contribution, but the primary shared concerns were (1) writing clarity and (2) hyperparameter tuning:
* Reviewer R2hp gave the lowest score of 5, and wrote in the discussion: "Although the rebutall does point to some locations in the paper where unclarity might be resolved, I still think this was and is a major factor in my overall rating. The authors used much terminology which felt unnecessary, and I had trouble sticking to the main storyline." But the same reviewer also observed, "I like the novel idea of this paper, which I think is relevant to the community. [...] However, on the downside I do not think the paper is very clear. [...] I will therefore vote borderline here, although I do think the idea has much potential."
* Reviewer bLjg raised their score from 5 to 6, assuming the clarity revisions discussed will be implemented ("on the assumption that many of the comments or confusions in mine and other reviews will be addressed"), and underscored that hyperparameter tuning prevented a higher score ("the current heurstics for picking hyperparameters aren't sufficiently justified to raise my score to a 7").
* Reviewer m2e7 was concerned about strong baselines beyond just EC methods, and not satisfied with the author rebuttal, but these concerns were not shared by the other reviewers, and even with these concerns the reviewer leaned towards acceptance ("the idea itself is valuable and inspiring for the field", "this paper has some genuine contributions").
* Reviewer Crem did not participate in the discussion but the main concerns raised were in my estimation addressed in the rebuttal.

To summarize, while reviewers agreed on shortcomings in the paper which should be addressed (especially clarity), there is consensus on the paper having interesting contributions and an aggregate lean towards acceptance. For that reason I recommend acceptance, and also encourage the authors to implement the improvements discussed.
Decision: Accept (Poster)