Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes to set up experience replay such that transitions are emphasized based on whether the AN2N algorithm decides to explore more, and how recent the transition was experienced. It is based on observations that the state distribution can dramatically drift over the course of policy improvement, and the intuition that it might be better to perform updates on states that the current policy is more likely to actually visit (recent transitions). They apply the proposed sampling method, *experience replay more* (ERM), to a variety of replay-based deep RL algorithms and evaluate them empirically.
main_review: ## Strengths

1. The idea is simple, and has intuitive reasons for why it could help. With sufficiently large replay buffers and enough changes made to the policy, it de-emphasizes updates on really old/stale transitions that the optimal policy would rarely visit.

## Weaknesses

1. The paper suffers many spelling, grammatical, and clarity issues. It can be difficult to follow, and greatly detracts from the quality of the paper.

2. The paper acknowledges the PER algorithm and criticizes how such prioritization introduces bias which must be corrected by importance sampling. However, the proposed ERM algorithm (or any replay method in general) is similarly biased in that after each update, the data distribution gets increasingly off-policy. Further, despite acknowledgement of this uncorrected off-policyness, many works simply don't correct the data distribution and find that things still tend to work well empirically. Along this note, it seems like PER is a pretty important baseline to compare to in the space of methods which select which transitions to sample from the replay buffer- can the authors comment on why PER was not compared to, and whether the claims of being "state of the art" are reasonable when only comparing to vanilla experience replay?

3. Only 5 runs were used, and across most of the figures, the differences generally don't seem statistically significant. Can the authors comment on whether enough runs were performed, what the shaded regions represent, and whether conclusions can be drawn from the results with reasonable confidence? Similarly, the following line in the abstract is concerning with regards to empirical methodology: "DDPG with ERM can even exceed the average performance of SAC under certain random seeds"- it's not a particularly fair comparison to suggest that one algorithm on specific random seeds can outperform the *average* performance of another algorithm. That said, the Figure showing this comparison (7b) doesn't appear statistically significant and it's unclear whether this conclusion can be drawn from it.
summary_of_the_review: In light of the above concerns surrounding writing clarity, missing notable baselines, and questionable empirical methodology, I'm recommending strong rejection at this time. I believe a substantial rewrite and significantly more evaluation is needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to change the sampling order in the replay buffer by prioritizing sampling the "key" states and most recently generated states. "Key" states are the states where the agent performs poorly, measured by the return value.
main_review: Strengths:

* The paper motivates why the "key" states should be sampled more frequently.

* The proposed sampling scheme is simple to implement, and can be incorporated into many existing offline RL algorithms.

Weaknesses:

* This paper proposes a different sampling scheme than uniform sampling. However, in the experiments, no other sampling schemes are compared at all. The only baseline is uniform sampling. Even though the paper mentioned prioritized experience replay in the Introduction, it is not considered a baseline in the paper. 
* The proposed sampling scheme (ERM) is built upon AN2N (Add Noise to Noise). Therefore, there are at least two variants compared to normal sampling schemes: (1) add more noise to actions following AN2N during exploration (2) sample key states and recent states more frequently. In the experiments, we only see the comparison of normal RL algorithms and RL+ERM (for example, DDPG vs DDPG+ERM, SAC vs SAC+ERM). It is unclear whether the performance improvement (if any) comes from AN2N or AN2N + ERM. To answer this, the authors should include the learning curves of DDGP+AN2N, TD3+AN2N, SAC+AN2N.
* From Figure 4,5,6, we can see the benefits of using ERM are minor. The proposed sampling scheme does not lead to better performance in most cases.
* The paper lacks the discussions on how sensitive the proposed sampling scheme is to the extra hyperparameters that it introduces. For example, does $Prt_{AN2N}$ and $K_t$ need to be tuned a lot when the environments change? The paper should include some ablation studies to show how the performance changes as one changes these hyperparameters.
* The most challenging task in the paper is `Walker2D`. How about other more difficult benchmarking tasks such as Humanoid?
* The writing of the paper needs to be improved substantially. Grammar issues should at least be fixed.
summary_of_the_review: The paper needs more baselines, ablations, benchmarking tasks to justify the effectiveness of the proposed sampling scheme.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper presents Experience Replay More, an algorithm that prioritizes certain transitions in a replay buffer for replay. The approach is combined with multiple base RL algorithms in four MuJoCo domains.
main_review: Experience replay is a really important component of many RL algorithms right now. The importance of the distribution of replay states has been convincingly demonstrated in prior work. So the topic of this paper is definitely relevant to the ICLR audience.

One of my big concerns is that the paper is poorly written. The grammar, spelling, and word choice are all sufficiently flaws to, at times, obscure meaning. Even beyond that, the substance of the paper is not communicated clearly. While I understand that the idea of the paper is to more frequently replay states that are more important in some way, the paper does not communicate how those states are identified or what makes them more important. While the paper does a reasonable job of motivating prioritized replay in general, it does not clearly motivate or justify its specific approach.

The biggest clarity problem comes from the fact that ERM relies heavily on the AN2N algorithm, which has not been peer-reviewed and has only been available on arXiv for literally a few days at time of reviewing. Given this, the AN2N algorithm needs to be far more precisely described and justified. From the text of this paper, I have no idea how AN2N works or why I should believe that it is a good base to build on. This is critical, because the ERM algorithm makes its decisions based on the output of the AN2N algorithm. Without a clear description and justification of AN2N, the description and justification of ERM is also incomplete. Even if the AN2N paper were more clearly written and its hypotheses clearly evaluated (from a brief skim this does not seem to be the case), it is important that *this* paper be sufficiently self-contained that the reader can understand the significance of the contributions.

Here is a non-exhausted list of some other clarity concerns I encountered:
- Why is Hafner et al. the citation for model-based RL? MBRL predates 2019 by a long shot.
- I don't think it is correct to claim that off-policy model-free learning is categorically more sample efficient than on-policy learning. There are things you can do with off-policy learning that you can't really (justifiable) do with on-policy learning, like experience replay, that can improve sample efficiency. However, the same is true in the reverse; for instance eligibility traces are far more sensible in the on-policy case.
- Page 2: I didn't understand the logical step between discussing (prioritized) experience replay and meta-RL.
- Page 2: I don't understand what the footnotes are meant to communicate.
- Page 3/4: I don't understand Figure 1.
  - Isn't HalfCheetah continuous in state? So what do s0, s1, etc. represent?
  - Each line on the plot is associated with a particular Q-value (I think??), so what does the y-axis represent? The caption says that it is "the specific value of the state."
  - The caption says that "the difference is obvious at states s1, s8, and s12." I don't know what I'm supposed to be comparing in order to see a difference. And I don't see why those three states are special compared to any other states.
  - The main text concludes that "the state is also changing." I don't know what "the state"  refers to here, or what change is being discussed.

My other major concern is that the conclusions are far too strong for the experimental evidence presented. Only five independent trials are run for each algorithm and the error bars are quite large. I couldn't find the explanation of what the error bars actually represent (maybe I missed it), but assuming that it is something like the sample standard deviation the differences in performance are well within the noise in all but a couple of experiments. It is true that across all of the experiments there is a tendency for the ERM curve to be slightly above the non-ERM curve, which lends some credibility to the hypothesis that ERM is an improvement in at least some of these experiments, but even then, the improvement is *very* modest.

Furthermore, the empirical results do not test the central hypotheses of the paper. The paper claims that ERM's (modest) success is a result of specific effects, namely replaying more important/relevant states. It's important to evaluate this hypothesis, for instance by performing ERM but without some of its key features (e.g. without prioritization of recent experience or without the prioritization of "key" states). I also note that the comparison is always between ERM and the base RL algorithm. Since ERM is built directly on top of AN2N, it seems critical to include results using AN2N without the additional enhancements of ERM in order to understand the significance of the contributions of both algorithms.
summary_of_the_review: While investigating an important problem, the paper is poorly written, the main contribution is not adequately described or evaluated, and the empirical findings that are presented to not convincingly support the conclusions. I recommend that this paper be rejected.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new method (ERM) to bias the decision over which transitions should be replayed more often. In particular, using ERM, some states are deemed as key and they are replayed more often (following also a recency bias). This new method is then tested in 4 environments on the DeepMind control suite by using 3 different RL algorithms (DDPG, SAC, TD3).

main_review: - The paper is very difficult to read as it contains many typos and fragmented sentences. This problem is so acute that some paragraphs are almost incomprehensible. 
- It is also very unclear how the similarity between states is calculated. For instance in paragraph 3.1.1 this is referred to as a key step to define what transitions are considered ‘key’, but little details are given. This leaving aside that s_c and s_d are not defined in formal terms.
- A clear downside of the paper is that only a single domain is used (DM Control suite) and even worse ERM was tested in only 4 environments, thus questioning the generality of the method.
- Figure 4 highlights the results of DDPG with and without ERM. The only environment where there seems to be a small advantage of DDPG+ERM is HalfCheetah, although the error of the baseline is overlapping. So I strongly disagree with the statements of the last paragraph of section 4.1. Also, visually there is no sign of better data efficiency, and if the authors want to make a claim in this direction they should provide more quantitative measures (e.g. area under the curve)
- Figure 6 highlights the results of SAC with and without ERM. Again, the authors claim that SAC+ERM is better than SAC in half of the task, but this conclusion is not supported by the plots.
- Figure 7b is particularly weird as, in deep reinforcement learning, it is very unscientific to put draw any conclusions based on a single seed. 
- Table 1 reports some numbers in bold even though the variance is so high that determining what is the best method is a bit arbitrary. Authors should perform proper statistical analysis before highlighting scores in bold.
- Some baselines are completely missing from the experiments, e.g. baseline+prioritise replay (Schaul et al. 2016)

summary_of_the_review: None of the claims in the abstract, introduction and conclusion are actually supported by the results. Also the paper is not well written, with lots of typos and fragmented sentences.
review_title: null

Meta-review: The paper proposes an interesting way of prioritizing samples in replay that is compatible with many RL methods. It is evaluated experimentally on different tasks and with different RL algorithms.

The reviewers highly appreciated the revised paper and the detailed replies and discussions.
While this iteration improved the paper substantially, it is still not ready for publication in its current form. In particular:
- The paper is still not self-contained enough
- The reviewers are still not convinced about the statistical significance
- More tasks should be added
- PER needs to be added as a baseline
The authors promised those changes for the final version, but those are so substantive that the paper will need to go thorough another complete review cycle. Hence, we'd like to encourage the authors to re-submit at a different venue.

P.S.: Careful with double-blind submissions, acknowledgements should not be included.
Decision: Reject