Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper empirically studies six variants of prioritized experience replay, typically used in online RL, in a batch RL setting. The comparison is performed using TD3BC on three D4RL Mujoco benchmark environment times 5 data sets. The experiments study the performance and bootstrapping errors. Among other things, it is shown that non-uniform sampling strategies are also interesting in a batch RL setting. The paper also discusses some shortcomings of these approaches and future directions.
main_review: **Strong points:**
- Sensible question: is a widely used method in online RL also useful in offline RL?
- While always inspired by online RL, I think at least some variants considered are not straightforward but rather an adaptation of an idea that works in online RL outside of PER. One example is using an idea that favors exploration in online RL to develop a sample weighting strategy.
- Empirical study are interesting imho.
- The paper offers interesting perspectives for further work.

**Weak points:**
- Imho, this paper assumes that the reader is extremely familiar with the state of the art in experience replay. I think this makes the paper perhaps harder than necessary to read. On the other hand, that type of paper must necessarily introduce a lot of algorithms quickly. I could not come up with a solution. Here are a few concrete examples:
   - Unless I missed it, some terms are never defined, such as behavior policy or temporal difference.
   - A lot of algorithms and variations are discussed and some are only introduced with a line of text. 
- The empirical experiments are well done, but limited in scope. The paper considers only one algorithm and three environments.
- The paper requires significant additional proofreading. A few typos are listed below.
- I saw no mention that the code would be provided, although the paper strongly emphasizes the empirical validation.

**Questions:**
- When reading the beginning of the paper, I thought it was an empirical evaluation of existing online RL strategies in batch RL. However, it seems that some new methods are also considered, such as Pseudo-Count for example. I am surprised this is not highlighted better as a contribution of the paper.

**Typos:**
- end of page 3: is could be the value
- p5: samples with higher n-step return is
- p5: optimizing following actor-critic loss functions
- p5: motivation of SIL is intuitive that
summary_of_the_review: Although mostly an empirical study, the research topic is interesting. I think the paper undersells a bit its contribution actually, as some variants considered are not trivial. It also provides interesting future directions. On the other hand, it is as bit hard to read, requires more proof-reading and the scale of the experiments is not big. Thus I believe it is a bit below the acceptance limit.

*After Rebuttal*: Thank you for your response. It seems that few changes have been made to the manuscript. I will keep my score unchanged. I wish the authors the best to improve the manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper empirically investigates several sample selection strategies in offline RL based on TD3BC and the PER framework, including TD errors, N-step return, Generalized SIL, Pseudo-count, Uncertainty, and Likelihood. The paper finds that some sampling strategies improve the performance on D4RL dataset but they fail to avoid the bootstrapping error. 
main_review: Strengths
1.	The paper is clearly written and easy to follow. It is self-contained and provides a good overview of existing literature.  
 
Weaknesses
1.	I think the findings in this paper are not novel and significant. I guess the paper uses different sampling strategies to run TD3BC until convergence or for a fixed number of steps. In this case, isn’t the only difference between different sampling strategies the weighting on the loss function? In the online setting, different sampling strategies not only affect the learning at each time step but also the transitions the agent would see in the future. However, in the offline setting, the dataset is fixed, so it seems like the only impact is the weighting on the loss function? If this is the case, the paper basically shows that different weighting results in different performance and I don’t think that is a novel or significant finding.  
2.	The empirical study is inconclusive. The paper mentions that the D4RL dataset is too complicated to draw any clear conclusion. Since the goal of the paper is not to propose a SOTA algorithm on “deep” RL environments, why not consider some smaller offline dataset or even discrete toy environments and see if any clear conclusion can be drawn? Moreover, in a small discrete environment, one might be able to show some relationship, e.g., using metric A performs well in environment B because it does C. I think that would be more informative than running on large complicated environments. 

Questions: 
1.  I am not sure what is the reason to use N-step return? It seems like N-step TD error might be a better choice since it matches the reasoning with PER. Also, what are the reasons for using Likelihood? It does not make sense to use only the state-action pairs closed to the behavior policy. 
2. I am not sure what is the bootstrapping error in Section 5 and how it is connected to $Q_1$ in Figure 2? 
3. The term “sampling selection” is confusing to me. Given a fixed dataset, even if we put a small sampling probability to a particular sample, we might still use the sample a sufficient number of times. I think it might be interesting to see a harder “sampling selection”: we drop a sample if the metric of the sample is below a threshold.  


summary_of_the_review: I am leaning towards a recommendation to reject the paper since the empirically findings in the paper are inconclusive and not significant as mentioned in the Main Review section. 

---
After Rebuttal: Thank you for the response. My concerns are not addressed and I think the paper is not ready for publication. I hope the authors can keep improving the paper since it is an interesting topic. I will keep my original score and evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigated the effect of non-uniform sampling in an offline RL setting. Using TD3BC (Fujimoto and Gu, 2021) as a backbone offline RL algorithm, the authors applied prioritized experience replay (PER) to the sampling of TD3AC with variants of priority metric, including standard TD error, rank-based return, pseudo-count using a hash table, and the other three metrics. The authors insist that non-uniform sampling can be helpful in offline RL compared with usual uniform sampling. They also found that there is no one outperforming metric for prioritized sampling in offline RL settings.
main_review: Strength

- Table 1 (List of proposed metrics) is well-explained and clarifies the considered sampling variants.


Weakness

- As shown in the numerical results, sampling in offline RL may not be a critical issue compared with online learning. Unlike online learning, most of the samples in a fixed dataset may be visited.

- The proposed method lacks novelty. This paper does not propose any new algorithm for offline RL settings. Instead, this work combines TD3BC with the existing six variants of metric.

- While TD3BC has an advantage in its simplicity in implementation, other backbone algorithms for model-free offline RL should also be considered. 


Question

- What will be the result if the following metric is used with proportional PER instead of rank PER: N-step Return, Uncertainty, and Likelihood? 
summary_of_the_review: Although this work raised an interesting question, this paper combines the existing algorithm and sampling metrics, which lack both novelty and empirical significance.


-------------------- Post Rebuttal ---------------------

Thanks for the response. I read the response carefully, and I maintain my original score. I hope the authors to further improve this work in the future.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper the authors' investigate the application of prioritized experience replay (PER) applied to offline/batch RL. They trial a variety of different priority metrics from the literature on the TD3+BC model-free algorithm and report results on the D4RL suite.
main_review: Pros:
* Overall the questions asked in this paper are worthwhile; I don't believe that PER has been applied to the offline setting before, so examining the efficacy of this is certainly an interesting direction.

Cons:
* This paper in its current state doesn't meet the standard for publication at a top venue as it lacks rigor for the following reasons:
   * Only one offline model-free RL algorithm was tested. More algorithms (e.g., CQL, MOPO, SAC etc.) should have been used. Perhaps using PER with a model-based method (and prioritising different hallucinated `(s,a,r,s')` tuples) could ameliorate the requirement of having a pessimistic MDP + reward penalty? Perhaps PER could be useful during model training?
   * Limited choice (6) of prioritization strategies; there should be more than this
   * Very low number of seeds (this should be at least 5)
   * No significance testing is done; are the blue highlighted results significant under a statistical test (e.g., Wilcoxon signed rank test)?
* The results themselves are not that compelling, and it doesn't appear that PER seems to help over uniform sampling much at all. I think a more directed piece of work that aims at creating a PER scheme that addresses distributional shift issues in offline RL, and show strong performance, would have been more suitable. Or perhaps work that seeks to explain why PER doesn't help that much, perhaps with some illustrative examples.
* The choice of citations is inappropriate at times. For instance, when model-based RL is introduced, the authors' cite recent work by Kaiser et al., but this is not a canonical nor an archetypal algorithm. Instead something like Dyna [1] or a review paper would be more appropriate. There are other examples of this littered throughout (e.g., citing Chua et al. for epistemic uncertainty), so the authors would do well to address this.

Nits:
* There are a few grammatical errors in this paper, for instance the use of 'pioneer' v.s. 'pioneering' at the top of section 3.2, and the repetition of 'metric' on page 9.

-------------------------------------

I have read the author's response but did not see much of what I requested in the updated manuscript (e.g., significance testing, additional algorithms, more compelling results, etc.). Therefore my rating stays the same.

Refs:

[1] "Integrated Architectures for Learning, Planning and Reacting based on Dynamic Programming", R. Sutton 1990.
summary_of_the_review: Overall this paper asks an interesting question, but fails in its execution for the reasons I have listed above. I believe it would be more suitable for publication at a workshop.

Based on this I recommend rejection.
review_title: null

Meta-review: The paper empirically benchmarks multiple sample selection strategies for offline RL based on the prioritized experience replay framework, including TD errors, N-step return, Generalized SIL, Pseudo-count, Uncertainty, and Likelihood. These are all benchmarked for the base algorithm TD3BC. The experiments study the performance and bootstrapping errors. Among other things, it is shown that non-uniform sampling strategies are also interesting in a batch RL setting. The authors show that non-uniform sampling can be helpful in offline RL compared to uniform sampling but they fail to avoid bootstrap error. They also found that there is no one outperforming metric for prioritized sampling in offline RL settings.

The reviewers are in agreement that the question studied is a sensible and interesting one - Are PER strategies which are effective in online RL also useful for batch RL? The overall study conducted by the paper is clear and well presented. 

While the study/benchmark and the results presented is clear, the reviewers point out the following shortcomings
1. The study is not comprehensive for this work to become a definitive exploration of this space of ideas. Only algorithm has been tested with these ideas. 
2. The results of the study are unfortunately inconclusive - while there are benefits these are achieved via different strategies and as mentioned by the paper no clear conclusions can be drawn. 

Since the paper is targeted purely as a benchmark, the originality aspect of the paper is naturally low. For benchmark papers in that case the impact factor squarely falls on comprehensiveness of the study and the emergence of some clear conclusions to further research in that area. The reviewers unanimously believe the paper falls short in both respects and therefore the decision. 

Hopefully the authors can consider the feedback provided and incorporate it to improve the paper.
Decision: Reject