Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper the authors identify two areas, namely policy over-“exploration” and value overestimation, where instabilities can arise in offline RL.

Using this knowledge, the authors propose two corresponding regularizations to ameliorate this issue and show competitive performance compared with state-of-the-art benchmarks.
main_review: Pros:
* I found the paper clearly written, and it motivates and identifies the issues that arise in offline RL well.
* The described approaches to fixing the identified extrapolation problems in offline RL are simple to follow and should be straightforward to implement onto existing algorithms.

Cons:
* Fundamentally there is not enough novelty introduced in this paper, and nor is it framed in a way that facilitates its distinguishing elements. For instance, the general discourse around extrapolation in offline RL is well-discussed in many prior works, so the paper is not the first to identify these issues (see [1,2], for instance). Regarding the actual ameliorations themselves, again variations of these have appeared in prior work; for instance the critic regularization is very similar to CQL, with two small modifications; a max is taken over the sampled actions (see Equation 2 of [3]) and a rectifier function that ensures positivity. Similarly, the actor regularization is already well discussed in [2,4], except a reverse-KL is chosen here to avoid learning a BC prior policy (or discriminator in a Fenchel dual form). My issue is that there is very little acknowledgement that the methods *heavily* lean on this prior work, and no ablations are provided to help understand whether the minor aforementioned modifications help or hinder compared to the original work (runtime, performance, etc.). Whilst I appreciate these two approaches (i.e., CQL + reverse KL policy constraint) haven’t been combined in this way before, that is not enough to merit acceptance. I think if a significant rewrite was done, framing it with a more explicit comparison with prior work (rather than just citing it in related work), that would improve my likelihood of accepting this paper.
* There are not enough seeds; 3 seeds is significantly below what is expected for the proprioceptive versions of MuJoCo benchmarks, and most current literature will use 5 and upwards, including more expensive model-based approaches.
* Important details, such as offline training epochs/iterations are omitted. The authors identified that OPE is important, and that in offline RL, performance can degrade with increasing training epochs due to over-extrapolation. However when reporting performance, did the authors simply stop training when they found the best online performance on the oracle environment, or did they select the best policy under their FQE critic as alluded to in Section 5.1?
* The results are not that compelling; it is true that on the benchmarks tested, CDC does appear the most performant, but in line with the seed issue above, it is unclear if this performance is truly universal. Furthermore, recent algorithms such as [5] aren’t included, nor are any model-based benchmarks, which are known to perform better than model-free equivalents (e.g., [6,7]).

Comments/Nits:
* The use of the word ‘exploration’ in the actor constraint is not ideal (i.e.,’exploration penalty’). The whole point of offline RL is that no exploration is possible, so claiming that a constraint reduces exploration when we literally have no interaction is confusing. I agree that if this algorithm was used in standard online RL, the constraint would likely reduce exploration, but this is precisely the point; exploration is the act of sampling those new states/trajectories through interaction. Using the term exploration in this pure offline problem setting gives the false impression we are able to do this sampling.

Conclusion:
Overall the paper is well written, and has nice elements to it, such as relative algorithmic simplicity, and clear problem framing of issues in offline RL. However, the actual details are relatively incremental compared to prior work (which isn’t made altogether clear in the manuscript and narrative), and indeed much of the discourse in this paper has been highlighted by previous work as well. I therefore believe this paper is below the level required for acceptance, and have given clear points in the cons section on how the authors’ should update their paper in order to move my position more towards acceptance.

[1] Off-Policy Deep Reinforcement Learning without Exploration, Fujimoto et al.

[2] Behavior Regularized Offline Reinforcement Learning, Wu et al.

[3] Conservative Q-Learning for Offline Reinforcement Learning, Kumar et al.

[4] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, Kumar et al.

[5] Offline Reinforcement Learning with Fisher Divergence Critic Regularization, Kostrikov et al.

[6] COMBO: Conservative Offline Model-Based Policy Optimization, Yu et al.

[7] MOReL : Model-Based Offline Reinforcement Learning, Kidambi et al.
limitations_and_societal_impact: No issues.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents two constraints for batch reinforcement learning. One is a Q regularizer; the other is a policy regularizer. The Q-value regularizer penalizes overestimation by ensuring out-of-distribution, policy-impacting Q values aren't too much larger than other Q values for a given state. The policy regularizer penalizes exploration by penalizing the divergence between the behavior and target policies.

These changes were tested on a suite of RL benchmarks using both ablation and comparison to other batch RL approaches. The performance of CDC was often superior.
main_review: The paper presented some very interesting results, but the limited number of runs (3) in the empirical section is a concern. The results seem novel and interesting, but can I rely on them with so few runs? I was impressed by how well written and clear the entire paper was. I have not worked through the appendix in detail to assess the theoretical foundation, but the theory described in the paper was framed well. I would love to see this paper published so I can share it and try it, but I think it needs more runs first.

Regarding **originality**, I'm not familiar enough with the field of batch RL to know how original it is. But the authors do an excellent job of describing how the two regularizers are novel, yet grounded in existing work. 

**Quality** is my biggest concern. I was originally concerned about the number of runs, but the authors pointed out batch RL requires fewer runs than online RL. The lack of error bars in Figure 2 leaves me concerned about whether any differences are significant.
I believe the claims are supported by theory. However, the majority of the theory was relegated to Appendix D, which I did not review.

Regarding **clarity**, the paper was very well written. It had a great flow between sections, it justified its problem, and it gave me the background I needed as I needed it. It was enjoyable to read, and I left with a deeper understanding of both the problem and where CDC fit as a solution. I believe enough information was provided to reproduce the results, though I hope code will be provided in the final submission.

Regarding **significance**, I believe these results are valuable and likely to be of benefit to the field. The results of these regularizers in both batch RL and offline evaluation are very interesting.

### Specific Notes

*Update: Many of these were resolved*

* In Figure 2, I'd like to see **error bars or confidence intervals**. Checklist 3c claims they were included, but they are not. (p-values are not the same as error bars and are not a good substitute.) I realize this might make the plots too busy (and your plots look great!) but it gives me more confidence in the results than a p-value. Update: the authors provided learning curves for CDC vs BCQ. That gave me more confidence in this point.
* I wasn't sure how to interpret **Figure 1**, including its units (e.g. is it normalized? an aggregate return or specific environment?) and how it was created. Plots on the first page are unusual, but this one clarified a point in the introduction, so it *might* be appropriate here. However, I don't recall seeing more details about it, even in the appendix, which is a problem.
* In **Figure 3**, I'm curious why some tasks are so much better or worse. When I look at Figure 2b and the task names in Table S1, it seems like CDC does best when the source data is poor. In cases when expert data is available, it's often not the best. I'd love to see more focus on this, since it's important in understanding when to pick it instead of another method. However, this could also be completely random; once again, adding error bars and more runs would make a big difference.
* There's a **typo** on line 351. I believe it should say, "For CDC, total normalized return..."
* The **ordering of Figures 2a and 2b** is the opposite of the text. Lines 350-356 discuss 2b, and then lines 357–363 discuss 2a. This was slightly jarring.
* Checklist 3d said the **total amount and type of compute** were included, but I couldn't find the *amount* of compute. How many hours did experiments take to complete?

### Update

The authors ran with extra seeds and pointed out that number of seeds for batch RL doesn't need to be as high as in online RL. And they  provided learning curves for CDC vs BCQ. That gave me more confidence in the (lack of) Figure 2 error bars.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This works proposes two regularization techniques that address severe extrapolation errors in offline RL. In various D4RL datasets, the method shows competitive performance compared to the previous state-of-the-art, outperforming it on average performance.
main_review: Pros
- The motivations behind the two regularization terms are solid.
- The proposed method performs well empirically.

Cons
- It is a bit unclear why the policy regularization and the Q-value regularization are both required. Conceptually, the policy regularization term will mitigate the Q-value overestimation, as OOD actions will be less sampled from equation (2). Likewise, the Q-value regularization term will lead the policy to be close to the behavior policy as the term encourages the dataset actions to have the maximum Q-value given a state.
- The hyperparameter setting is not fully specified. I strongly recommend the authors to specify the value of $\lambda$ and $\eta$, the two core parameters, used for **each** dataset. This is especially important for offline RL methods since evaluating the learned policy is difficult without online deployment.
- Also, the hyperparameter search range seems to be a bit arbitrary (e.g., {.1, .2, .5, .6, .8, 1, 5, ...} for $\eta$). Can the authors provide some results with regard to the hyperparameter sensitivity?

Other comments
- A concurrent work [1] also proposes to use a behavior cloning term for offline RL. While this does not hurt the originality of this paper as both works are concurrent, it is interesting that this paper and [1] have very different conclusions about the behavior cloning term. While Table S3 from this paper suggests the behavior cloning alone is not sufficient to train offline RL policies, [1] claims the behavior cloning term works well for all datasets (at least for D4RL Gym). Can the authors provide some intuitions on why the results are so different?

[1] A Minimalist Approach to Offline Reinforcement Learning, Fujimoto et al., arXiv 2021.
limitations_and_societal_impact: Limitations
- Listed above

Societal Impact
- N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes two regularizers for actor-critic RL methods to make them applicable for offline RL setting when only offline data is available. One regularizer is added to Q learning step in order to alleviate overestimation problem and one regularizer is added to policy optimisation step to avoid sampling out of distribution (OOD) actions. 
main_review: Originality:

To the best of my knowledge in prior works either a regularizer is added to policy optimization step or in critic learning step and not in both steps. So to me the main contribution of the paper is considering regularizing both policy evaluation and policy  optimisation steps at the same time. The other claimed contributions such as choice of the regularizer in each step is minimal given that adding BC loss to policy optimisation loss is not new and also the Q-penalty added to policy evaluation being explored in various prior works. Overall i think studying the dynamics of combining these high-level ideas from prior works is valuable.

Quality:

I think this paper does not support the idea of adding regularizers to both policy evaluation and policy optimisation steps well enough. There are some intuitive reasoning why the role of these two regulaizers are orthogonal but i'd like to see more principled reasoning that can explain what is the relation between these two regulaizers and in which condition both can have a same effect. This is crucial as we dont want to add a redundant hyper parameter to our RL algorithm.  I'm asking this because intuitively i think a regularizer on policy to stay close enough to the data should be able to avoid exploiting over-estimated Q values without a need for an extra regularizer in policy evaluation step. 

I'm a bit confused why two methods are employed to avoid overestimation here: 
    1- one by using ensemble of q networks (equation 5) 
    2- the other one by adding the proposed regularizer. 
How these two parallel methods influence eachother?

Adding a BC loss to RL loss makes a lot of sense but this has been proposed before in other prior works; for example see  SAC+BC in [1]. The question remains here is that how to choose the trade-off. It seems to me the RL loss and BC loss can be in a very different scale. How to deal with this issue?

Adding Q penalty to policy evaluation step is similar to what CQL does with the difference that here a different regularizer is proposed. Looking at the results in Table S2 it seems CQL outperforms the proposed method with \lambda=0.0 . This bring me to the question that why authors did not choose to use the CQL regularizer? In general it is inconclusive that the  choice of regularizer  for policy evaluation step is a good one compare to prior works.

Overall i think paper could do a better job in supporting the claims.

clarity:

This paper is written clearly and quite educational.

[1] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement
learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.


limitations_and_societal_impact: authors adequately has discussed the possible limitations and societal impact of their work in the paper.
review_title: null

Meta-review: The reviewers have carried out an extensive discussion on several points here, with several common themes being raised:
1) "Pitch" of the novelty with respect to the multitude of similarly motivated approaches; 
2) Need for further explanation or analysis on the core technical advance: combining value and policy regularization; and 
3) Further seeds, ablations, and experimental details.

With their responses, I believe the authors have nicely addressed 3), and with their reply text as well as several image links provided. The submitted method is reported to often be a top-performer on important benchmarks, and that the experiments have been done rather exhuastively relative to the recent competition in this field.

The discussion also offers hope that point 1) can be addressed, but this is harder to judge without seeing a new revision of the paper. Point 2) may be the most challenging and difficult to assess (concrete analysis of the real performance of deep RL is at an early stage overall). Ablations show "the combination works best", but I did agree with the reviewers that an expanded and unified discussion about how the two constraints may interact, would be valuable. Is it possible to explain how they are the right choices not independently but in combination?

In summary, this is an intriguing and valuable paper, but the writing enhancements requested are rather large, making it just slightly below threshold in the competitive NeurIPS2021 field. 

Decision: Accept (Poster)