Title: Continuous Doubly Constrained Batch Reinforcement Learning
Authors: Rasool Fakoor
Jonas Mueller
Kavosh Asadi
Pratik Chaudhari
Alex Smola
Venue: NeurIPS 2021
Keywords: batch reinforcement learning
overestimation bias
extrapolation
offline reinforcement learning
batch rl
offline rl

Abstract: Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of $32$ continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.

TL;DR: CDC is an effective batch RL algorithm that addresses overestimation bias and extrapolation errors by a simple pair of regularizers.

Review discussion:
Originality:
To the best of my knowledge in prior works either a regularizer is added to policy optimization step or in critic learning step and not in both steps.
So to me the main contribution of the paper is considering regularizing both policy evaluation and policy  optimisation steps at the same time. The other claimed contributions such as choice of the regularizer in each step is minimal given that adding BC loss to policy optimisation loss is not new and also the Q-penalty added to policy evaluation being explored in various prior works.
Overall i think studying the dynamics of combining these high-level ideas from prior works is valuable.
Quality:
I think this paper does not support the idea of adding regularizers to both policy evaluation and policy optimisation steps well enough. There are some intuitive reasoning why the role of these two regulaizers are orthogonal but i'd like to see more principled reasoning that can explain what is the relation between these two regulaizers and in which condition both can have a same effect. This is crucial as we dont want to add a redundant hyper parameter to our RL algorithm.  I'm asking this because intuitively i think a regularizer on policy to stay close enough to the data should be able to avoid exploiting over-estimated Q values without a need for an extra regularizer in policy evaluation step.
I'm a bit confused why two methods are employed to avoid overestimation here:
1- one by using ensemble of q networks (equation 5)
2- the other one by adding the proposed regularizer.
How these two parallel methods influence eachother?
Adding a BC loss to RL loss makes a lot of sense but this has been proposed before in other prior works;
for example see  SAC+BC in [1]. The question remains here is that how to choose the trade-off. It seems to me the RL loss and BC loss can be in a very different scale. How to deal with this issue?
Adding Q penalty to policy evaluation step is similar to what CQL does with the difference that here a different regularizer is proposed. Looking at the results in Table S2 it seems CQL outperforms the proposed method with \lambda=0.0 . This bring me to the question that why authors did not choose to use the CQL regularizer? In general it is inconclusive that the  choice of regularizer  for policy evaluation step is a good one compare to prior works.
Overall i think paper could do a better job in supporting the claims.
clarity:
This paper is written clearly and quite educational.
[1] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement
learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
No
Thank you for your feedback.
> I think this paper does not support the idea of adding regularizers to both policy evaluation and policy optimisation steps well enough.
Although our paper demonstrated that each regularizer can be used independently and their implementation is modular, we emphasize that they complement each other: the value-update mitigates extra-overestimation error while the policy regularizer ensures candidate policies do not stray too far from the offline data. Our ablation studies clearly show (see Figure 2a and Table S2) the best performance will be achieved only when both co-exist. We value the simplicity of our regularization terms: each penalty can be added to existing actor-critic RL frameworks with minimal extra code and the addition of both penalties involves no further complexity beyond the sum of the parts.
> I'm a bit confused why two methods are employed to avoid overestimation here: 1- one by using ensemble of q networks (equation 5) 2- the other one by adding the proposed regularizer. How these two parallel methods influence each other?
In this paper, we show that using Q-ensemble alone fails to mitigate extra-overestimation as it is a severe issue in batch RL.
Although Q-ensemble helps lessen this issue to some extent, the lower confidence bound Q in (equation 5) cannot, on its own, resolve extra-overestimation, unless we use extremely pessimistic Q-estimates that no longer faithfully value different actions.
We provide results for $\eta, \lambda =0$ (see Fig 2a and Table S2), which is a standard off-policy actor-critic adapted to batch RL without our CDC enhancements.
Significantly worse performance is obtained without our proposed enhancements (CDC is 9x better), despite this base case still utilizing Q-ensemble.
> Adding a BC loss to RL loss makes a lot of sense but this has been proposed before in other prior works;
for example see SAC+BC in [1]. The question remains here is that how to choose the trade-off. It seems to me the RL loss and BC loss can be in a very different scale. How to deal with this issue?
Thanks for the pointer. As we discussed in the paper, the goal of policy regularization is to ensure candidate policies do not stray too far from the offline data.
While it is an important piece, it doesn’t help with extra-overestimation.
Good observation.
While it is true that our policy loss and MLE can be on different scales, we didn’t observe any major issues while training our methods in that regard. That being said, normalizing Q-values to be the same scale as action space can further stabilize the policy learning process or eventually learning process.
While we haven’t looked into this problem, the ideas from a method like PopArt* can be a good starting point to think about this problem.
* PopArt: https://arxiv.org/pdf/1602.07714.pdf
> Adding Q penalty to policy evaluation step is similar to what CQL does with the difference that here a different regularizer is proposed. Looking at the results in Table S2 it seems CQL outperforms the proposed method with \lambda=0.0 . This bring me to the question that why authors did not choose to use the CQL regularizer? In general it is inconclusive that the choice of regularizer for policy evaluation step is a good one compare to prior works.
The main goal of this work is to propose a new and simple batch RL approach that extends standard off-policy actor-critic RL methods to batch setting by adding a simple pair of regularizers without significant changes to the underlying algorithm. Figure 2a and Tables S2 show that both penalties are critical for the strong performance of CDC, with the extra-overestimation penalty being of greater importance than policy constraint.
Pros
- The motivations behind the two regularization terms are solid.
- The proposed method performs well empirically.
Cons
- It is a bit unclear why the policy regularization and the Q-value regularization are both required. Conceptually, the policy regularization term will mitigate the Q-value overestimation, as OOD actions will be less sampled from equation (2).
Likewise, the Q-value regularization term will lead the policy to be close to the behavior policy as the term encourages the dataset actions to have the maximum Q-value given a state.
- The hyperparameter setting is not fully specified. I strongly recommend the authors to specify the value of $\lambda$ and $\eta$, the two core parameters, used for **each** dataset. This is especially important for offline RL methods since evaluating the learned policy is difficult without online deployment.
- Also, the hyperparameter search range seems to be a bit arbitrary (e.g., {.1, .2, .5, .6, .8, 1, 5, ...} for $\eta$). Can the authors provide some results with regard to the hyperparameter sensitivity?
Other comments
- A concurrent work [1] also proposes to use a behavior cloning term for offline RL.
While this does not hurt the originality of this paper as both works are concurrent, it is interesting that this paper and [1] have very different conclusions about the behavior cloning term.
While Table S3 from this paper suggests the behavior cloning alone is not sufficient to train offline RL policies, [1] claims the behavior cloning term works well for all datasets (at least for D4RL Gym). Can the authors provide some intuitions on why the results are so different?
[1] A Minimalist Approach to Offline Reinforcement Learning, Fujimoto et al., arXiv 2021.
No
Thank you for your feedback.
> It is a bit unclear why the policy regularization and the Q-value regularization are both required. Conceptually, the policy regularization term will mitigate the Q-value overestimation, as OOD actions will be less sampled from equation (2).
Likewise, the Q-value regularization term will lead the policy to be close to the behavior policy as the term encourages the dataset actions to have the maximum Q-value given a state.
Although our paper demonstrated that each regularizer can be used independently and their implementation is modular, we emphasize that they complement each other: the value-update mitigates extra-overestimation error while the policy regularizer ensures candidate policies do not stray too far from the offline data. Our ablation studies clearly show (see Figure 2a and Table S2) the best performance will be achieved only when both co-exist. We value the simplicity of our regularization terms: each penalty can be added to existing actor-critic RL frameworks with minimal extra code and the addition of both penalties involves no further complexity beyond the sum of the parts.
Compared to policy-constrained batch RL, CDC doesn't need to severely constrain candidate policies around behavior policy, which reduces achievable returns. Even in the presence of a policy constraint, the resulting policy is still affected by the learned Q-value, so we still must correct Q-value issues.
Specifically, we must restrict extra-overestimation in Q-values, albeit only where it matters, that is, only when this leads to a policy exploiting overly optimistic estimates.
As such, we only need to penalize suspiciously large Q-values for actions potentially selected by our candidate policy (e.g. if their estimated Q-value greatly exceeds the Q-value of actually observed actions).
> The hyperparameter setting and hyperparameter search range.
We use the exact same network architecture, optimizer (ie. Adam), and network size (ie. 4 layers, 256-hidden size) for all environments. Note that CDC works well using the same architecture and training procedure (optimizer hyperparams) for all environments without the need for different reward normalization, action clipping, etc per domain/task.
Like most other batch-RL baselines in our comparisons and following Section 8 in Agarwal et. al.**, we did a minimal random search over choices in Table S5 to tune our hyperparams $\eta$ and $\lambda$.
**Agarwal et. al.: An Optimistic Perspective on Offline Reinforcement Learning
> Compare with “A Minimalist Approach to Offline Reinforcement Learning”
Thank you for pointing out ‘A Minimalist Approach to Offline Reinforcement Learning’. We were unaware of this work as it was only posted to Arxiv in June 2021, two weeks after the Neurips submission deadline.
While the result is interesting to some extent and it needs a closer examination, it is unfortunate that they only report results for D4RL Gym. Policy constraint methods like BCQ perform well on D4RL Gym and to our surprise, BCQ has not been included in that paper. It would have been conclusive if the authors reported on more challenging environments like AntMaze, Adroit, and FrankaKitchen to see if their hypothesis holds.
For example, AntMaze poses a more challenging scenario for a batch RL method due to its dataset composition (i.e. a batch RL method needs to stitch together pieces of existing observed trajectories to solve a given task) and its high dimensionality. Without that, it is hard to draw a clear conclusion.
On the other hand, we examined our method, CDC, not only on D4RL Gym but also on AntMaze, Adroit, and FrankaKitchen which provide far more conclusive results.
Thank you for the response.
While the response addresses some of my concerns, I am worried that the authors do not provide the detailed hyperparameter settings for their algorithm ($\eta$ and $\lambda$). I did not ask the authors about the choice of network architecture or the training procedure. To quote from my first review, I asked the authors to **specify the value of $\eta$ and $\lambda$, the two core parameters, used for each dataset**.
However, the authors did not provide any detailed response.
Also, the authors did not provide results with regard to hyperparameter sensitivity. I concern this may indicate the proposed algorithm is highly sensitive to its hyperparameters.
Thanks again for your comment. We’d like to acknowledge that we misinterpreted the reviewer’s comment in regard to the hyper-parameter settings and we apologize for that. To address the reviewer comment, we’ve updated Table S5 and showed those parameters per environment where index (index:value) indicates environments as in Table S1 (Note these hyper-parameters are same as before, but are shown per environment now) Policy constraint coef: {0:.1,1:.1,2:1,3:1,4:1,5:.1,6:.5,7:1,8:1,9:1,10:.1,11:.1,12:1,13:1,14:1,15:.5,16:1,17:.1,18:1,19:1,20:1,21:1,22:1,23:1,24:2,25:1,26:1,27:1,28:1,29:1,30:.5,31:1}. Extra-overestimation coef {0:.1,1:.2,2:0.2,3:25,4:200,5:.1,6:1,7:.2,8:50,9:20,10:1,11:1,12:0.6,13:20,14:.8,15:.5,16:5,17:.5,18:.2,19:.1,20:0.1,21:200,22:25,23:1,24:100,25:25,26:100,27:100,28:25,29:50,30:5, 31:.6}. Following [1], we did a minimal random search to tune our hyper-parameters. Per our observations, policy coef can be assigned to 1 for all environments without significant change in performance.
Finally, to make it easier for others to reproduce our results, we will release the code for CDC with the final version of the paper.
In addition, we’d like to bring the reviewer attention to new ablation studies that we have run in response to other reviewers. These new results further highlight the utility of our both   regularizers. Please see our responses to Reviewer 4YAR.
The paper presented some very interesting results, but the limited number of runs (3) in the empirical section is a concern. The results seem novel and interesting, but can I rely on them with so few runs? I was impressed by how well written and clear the entire paper was. I have not worked through the appendix in detail to assess the theoretical foundation, but the theory described in the paper was framed well. I would love to see this paper published so I can share it and try it, but I think it needs more runs first.
Regarding **originality**, I'm not familiar enough with the field of batch RL to know how original it is.
But the authors do an excellent job of describing how the two regularizers are novel, yet grounded in existing work.
**Quality** is my biggest concern. I was originally concerned about the number of runs, but the authors pointed out batch RL requires fewer runs than online RL. The lack of error bars in Figure 2 leaves me concerned about whether any differences are significant.
I believe the claims are supported by theory.
However, the majority of the theory was relegated to Appendix D, which I did not review.
Regarding **clarity**, the paper was very well written. It had a great flow between sections, it justified its problem, and it gave me the background I needed as I needed it. It was enjoyable to read, and I left with a deeper understanding of both the problem and where CDC fit as a solution. I believe enough information was provided to reproduce the results, though I hope code will be provided in the final submission.
Regarding **significance**, I believe these results are valuable and likely to be of benefit to the field. The results of these regularizers in both batch RL and offline evaluation are very interesting.
### Specific Notes
*Update: Many of these were resolved*
* In Figure 2, I'd like to see **error bars or confidence intervals**. Checklist 3c claims they were included, but they are not. (p-values are not the same as error bars and are not a good substitute.) I realize this might make the plots too busy (and your plots look great!) but it gives me more confidence in the results than a p-value. Update: the authors provided learning curves for CDC vs BCQ. That gave me more confidence in this point.
* I wasn't sure how to interpret **Figure 1**, including its units (e.g. is it normalized? an aggregate return or specific environment?) and how it was created. Plots on the first page are unusual, but this one clarified a point in the introduction, so it *might* be appropriate here.
However, I don't recall seeing more details about it, even in the appendix, which is a problem.
* In **Figure 3**, I'm curious why some tasks are so much better or worse.
When I look at Figure 2b and the task names in Table S1, it seems like CDC does best when the source data is poor. In cases when expert data is available, it's often not the best. I'd love to see more focus on this, since it's important in understanding when to pick it instead of another method.
However, this could also be completely random; once again, adding error bars and more runs would make a big difference.
* There's a **typo** on line 351. I believe it should say, "For CDC, total normalized return..."
* The **ordering of Figures 2a and 2b** is the opposite of the text. Lines 350-356 discuss 2b, and then lines 357–363 discuss 2a. This was slightly jarring.
* Checklist 3d said the **total amount and type of compute** were included, but I couldn't find the *amount* of compute. How many hours did experiments take to complete?
### Update
The authors ran with extra seeds and pointed out that number of seeds for batch RL doesn't need to be as high as in online RL. And they  provided learning curves for CDC vs BCQ. That gave me more confidence in the (lack of) Figure 2 error bars.
No
Response:
Thank you for your feedback.
> I’m concerned about having only 3 random seeds and In Figure 2, I'd like to see error bars or confidence intervals. With only three runs, the performance improvement seen across the 31 tasks could be nothing more random.
Per the reviewer’s request, we repeat our experiments for CDC (our method), CQL, and BCQ for 5 different seeds and report results here: [ https://imgur.com/a/lG8uIMy ].
It is worth noting that batch RL papers often use 5 seeds for their experiments at the expense of fewer benchmark tasks. Since no further interaction with the environment is permitted, result variability in batch RL primarily stems from network initialization, stochastic gradient updates, and deployment time MDP variability.
Thus batch RL is less susceptible than online RL to problems that arise when using few seeds. As it is shown here: [ https://imgur.com/a/lG8uIMy ] our results are consistent across 3 and 5 seeds. We will update the paper with these numbers in the final version.
As you rightfully mentioned, it is very hard if not impossible to clearly visualize 32 tasks for 8 different methods in a single plot without sacrificing clarity.
Nevertheless, to give you a better picture of our method, we include the learning curves for our algorithm vs BCQ for all 32 games averaged over 5 seeds [ https://imgur.com/a/XLNxzbD ]. In these plots, the shaded area shows the standard deviation across 5 seeds. As these plots show, CDC has consistent performance across different seeds except for a few environments. We will update the paper with these numbers in the final version.
Lastly, we would like to draw your attention (again) to our comprehensive ablation studies (Figure 2a and Table S2). These results distinctly show that omitting our proposed penalties produces far worse returns than CDC, demonstrating the effectiveness of our proposed penalties. If there were any doubts about our method, these results would reassure that our method is effective and its performance is not by chance.
> I wasn't sure how to interpret Figure 1, including its units (e.g. is it normalized? an aggregate return or specific environment?) and how it was created.
Thanks for pointing out this ambiguity, it is easy fix to clarify this in the final version. Walker2d-expert is used to plot this figure. Figure 1b shows the learning curve for both CDC and standard actor-critic ( a.k.a. Algorithm 1 where $\eta=\lambda = 0$) and the y-axis shows average undiscounted and unnormalized return over the 10 evaluation episodes and 3 seeds.
> In Figure 3, I'm curious why some tasks are so much better or worse.
When I look at Figure 2b and the task names in Table S1, it seems like CDC does best when the source data is poor. In cases when expert data is available, it's often not the best. I'd love to see more focus on this, since it's important in understanding when to pick it instead of another method.
Good observation and good question.
When data is collected from a high-quality (expert) behavior policy, then behavioral cloning should be the method of choice as the behavioral policy is already optimal and we only need to replicate that policy (i.e. it simply becomes an imitation learning problem).
However behavioral cloning and policy-constraint methods fall short when a behavior policy is suboptimal or poor. This in fact is more closer to real-world applications where data can come from different sources with varying quality. This is where CDC ( and similar methods) will come to play and can play a significant role in these applications.
It is worth noting that CDC can be deployed in either scenario (i.e. expert data or poor data) by controlling the value of the penalties which is not the case with the policy constraint method.
> Checklist 3d said the total amount and type of compute were included, but I couldn't find the amount of compute. How many hours did experiments take to complete?
You are right, we only provided full details of computing infrastructure and software libraries used in this paper (Table S4) as our understanding was that this question concerns reproducibility where computing infrastructure and software libraries are more important than computing time. We have clarified:  Each experiment approximately takes 16-24 hours for 1 million training steps when a policy is evaluated every 5000 training steps.
Pros:
* I found the paper clearly written, and it motivates and identifies the issues that arise in offline RL well.
* The described approaches to fixing the identified extrapolation problems in offline RL are simple to follow and should be straightforward to implement onto existing algorithms.
Cons:
* Fundamentally there is not enough novelty introduced in this paper, and nor is it framed in a way that facilitates its distinguishing elements.
For instance, the general discourse around extrapolation in offline RL is well-discussed in many prior works, so the paper is not the first to identify these issues (see [1,2], for instance).
Regarding the actual ameliorations themselves, again variations of these have appeared in prior work;
for instance the critic regularization is very similar to CQL, with two small modifications; a max is taken over the sampled actions (see Equation 2 of [3]) and a rectifier function that ensures positivity.
Similarly, the actor regularization is already well discussed in [2,4], except a reverse-KL is chosen here to avoid learning a BC prior policy (or discriminator in a Fenchel dual form). My issue is that there is very little acknowledgement that the methods *heavily* lean on this prior work, and no ablations are provided to help understand whether the minor aforementioned modifications help or hinder compared to the original work (runtime, performance, etc.). Whilst I appreciate these two approaches (i.e., CQL + reverse KL policy constraint) haven’t been combined in this way before, that is not enough to merit acceptance. I think if a significant rewrite was done, framing it with a more explicit comparison with prior work (rather than just citing it in related work), that would improve my likelihood of accepting this paper.
* There are not enough seeds; 3 seeds is significantly below what is expected for the proprioceptive versions of MuJoCo benchmarks, and most current literature will use 5 and upwards, including more expensive model-based approaches.
* Important details, such as offline training epochs/iterations are omitted. The authors identified that OPE is important, and that in offline RL, performance can degrade with increasing training epochs due to over-extrapolation.
However when reporting performance, did the authors simply stop training when they found the best online performance on the oracle environment, or did they select the best policy under their FQE critic as alluded to in Section 5.1?
* The results are not that compelling;
it is true that on the benchmarks tested, CDC does appear the most performant, but in line with the seed issue above, it is unclear if this performance is truly universal.
Furthermore, recent algorithms such as [5] aren’t included, nor are any model-based benchmarks, which are known to perform better than model-free equivalents (e.g., [6,7]).
Comments/Nits:
* The use of the word ‘exploration’ in the actor constraint is not ideal (i.e.,’exploration penalty’). The whole point of offline RL is that no exploration is possible, so claiming that a constraint reduces exploration when we literally have no interaction is confusing. I agree that if this algorithm was used in standard online RL, the constraint would likely reduce exploration, but this is precisely the point; exploration is the act of sampling those new states/trajectories through interaction. Using the term exploration in this pure offline problem setting gives the false impression we are able to do this sampling.
Conclusion:
Overall the paper is well written, and has nice elements to it, such as relative algorithmic simplicity, and clear problem framing of issues in offline RL.
However, the actual details are relatively incremental compared to prior work (which isn’t made altogether clear in the manuscript and narrative), and indeed much of the discourse in this paper has been highlighted by previous work as well. I therefore believe this paper is below the level required for acceptance, and have given clear points in the cons section on how the authors’ should update their paper in order to move my position more towards acceptance.
[1] Off-Policy Deep Reinforcement Learning without Exploration, Fujimoto et al.
[2] Behavior Regularized Offline Reinforcement Learning, Wu et al.
[3] Conservative Q-Learning for Offline Reinforcement Learning, Kumar et al.
[4] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, Kumar et al.
[5] Offline Reinforcement Learning with Fisher Divergence Critic Regularization, Kostrikov et al.
[6] COMBO: Conservative Offline Model-Based Policy Optimization, Yu et al.
[7] MOReL : Model-Based Offline Reinforcement Learning, Kidambi et al.
No
Thank you for your feedback.
> Fundamentally there is not enough novelty introduced in this paper, and nor is it framed in a way that facilitates its distinguishing elements.
For instance, the general discourse around extrapolation in offline RL is well-discussed in many prior works, so the paper is not the first to identify these issues (see [1,2], for instance).
We emphasize that this paper never claimed to be the first to identify issues around extrapolation and overestimation in batch RL.
However, our paper highlights the underappreciated problem of extra-overestimation that causes all (not only OOD) Q-values to be distorted during training (due to propagation of extrapolated temporal difference errors). We find that the foundations of many batch RL algorithms: either overall pessimism in estimated values or conservatism enforced purely through policy constraints, are alone insufficient to mitigate issues caused by extra-overestimation, and we propose a superior strategy that combines policy/value regularization in a straightforward yet effective manner.
Our ablation with $\eta,\lambda = 0$ corresponds to using a standard pessimistic lower-confidence bound batch RL (akin to EMaQ, BCQ), which still performs quite poorly as it fails to resolve extra-overestimation.
While BCQ learns a constrained policy that tries to eliminate extrapolation by staying close to the data, it doesn't have a mechanism in place to deal with an extra-overestimation error once it happens (poor performance of BCQ in Table S1 vs CDC can be explained by that especially in low to medium quality data). Compared to policy-constrained batch RL (BCQ/BRAC/BEAR), CDC doesn't need to severely constrain candidate policies around $\pi_b$, which reduces achievable returns. Even with a strong policy constraint, the resulting policy is still affected by the learned Q-value, so we still must correct Q-value issues.
Lastly, CDC's $\Delta$ differs from the CQL penalty because CQL penalizes the Q-value for all proposed actions (its goal is pessimism, ie.\ to lower-bound the Q-value), whereas CDC only penalizes Q-values for proposed actions that were previously unobserved with large (possibly extrapolated) values (CDC only aims to mitigate extra-overestimation).
> Whilst I appreciate these two approaches (i.e. CQL + reverse KL policy constraint) haven’t been combined in this way before, that is not enough to merit acceptance.
The main goal of this work is to propose a new and simple batch RL approach that extends standard off-policy actor-critic RL methods by adding a simple pair of regularizers without significant changes to the underlying algorithm. Our goal is not to extend existing batch RL methods some of which may involve nontrivial components like variational autoencoders and other complex pieces. In our straightforward method, the first regularizer combats the extra-overestimation bias in regions that are out-of-distribution and the second is designed to hedge against the adverse effects of policy updates that severely diverge from $\pi_b(a|s)$. In our view, the simplicity of our work should not detract from its effectiveness.
Since the reviewer is interested to see how CQL works with behavioral cloning, we would like to refer reviewer to see CQL code: https://github.com/aviralkumar2907/CQL/blob/d67dbe9cf5d2b96e3b462b6146f249b3d6569796/d4rl/rlkit/torch/sac/cql.py#L184#L191 as the published version of CQL is in fact utilizing behavioral cloning for part of its training steps and their published results are obtained with behavioral cloning.
> no ablations are provided to help understand whether the minor aforementioned modifications help or hinder compared to the original work (runtime, performance, etc.).
We comprehensively studied the effect of each proposed regularizer in this paper. Please see Figure 2a in the main text and Table S2 in the appendix. These results show that the standard off-policy actor-critic adapted to batch RL without our proposed enhancements significantly achieve worse performance than our CDC (CDC is 9x better). We have included performance curves in the revised paper that illustrate our modifications do not have negative effects on the training process. We emphasize the extra computational complexity introduced through our simple penalties is extremely minor.
> There are not enough seeds; 3 seeds is significantly below what is expected for the proprioceptive versions of MuJoCo benchmarks
We repeated our experiments for CDC (our method), CQL, and BCQ for 5 seeds and report results here: [ https://imgur.com/a/lG8uIMy ].
It is worth noting that batch RL papers often use 5 seeds for their experiments at the expense of fewer benchmark tasks. Since no further interaction with the environment is permitted, result variability in batch RL primarily stems from network initialization, stochastic gradient updates, and deployment time MDP variability.
Thus batch RL is less susceptible than online RL to variability problems that arise in trying to draw conclusions from fewer seeds. As it is shown here: [ https://imgur.com/a/lG8uIMy ] our results are consistent across 3 and 5 seeds. We have updated the final version of our paper with these 5 seed numbers.
> Important details, such as offline training epochs/iterations are omitted.
CDC was trained for 1M steps without early stopping like BCQ, CQL, etc and we closely followed them to report results (see lines 621-642,  Table S5, S6 in our Appendix). We will further clarify these in the final version in addition to releasing the code.
> it is true that on the benchmarks tested, CDC does appear the most performant, but in line with the seed issue above, it is unclear if this performance is truly universal.
Furthermore, recent algorithms such as [5] aren’t included, nor are any model-based benchmarks, which are known to perform better than model-free equivalents (e.g., [6,7]).
It is unclear what is meant by “truly universal” in this comment. We have added clarification that:
We do not claim CDC to be the best possible batch RL algorithm for all use-cases, merely that it outperforms many existing model-free batch RL methods over many datasets in the large-scale D4RL benchmark which contains many diverse environments and datasets collected from widely varying behavior policies.
The reviewer has mentioned a number of drastically different algorithms as candidate baselines, some of which are model-based algorithms.  We are happy to compare---and have done so in the paper---with numerous existing algorithms that are similar to CDC, namely, CQL, BCQ, BRAC-V, BRAC-P, BEAR that are model-free algorithms. Comparing against model-based methods is not an apples to apples comparison as model-based methods are computationally more complex, and it is commonly accepted practice for model-free RL papers to compare against other model-free methods.
>  “Offline Reinforcement Learning with Fisher Divergence Critic Regularization, Kostrikov et al. ” not cited.
Thank you for pointing out the related work of Kostrikov et al., which we were originally unaware of as it was only posted to Arxiv in March 2021, two months before this submission deadline. We will add a discussion comparing our work with Kostrikov et al. to our revised paper.
> recent algorithms such as “Offline Reinforcement Learning with Fisher Divergence Critic Regularization, Kostrikov et al. ” aren’t included.
We believe our numerous baselines in the current paper provide a fairly thorough picture of the advantage of our methods vs others. Requiring comparison against a (non peer reviewed) paper posted to arxiv just two months before the submission deadline seems like an unfair expectation, where this would typically be deemed concurrent work.
> when reporting performance, did the authors simply stop training when they found the best online performance on the oracle environment, or did they select the best policy under their FQE critic as alluded to in Section 5.1?
No, CDC was trained for 1M steps without early stopping like previous methods.
> The use of the word ‘exploration’
We agree with the reviewer that use of the word ‘exploration’ may cause some confusion and have replaced this with clearer substitutes in the final version.
Thanks for the in depth reply, it certainly clarified a few things to the benefit of the paper. I am pleased to see the new seeds and clarity around policy evaluation.
If I now understand correctly, the main contribution lies in the fact that successful (model free) Offline RL requires *appropriate* regularization of *both* the policy and value function, whilst previous works have mainly focused on one or the other (for instance, in BRAC, value *or* policy regularization is tested, not both). I do believe this is a worthwhile contribution as such, and if the authors agree with this, then this should be made clearer in the abstract (and throughout), e.g.: “Whilst previous work has largely explored mitigating bias through penalizing either the value function or policy, we motivate and show that it is vital to penalize both *simultaneously*”. I believe this messaging should further be made clearer in the paper, in line with my comment about “facilitat(ing) its distinguishing elements”.
I appreciate the key differences between the authors CDC critic penalty and CQL, (indeed I wrote these differences down in my review), and I reiterate my point accordingly; is OOD action pessimism ill-suited in comparison to CDC’s *over*estimation penalty in their doubly constrained setting? And this leads me on to a more general point: I am aware there are ablations in Table S.2 regarding the CDCs *own two penalties*, but given a contribution of this paper is indeed these two *new* penalties, they should be compared more closely with prior work, and on a more counterfactual basis.
The interesting question for me is therefore this; an algorithm that penalizes both value function and policy will likely lead to different design choices than existing work (which mainly considers one or the other). I am therefore willing to believe that the two proposed penalties here are optimized for this doubly constrained approach (indeed the theory shows desirable properties), but have not seen enough evidence that existing approaches, which penalize one or the other, can’t just be ported together to get similar performance (e. g. , CQL + forward KL policy regularized BRAC). This is exactly my point about just combining two prior algorithms being not enough to merit acceptance; if the authors could show that there is a unique problem in their doubly constrained setting, and they have accounted for it (empirically), I’d be much more willing to move to acceptance.
A starting point for a set of experiments to assuage my concerns would be: a) to remove the rectifier in their critic loss to see if CQL-style pessimism is better than overestimation reduction; b) compare forward v. s. reverse KL policy penalties (my intuition is run-time could be favourable as no behaviour policy needs to be learned beforehand, but might affect performance with expert demonstrations, an issue that exists as pointed out by another reviewer); c) much is made of now having freedom to optimize the policy with looser BC constraints, but does this freedom truly pay dividends over an EMaQ approach of just sampling from a BC policy (e. g. , a column in Table S. 2 showing $\lambda \rightarrow \infty$)?
Whilst I was not aware of the CQL code-level detail (which runs counter to that paper’s narrative!), and thank the reviewers for pointing this out, I still think it’s not fair to say that comparing with CQL is the same experiment.
For instance, the CQL($\mathcal{H}$) they mainly use has softmax values/entropy regularization, which I don’t believe is used in the value update in CDC.
Furthermore, implementation level details, such as hidden layers (3 v.s. 4) are different, and given the sensitivity of RL algorithms, could result in performance differences orthogonal to algorithm design.
As it stands, I believe I see the merits of this paper more clearly now, but reiterate concerns about the clarity of messaging, and the novelty in discourse (I agree the discourse of direct over-extrapolation penalization is somewhat novel but not really enough on its own for me) and methodology. I believe these points are all interlinked, and hence if addressed in the ways I’ve suggested above, I certainly would consider revising my score upwards.
Thank you for your feedback and comments. We are glad the reviewer found this work “a worthwhile contribution” and our detailed responses and new experiments clarify and highlight the contributions of our work.
Unlike previous work, this paper highlighted the importance of simultaneous policy and value regularization in batch RL. Our CDC penalties are particularly simple to incorporate into actor-critic RL frameworks and operate synergistically as illustrated in the ablation study from Figure 2a and Table S2 (where for various environments, the performance boost from the simultaneous application of both penalties far exceeds the individual gains from either one).  We have further clarified this in the updated version.
Per the reviewer’s request, we have run additional ablation studies: 1) We removed the rectifier and square (called $\Omega$ in the figure) 2) $\lambda \rightarrow \infty$.  Updated ablation plot results is here: [  https://imgur.com/a/M9KIeo7  ] demonstrate that CDC outperforms both and further show the utility of both of our proposed penalties.
> they mainly use has softmax values/entropy regularization, which I don’t believe is used in the value update in CDC.
As stated before, our comprehensive ablation (please see Figure 2a and Table S2) clearly shows that CDC gains come from our proposed regularizers.
Significantly worse performance is obtained without our proposed regularizers (CDC is 9x better), despite the fact the only difference between CDC and these variants (i.e. $\lambda= 0$ $\\&$ $~\eta = 0$, $\eta = 0$, $\lambda=0$) in these experiments is either $\eta$ or $\lambda$ or both are set to zero in Algorithm 1 and all other details are exactly the same.
>  Furthermore, implementation level details, such as hidden layers (3 v.s. 4) are different, and given the sensitivity of RL algorithms, could result in performance differences orthogonal to algorithm design.
Our CDC implementation is exactly as described in the paper (Algorithm 1) and we try to be as transparent as possible about all implementation details and we’d like to emphasize that our experiment setup is exactly the same as the other batch RL method such as CQL/BCQ/BRAC/BEAR.
In addition, CQL uses 4 layers with 256-hidden too. Please see https://github.com/aviralkumar2907/CQL/blob/d67dbe9cf5d2b96e3b462b6146f249b3d6569796/d4rl/rlkit/torch/networks.py#L21#L62  . The hidden_sizes variable in their code is [256.256.256] (see https://github.com/aviralkumar2907/CQL/blob/d67dbe9cf5d2b96e3b462b6146f249b3d6569796/d4rl/examples/cql_mujoco_new.py#L38 ). Note three layers created in the for-loop and 1 layer after the for-loop.
Thanks for the reply, and the new results are certainly interesting. Clearly the rectifier has a positive impact, and I think this validates its difference from CQL. The BC ($\lambda\rightarrow\infty$) ablation is also appreciated, and gives me more confidence in CDC's qualitative improvement over something like EMaQ/BCQ.
I'm still intrigued about the forward v.s. reverse KL regularization in the policy, and whilst I appreciate the regularizers operate 'synergistically', more discussion about why they are uniquely suitable for each other would be nice.
I'll raise to a weak accept, as I believe now there is interest to the wider community (indeed the ablations now properly compare directly to some popular approaches, such as CQL and EMaQ, and the author's assuaged many of my niggling doubts), the elements I mentioned above still need a bit more discourse.
Furthermore, I do agree with another reviewer regarding hyperparameter sensitivity; it's good that the final chosen hyperparameters were evaluated fairly (e.g., 1m epochs), but did they require excessive querying of the 'true' online environment to discover in the first place?
Thank you for suggesting those experiments, comments, and for increasing the score. We've used them to greatly improve the paper.
> I'm still intrigued about the forward v.s. reverse KL regularization in the policy
We use the reverse KL direction because it’s simpler (no behavior policy estimation required) and works well in practice with our eta-penalty.
>  but did they require excessive querying of the 'true' online environment to discover in the first place?
No, it only required a few queries e.g. query after every 100k gradient updates and in total like 10 queries (10 queries = 1M training steps/100K) or even less for some environments.

Meta-review:
role: PC
data: comment: The reviewers have carried out an extensive discussion on several points here, with several common themes being raised:
1) "Pitch" of the novelty with respect to the multitude of similarly motivated approaches; 
2) Need for further explanation or analysis on the core technical advance: combining value and policy regularization; and 
3) Further seeds, ablations, and experimental details.

With their responses, I believe the authors have nicely addressed 3), and with their reply text as well as several image links provided. The submitted method is reported to often be a top-performer on important benchmarks, and that the experiments have been done rather exhuastively relative to the recent competition in this field.

The discussion also offers hope that point 1) can be addressed, but this is harder to judge without seeing a new revision of the paper. Point 2) may be the most challenging and difficult to assess (concrete analysis of the real performance of deep RL is at an early stage overall). Ablations show "the combination works best", but I did agree with the reviewers that an expanded and unified discussion about how the two constraints may interact, would be valuable. Is it possible to explain how they are the right choices not independently but in combination?

In summary, this is an intriguing and valuable paper, but the writing enhancements requested are rather large, making it just slightly below threshold in the competitive NeurIPS2021 field. 


Decision: Accept (Poster)