Continuous Doubly Constrained Batch Reinforcement Learning
Authors: Rasool Fakoor
Jonas Mueller
Kavosh Asadi
Pratik Chaudhari
Alex Smola
Venue: NeurIPS 2021 Poster (2021)
Keywords: batch reinforcement learning
overestimation bias
extrapolation
offline reinforcement learning
batch rl
offline rl

Abstract: Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of $32$ continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.

Continuous Doubly Constrained Batch Reinforcement Learning
Rasool Fakoor', .onas Mueller', Kavosh Asadi', Pratik Chaudhari 1.2, Alexander J. Smola *Amazon Web Services, 2 University of Pennsylvania fakoor@amazon.com com
Abstract
Reliant on too many experiments learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and value-constrain that discourages overly optimistic estimates. Over a comprehensive set of 32 continuous action batch RL benchmarks, our approach compares favorably to state-of-the- methods, regardless of how the offline data were collected.
Introduction
Deep RL algorithms have demonstrated impres- sive performance in simulable digital environ- ments like video games [41, 54, 55]. In these settings, the agent execute different policies and observe their performance. Barring a few ex- amples [37], advancements have not translated quite as well to real-world environments, where it is typically infeasible experience millions of environmental interactions [11]. Moreover, in presence of an acceptable heuristic, it is in- appropriate to deploy an agent that learns from scratch hoping that may eventually outperform the heuristic after sufficient experimentation.
Daanket Aebee (oun)
Average Q-values
6.25
8.50
0.35
150
Herations
0.C C.)
6.7 130 122
Figure 1: Batch RL with CDC vs. No CDC. Left: Stan- dard actor-critic overestimates Q-values whereas CDC estimates are well controlled. Right: Wild overestima- tion leads to worse- porse-performing policies whereas CDC performs well.
The setting of batch offline RL instead offers policies for real-world applications [34, 57]. Batch RL is widely applicable because this setting does not require that: a proposed policy be tested through real environment interactions, or that data be collected under a particular policy. Instead, the agent only has access to a fixed dataset D collected through actions taken according some unknown behavior policy mp- The main challenge in this setting is that data may only span small subset of the possible state-action pairs. Worst yet, the agent cannot observe the effects of novel out-of-distribution (OOD) state-action combinations that, by definition, are not present in D.
pertinent framework to learn performant
A key challenge stems from the inherent uncertainty when learning from limited data [28, 36). Failure to account for this can lead to wild extrapolation [17, 29] and over/under-estimatioo bias in value
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

estimates [22, 23, 32, 58]. This is systemic problem that is exacerbated for (OOD) state-actions where data is scaree. Standard temporal difference updates to Q-values rely on the Bellman optimality operator which implies these updates. As Q-values are updated with overestimated targets, they become upwardly biased even for state-actions well-represented in DD In turn, this can further increase the upper limit of the extrapolation errors at OOD state-actions, which forms a vicious cycle of extrapolationiinfal overestimation eextra-overestimaaiion for short) shown in Figure 1. This extra-overesimmatio is much more severe than the usual overestimation bias encountered in online RL [22. 58]. As such, we critically need to constrain value estimates whenever they lead to situations that look potentially 'too good to be true', in particular when they occur where a policy might exploit them.
estimates tend to dominate
Likewise, naive exploration can lead to policies that diverge significantly from TT. This, in turn, leads to even greater estimation error since we have very little data in this un(der)-explored space. Note that this is not reason for particular concern in online RL: after all, once we are done exploring a region of the space that turns out to be less promising than we thought, we simply update the value function and stop visiting or visit rarely. Not so in batch RL where we cannot adjust our policy based on observing its actual effects in the environment. These issues are exacerbated for applications with large number of possible states and actions, such as the continuous settings considered in this work. Since there is no opportunity to try out a proposed policy in batch RL, learning must remain appropriately conservative for the policy to have reasonable effects when it is later actually deployed. Standard regularization techniques are leveraged in supervised learning to address such ill-specified estimation problems, and have been employed in the RL setting well [12, 50, 62).
This paper adapts standard off-policy actor-critic RL. the batch setting by adding a simple pair of regularizers. In particular, our main contribution is to introduce two novel batch-RL regularizers: The first regularizer combats the extra-overestimation bias in regions that are out-of-distribution. The second regularizer is designed to hedge against the adverse effects of policy updates that severly diverge from (a(ss)) The resultant method, Continuous Doubly Constrained Batch RL (CDC) exhibits state-of-the-ard performance across 32 continuous control tasks from the D4RL benchmark [14] demonstrating the usefulness of our regularizers for batch RL.
2 Background
Here S is the state space, A C TR.d is a (continuous) action space, T S x A x S - R. encodes transition probabilities of the MDP, (4o denotes the initial state distribution, r(s, a) is the instantaneous reward obtained by taking action E A in state E S, and 7 E [0, 1] is discount factor for future rewards.
Consider an infinite-horizon Markov Decision Process (MDP) [48],
Given a stochastic policy m(ass), the sum of discounted rewards generated by taking series of actions ae ~ π x11(st) corresponds to the return R; = Ee - "r(s,, a;) achieved under policy æ,, Q²=(s, a), is defined as the expected return starting at state s, taking a, and acting according to π thereafter, (=(ssa) = E, Exc-Taar 1E yrr, (So, ag) = (s, a)). (T, a) obeys the Bellman equation [5]:
л. The action-value function (Q-value for short) corresponding
(s, r(s, a) + yE -BB-TT(laa))) (1s (s',aa))
(1)
Unlike in online RL, no interactions with the environment allowed here, so the agent does not have the luxury of exploration. D is previously collected via actions taken according to some unknown behavior policy To ma(a)s). In this work, we assume D consists of 1-step transition: {(s,, a,, Ti. of = where no further sample collection is permitted. In particular, our method, like [60, 63), only needs a dataset consisting of a single-step transitions and does not require complete episode trajectories. This is valuable, for instance, whenever data privacy and sharing restrictions prevent the use of the latter [34]. is also useful when combining data from sources where the interaction still in progress, e.g. from ongoing user interactions.
We aim to learn optimal policy x² that maximizes the expected return, denoting the corresponding Q-values for this policy as Q' QF" Q" is the fixed point of the Bellman optimality operator [5]: TQ' (s. a) = r(s. a) + ((, [max, (ss,''')) One way to learn x² is via actor-critic methods [27], with policy no and Q-value Qo. parametrized by and respectively.
Learning good policies becomes far more difficult in batch RL as depends on the quality/quantity of available data. Moreover, for continuous control the set of possible actions is infinite, making

---

nontrivial to find the optimal action even for online RL. One option is to approximate the maximization above by only considering finitely many actions sampled from some x. This leads the Expected Max-Q (FMaQ) operator of Ghasemipour et al. [18]:
TQ a) == r(s, a) + ((1s.a) max (s' ab) {of}}
(2)
Here dg. ~ [To(-ss') for l = L,.., NN, i.. the candidate actions are drawn IID from the current (stochastic) policy rather than over all possible actions. When drawing only single sample from T this reduces to the standard Bellman operator (in expectation). Conversely, when N + oo and me has support over A, this turns into the Bellman optimality operator. We learn Q by minimizing the standard 1-step temporal difference (TD) error.
That is, we update at iteration
O₂ f- argmin B(s.))DD (Qunss.)) TQQ, (8, a)) 27
(3)
Throughout, the notation Es (a.D)) denotes an empirical expectation over dataset D, whereas expec- tations with respect to are taken over the true underlying distribution corresponding to policy Next, we update the policy by increasing the likelihood of actions with higher Q-values:
Φ f argmax E (s
Qo. (s. a)]
(4)
using off-policy gradient-based updates [53]. Depending on the context, we omit t from Qo and To.
2.1 Extrapolation-Iniae Inflated Overestimation
When our Q-values are estimated via function approximation' (with parameters 0), the Q-update can be erroneous and noisy [58]. Let Qu, (s,a) denote the estimates of true underlying Q" a) values at iteration of batch RI. algorithm that iterates steps (3) and (4), with To, denoting the policy that maximizes Qa For proper learning method, we might hope that the estimation error, ER := Q9, (s. a) - Q* a). has expected value = 0 and variance o > 0 for particular states/actions (the expectation here is over the sampling variability in the dataset D and stochastic updates in our batch RL algorithm). However even in this desirable scenario, Jensen's 's inequality nonetheless implies there will be overestimation error OE == E [max. Qo, (s, a)] max, Q" (a s) 2 0 for the actions currently favored by To. Here the expectation is over the randomness of the underlying dataset D and the learning algorithm. OE will be strictly positive when the estimation errors are weakly correlated and will grow with the ER-variance a [32, 58]. Under the Bellman optimality or EMaQ operator, these inflated estimates are used as target values in the next Q-update in (3), which thus produces Q (s, a) estimate that suffers from overestimation bias, meaning it is expected to exceed the true Q value even if this was not the case for initial estimate Qo, (s, a) [13, 16, 22, 23, 30, 33, 58].
In continuous batch RL, ER may have far greater variance (larger σ) for OOD states/actions poorly represented in the dataset D, our function approximator Qo, may wildly extrapolate in these data-scarce regions [4]. This in turn implies the updated policy Te. will likely differ significantly from To and favor some action à argmax, Qo, (s, a) that is OOD [17]. The estimated value of this OOD action subsequently becomes the target in the Q-update (3), and its OE will now be more severe due to the larger a [7]. Even though we only apply these Q-updates to non-OOD (s,a) a) E DD whose ER may be initially smaller, the severely overestimated target values can induce increased overestimation bias in Q+.:1 a) for (s,a) E D. In vicious cycle, the increase in Q (s,a) a) for (s, a) E D can cause extrapolated Qe... estimates to also grow for OOD actions (as there is no data to ground these OOD estimates), such that overestimation at s, a E D is further amplified through additional temporal difference updates. After many iterative updates, this extra-ooverssimaatin can eventually lead to the disturbing explosion of value estimates seen in Figure 1.
Several strategies address overestimation [13, 16, 22, 23, 30, 33]. Fujimoto et al. [17] proposed straightforward convex combination of the extremes of an estimated distribution over plausible Q values. Given a set of estimates Q0, for j = 1,..MM, they combine both the maximum and the minimum value for a given (s, a) pair:
(s,5 = min Qa a) + (1 - v) max (s,
(5)
'hiiee overestimation bias has been mainly studied in regard to function approximation error [13, 16, 23, 33, 58), Hasselt [22] shows that overestimation also arise in tabular MDPs due noise in the environment.
3

---

Here E (0. 1) determines how conservative we wish to be, and the min/max are taken across M Q-networks that only differ in their reight-initializttinn but are otherwise (independently) estimated. For larger > 0.5, Q may be viewed as a lower confidence bound for Q" where the epistemic uncertainty in Q estimates is captured via an ensemble of deep Q-networks [9].
3 Methods
Our previous discussion of extra-overestimation suggests two key sources of potential error in batch RL. Firstly, policy learned by our algorithm might be too different from the behavior policy, which can lead to risky actions whose effects are impossible to glean from the limited data. To address this, we propose to add an exploration-penalty in policy updates that reduces the divergence between our learned policy & and the policy my that generated the data. Secondly, we must restrict overestimation in Q-values, albeit only where it matters, that is, only when this leads to a policy exploiting overly optimistic estimates. As such, we only need to penalize suspiciously large Q-values for actions potentially selected by our candidate policy Th (e.g. if their estimated Q-value greatly exceeds the Q-value of actually observed actions).
3.1 Q-Value Regularization
While sequential interaction with the environment is a strong requirement that limits the practical applicability of online RL (and leads to other issues like exploration vs. exploitation), has one critical benefit: although unreliable extrapolation of Q-estimates beyond the previous observations happens during training, it is naturally corrected through further interaction with the environment. OOD state-actions with wildly overestimated values are in fact likely to be explored in subsequent updates, and their values then corrected after observing their actual effect.
In contrast, overesstttsttnnionnnnn is far more severe issue in batch RL, where we must be confident in the reliability of our learned policy before it deployed. The issue can lead to completely useless Qeestimates. The policies corresponding to these wildly extrapolated Q-function will perform poorly, pursuing risky actions whose true effects cannot be known based on the limited data in D (Figure shows an example of how extra-overetimmation can lead to the disturbing explosion of Q-value estimates).
To mitigate the key issue of extra-overestimaation in Qe(s, a), we consider three particular aspects:
An overall shift in Q-value is less important. A change from, say Qo a) to Qo a) + c(s) changes nothing about which action we might want to pick. As such, we only penalize the relative shift between Q-values.
An overestimation of Qo(s, â) which still satisfies Qa a) < Qo(s, a) for well-established a, s E ID will not change behavior and does not require penalization.
Lastly, overestimation only matters if our policy capable of discovering and exploiting
We use these three aspects to design penalty for Q-value updates to be more pessimistic [7, 25).
2
( a) = (a ( (+1)) Qo(s, à) - Q0(s,a)
max
where a E D. We can see that the first requirement is easily satisfied, since we only compare differences Qe(s, a) - Qo a) for different actions, given the same state s. The second aspect is addressed by taking the maximum between 0 and Qo(s. a) Qa(s, a). As such, we do not penalize optimism when it does not rise to the level where it would effect a change in behavior. Lastly, taking the maximum over actions drawn from the æ rather than from the maximum over all possible actions ensures that we only penalize when the overestimation would have observable consequences. As such, we limit ourselves to a rather narrow set of cases. As a result, we add this penalty to the Q-update:
0x < argmin B(S,a)~D (0otss - TQo,
A(s,a)
Anatomy of the extra-overestimation penalty A. Our proposed A penalty (6) mitigates extra- overestimation bias by hindering the learned Q-value from wildly extrapolating large values for OOD

---

state-actions, Estimated values of actions previously never seen in (known) state E D are instead encouraged to not significantly exceed the values of the actions whose effects we have seen at Note that the temporal difference update and the extra-overestimation penalty A in (7) are both framed on a common scale as a squared difference between two Q-functions.
How A affects / becomes evident through its derivative:
V VoA(s,a)= - {
( VoQo(s,,) - Volo(s,a)) i
(8)
otherwise
Here = mag Qo(s, âx) again taken N actions sampled from our current policy π, and ε = - a). A only affects certain differel-diferrences where Q-values of (possibly OOD) state-actions have higher values than the (s,a) E D. In this case, A not only reduces (s ä) by an amount proportional to e, but this penalty also increases the value of the previously-observed action (s a) to the same degree. A thus results in a value network that favors previously observed actions. We will generally want to choose a large conservative value of r in applications where we know either: that the behavior policy was of high-quality (since its chosen actions should then be highly valued), or that only a tiny fraction of the possible state-action space is represented in D, perhaps due small sample-size restricted behavior policy (since there may be severe extrapolation error).
3.2 Policy Regularization
In batch RL, the available offline data can have varying quality depending on the behavior policy Te used to collect the data. Since trying out actions is not possible in batch settings, our policy network instead updated to favor not only actions with the highest estimated Q-value but also the actions observed in D (whose effects can be more certain of). Thus we introduce an exploration penalty
to regularize the policy update step: & <- argmax & EewD (-11)))))) Qo a) X D(Tb: To)-
In principle, various f-divergences [10] or Integral Probability Metries [42] could employed in D(.,-).
In practice, we limit our choice to quantities that do not require estimating the behavior policy T&. This leaves us with the reverse Kl-ddivegeence and IPMs in Hilbert Space [2]. we further restrict ourselves to distances that do not require sampling from क.. then only the reverse KL--divergence remains. We thus estimate
(TT, no) EL Et To (a|s)] E (- F (a[s)]
--Rawm (a
log (a,ls)
(10)
whenever a, TT(sss). This is exactly what happens in batch RL where we have plenty of data drawn from the behavior policy, albeit no access to its explicit functional form. Note the first entropy
term in (9) can be ignored when we aim to minimize the estimated KI in terms of To (as will be done in our exploration penalty). Using (10), we can efficiently minimize an estimated reverse KI. divergence without having know/estimate ma or sample from Tg.
Lemma argmaax Esgan 1a,awa/[Qoss,, a)] - A A. ((T)(()))) (o(()))) is given by
(a(s))
Qo(s,a) a)
(T(sa))=
Z exp
D is the forward KL divergence (iteerenee_KKrrt(lll)s....)
(a(a)s)
ns(s)a)=
if the reverse KL divergence = KT(mo(-)s) Qo(s, a)A
where Z E R is normalizing constant in each case. Lemma shows that using either forward or reverse KL-divergence as an objective, we recover Fe = n5 in the limit of X xo. This is to be expected. After all, in this case use the distance in distributions (thus policies) as only criterion, and we prefer reverse KL to avoid having to estimate Ty. CDC thus employs the following policy-update (where the reverse KL. is expressed as a log- likeeloood as in (10)
f- argmax E, B, P amm (11s) (0(s.a)) a)] + A (s,a)) (ags)(a(s)
(11)
5

---

The exploration penalty helps ensure our learned ve is not significantly worse than Th, which is far from guaranteed in batch settings without ever testing an action. If the data were collected by fairly random (subpar) behavior policy, then this penalty (in expectation) acts similarly to maximum-entropy term. The addition of such terms to similar policy-objectives has been shown to boost performance in RL methods like soft actor-critic [21].
Note that our penalization of exploration stands in direct contrast to online RL methods that specifi- cally incentivize exploration [4, 44]. In the batch RL, exploration is extremely dangerous as it will only take place during deployment when policy is no longer being updated in response to the effect of its actions. Constraining policy-updates around an existing data-generating policy has also been demonstrated as reliable way to least obtain an improved policy in both batch [17. 63] and online [51] settings. Even moderate may be too much ask for with data of limited size or coverage of the possible mate-actions). Reliable improvement is crucial in batch settings as we cannot first test out our new policy.
can often be extremely valuable (the optimal policy
Remark 1 (Behavioral cloning occurs as X oc) Regularized policy updates with strong regular- ization (large X) is in the limit imitation learning. In fact, this is the well well-known likelihood based behavioral cloning algorithm used by [47].
If the original behavior policy ni was optimal (e.g. demonstration by a human-expert), then behavioral cloning should be utilized for learning from D [43]. However in practice, data are often collected from a subpar policy that we wish to improve upon via batch RL rather than simple imitation learning.
3.3 CDC Algorithm
Algorithm I Continuous Doubly Constrained Batch RL
Furnished with the tools for Q-value and policy regularization proposed in previous sections, introduce CDC in Algorithm 1. CDC utilizes
I: Initialize policy 5u and Qs: [Qo,]==
2: Initialize Target Qs: (Qo; 03 + 0,1341
3: for in [I..... T) do
an actor-critic framework [27] for continuous actions with stochastic policy To and Q-value Qo. parameterized by & and / respectively. Our major additions to that A penalty that mitigates overestimation bias by reducing wild extrapo- lation in value estimates and the exploration penalty (log Tÿ)) that discourages the estimated policy from straying to OOD state-actions very different from those whose effects we have ob- served in D.
4: Sample = {(s.a,
~
5: For each s, E B: sample N actions (ax Ran
<a(+))), 1o())=11 mo(-1s")
6: Qo value update:
u(s'r) ()),r):=ro := + of
To (s. a4) given by Eq
(,(r,) = ([ max Qu, (x. - Qu, (v.00) )
0g +- argmin E [ (2.)) - 2(()r:))
+ n A5(, for 1,..,
Although the particular form of CDC presented in Algorithm I optimizes a stochastic policy with the off-policy updates of [53] and temporal difference value-updates using (3), we empha- size that the general idea behind CDC can be utilized with other forms of actor-critic updates such as those considered by [12. 16, 21]. In practice, CDC estimates expectations of quanti- ties introduced throughout via mini- estimates derived from samples taken from D, and each optimization is performed via a few stochastic gradient method iterates.
7: Fy policy update:
> (o)ls + 22
argmax
8:
Update Target Networks:
8
+ +
(1
()] Vj E M
end for
To account for epistemic uncertainty due the limited data, the value update in Step 6 of Algorithm I uses 2, from (5) in place of Qg. In CDC,
simply utilize the same moderately conservative value of 0.75 used by [17], since are not purely relying on the lower confidence bound Qa to correct all overestimation. For this reason, CDC is able to achieve strong performance with a small ensemble of M = 4 Q-networks (used throughout this work), whereas [18] require larger ensembles of 16 nentwwrrks and an extremely conservative V/ = in order to achieve good performance.
To correct oxeretoiverttiimatioo within each of the M individual Q-networks, Algorithm actually applies separate extra-overestimation penalty A specific to each Q-network. The steps of our proposed CDC method are detailed in Algorithm 1. In blue, we highlight the only modifications CDC makes to standard off- off-policy actor-critic framework that has been suitably adapted for continuous batch RL via the aforementioned techniques like EMaQ [18] and lower-confidence bounds for Q-

---

values [17]. Throughout, we use 7 0 & À = 0 to refer to this baseline framework (without our proposed penalties), and note that majority of modern batch RL methods like CQL [29], BCQ [17], BEAR [28], BRAC [63] are built upon similar frameworks.
Although each of our proposed regularizers is modular, we emphasize that they complement each other: the Q-Value regularization mitigates extra-overestimation error while the policy regularizer ensures candidate policies do not stray too far from the offline data. Ablation studies show that the best performance is only achieved through simultaneous use of both regularizers (Figure 2a, Table S1). Note that CDC is quite simple to implement: each penalty can be added to existing actor-critic RL frameworks with minimal extra code and the addition of both penalties involves no further complexity beyond the sum of the parts.
be used independently and their implementation
Theorem For Qo in (5), let TCnC : Qo. - To.,, denote the operator corresponding to the Qo- updates resulting from the tth iteration of Steps 6-7 of aAlgorithm I. TTDc is Los contraction under standard conditions that suffice for the ordinary Bellman operator to be contractive [3, 6, 8, 56).
The proof and formal list of assumptions are in Appendix D.1. Together with Banach's 's theorem, the contraction property established in Theorem above guarantees that our CDC updates converge to a fixed point under commonly-assumen conditions that suffice for standard RL algorithms to converge [31]. Due to issues of (nonconvex) function approximation, it is difficult to guarantee this in practice or empirical optimality of the resulting estimates [38, 40]. We do note that the addition of our two novel regularizers further enhances the contractive nature and stability of the CDC updates when n7 A > by shrinking Q-values and policy cetton-probabilitee toward the corresponding values estimated for the behavior policy (i.e. values computed for observations in D). Our CDC penalties can thus not only lead to less citty-eetrapplated batch estimates, but also faster (and more stable) convergence of the learning process (as shown in Figure 1, where Standard actor-critic refers to Algorithm where 1)
0).
Let T& E 11 be the policy learned by CDC, 7 denote discount factor, and denote the sample size of dataset D generated from Th- Also let (xx) represent the true expected return produced by deploying policy πt in the environment. Under mild assumptions listed in Appendix D, there exist constants r", ,Cx, V such that with high probability 2 1-
Theorem 2
J(TT) 2 J(Tx) -( - + -V--log 8)/n
r"
Appendix D.2 contains proof and descriptions of the assumptions in this result. Theorem 2 assures us of the reliability of the policy Fs produced by CDC, guarantceing that with high probability ne will not have much worse outcomes than the behavior policy Te. where the probability here depends on the size of the dataset D and our choice of policy regularization penalty À (the constant Cx is decreasing function of A). In batch settings, expecting to learn the optimal policy is futile from limited data. Even ensuring any improvement at all over an arbitrary T. is ambitious when we cannot ever test any policies in the environment, and reliability of the learned Tı is thus major concern.
Let Ogag E [max, Qo(s, a)] - max, Q' a) be the overestimation error in actions favored by an agent ag. Here Qa denotes the estimate of true Q-value learned by ag, which may either use CDC (with n > 0) or a baseline version of Algorithm I with 0 (with the same value of A). Under the assumptions listed in Appendix D).3, there co-exist constants Lı and To such that
Theorem 3
OEeDe S L1 11 S OEnaseline
This theorem (proved in Appendix D.3) underscores the influence of the 1) parameter in terms of containing the overestimation problem in offline Q-learning. Mitigating this overestimation, which can be done using non-zero 77, can ultimately lead into better returns as we show in the experimental section. In particular, CDC achieves lower overestimation by deliberately underestinating Q-values for non-observed state-actions (but it limits the degree of downward bias as described in Remark 2). Buckman al. [7], Jin al. [25] prove that some degree of pessimism is unavoidable to ensure non-catastrophic deployment of batch RL in practice, where it is unlikely there will ever be sufficient data for the agent to accurately estimate the consequences of all possible actions in all states.
Remark 2 (Pessimism is limited in CDC) Extreme pessimism leads to overly conservative policies with limited returns. The degree of pessimism in CDC remains limited (capped once A = 0), unlike lower-confidence bounds which can become arbitrarily pessimistic and hence limited in their return.
7

---

4 Related Work
Aiming for practical framework to improve arbitrary existing policies, much research has studied batch RL [34, 36] and the issue of overestimation [22, 23, 58). [26, 64] consider model-based approaches for batch RL, and [1] find ensembles partly address some of the issues that arise in batch settings, To remain suitably conservative, a popular class of approaches constrain the policy updates to remain in the vicinity of TT, via, e.g.. distributional matching [17], support matching [28, 63], imposition of a behavior-based prior [52], or implicit constraints via selective policy-updates [46, 60]. Similar to imitation learning in online setting [24, 43, 47, 49], many of such methods need to explicitly estimate the behavior policy [17. 18, 28]. Although methods like [46, 60] do not have an explicit constraint the policy update, they still can be categorized
policy constrained- based approach as the policy update rule has been changed in a such a way that it selectively updates the policy utilizing information contained in the Q-values. Although these approaches show promising results, policy-constraint methods often work best for data collected from a high-quality (expert) behavior policy, and may struggle to significantly improve upon highly suboptimal Th- Compared to the previous works, our CDC does not need to severely constrain candidate policies around Ty. which reduces achievable returns. Even with a strong policy constraint, the resulting policy is still affected by the learned Q-value, thus we still must correct Q-value issues. Instead of constraining policy updates, [29] advocate conservatively lower-bounding estimates of the value function. This allows for more flexibility to improve upon low-quality т-- [39] considers pessimistic and conservative approach to update Q-value by utilizing the marginalized state-action distribution of available data. and literature, demonstrating these address complementary issues of the batch RL problem and are both required in a performant solution.
Our proposed CDC algorithm is inspired by ideas from both the
Experiments
In this section, we evaluate our CDC algorithm against existing methods 32 tasks from the D4RL benchmark [14]. We also investigate the utility of individual CDC regularizers through ablation analyses, and demonstrate the broader applicability of our extra-overestimation penalty to off-policy evaluation in addition to batch RL. Our training/evaluation setup exactly follows existing work [14, 17, 28, 29J. See Appendices A, B, and C for a complete description of our experimental pipeline.
Setup. We compare CDC against existing batch RL methods: BEAR [28), BRAC-V/P [63], BC [63], CQL [29]. BCQ [17], RBVE2 [20] and SAC [21]. This covers a rich set of strong batch RL methods ranging from behavioral cloning to taltue-oonntrinnd--aaaee
pessimistic methods, with the exception of SAC. SAC is a popular off-policy method that empirically performs quite well in online RL, and is included to investigate how online RL methods fare when applied to the batch setting. Note that CDC was simply run on every task using the same network and the original rewards/actions provided in the task, without any manual task-specific smoothing. Moreover, all these baseline methods also utilize an ensemble of Q networks as in (5).
Results. Figure 2b and Table illustrate that CDC performs better than the majority of the other batch RL methods the D4RL tasks. Across all 32 tasks, CDC obtains normalized return of 1397, whereas the next-best method (CQL) achieves 1 1245. In head-to-head comparisons, CDC generates statistically significantly greater overall returns (Table 1). Unsurprisingly, behavioral-cloning (BC) works well on tasks with data generated by an expert Tr. while the online RI. method, SAC, fares poorly in many tasks. CDC remains reasonably competitive across all tasks, regardless of the environment or the quality of Th (i.e. random vs. expert).
Next we perform a comprehensive set of ablation studies to gauge the contribution of our proposed penalties in CDC. Here we run additional variants of Algorithm without our penalties (i.e. 17 = A 0), with only our estra-overestimation penalty (A = 0), and with only our exploration penalty (1) = 0). Figure 2a and Tables SI show that both penalties are critical for the strong performance of CDC, with the oxtra-overestimatiio penalty A being of greater importance than exploration (see also Figure 2a). Note that all our ablation variants still employ the lower confidence bound from (5), which alone clearly does not suffice correct oetresoverstiintioon.
Comparison between CDC and the concurrently-proppeee RBVE method [20] is relegated to Section A.I.
8

---

D0C 7-08
OOL
BTACA
DPAGP
-20
-20
-00
-100
12
05 Taska
10
14-15
16 Taaka
(b)
Figure 2:
Difference in (normalized) return achieved by various algorithms vs CDC in 32 D4RL tasks. X-axis colors indicate environments (sce Table 1), and points below the line ( (-) indicate worse performance than CDC. Figure 2a shows that fixing 1) or À to zero (i.c. omitting our penalties) produces far worsc returns than CDC (sec also Table S1). This ablation study proves that major performance gains for CDC stem from our novel pair of regularizers, as the only difference between CDC and these ablated variants either 1) or or both are set to zero in Algorithm (all other details are exactly the same). Figure 2b compares CDC against existing batch RL algorithms, where CDC overall compares favorably to each other method in head-to-head comparisons (see also Table 1). Note these figures can be compared to each other well.
1.00
5.1 Offline Policy Evaluation
Correlation Coefficient
0.50
The true practical applicability of batch RL re-
mains however hampered without the ability to do proper algorthnnhhypeppramettrr selec- tion. Table shows that no algorithm universally dominates all others across all environments or In practice, it is difficult to know which technique will perform best, un- less can do proper offline policy evaluation (OPE) of different candidate policies before their actual online deployment [15, 45).
0.00
-0.25
-0.90
FGE
-0:75
behavior-policee
Figure 3: How well OPE estimates correlate with ac- tual return achieved by 20 different policies for each D4RL task. Due unmitigated overestimation, FQE estimates correlate negatively with true returns 15 of 32 tasks (using A in FQE reduces this to 4).
Tasks
OPE aims to estimate the performance of a given policy under the same setting considered here, with offline data collected by unknown behavior policy [35, 45]. Beyond segorithn//yperpaaameee comparison, OPE is often employed for critical policy-making decisions where environment interac- tion is no longer an option, e.g., sensitive healtheare applications [19]. One practical OPE method for data of the form in D is Fitted Q Evaluation (FQE) [35]. To score given policy æ, FQE iterates temporal difference updates of the form (3) using the standard Bellman operator from (1) in place of EMaQ. After learning an estimate QT, FQE simply estimates the return of via the expectation of Q (sss a) over the initial state distribution and actions sampled from æ.
However, like batch RL, OPE also relies on limited data and thus can still suffer from severe Q-value estimation errors. To contain the overestimation bias, we regularize the FQE temporal difference updates with our A penalty, in a similar manner to (7). Figure 3 compares the performance of throughout) against the standard unregularized FQE. Here
A A-penalization of FQE (with r we use both OPE methods to score 20 different policies (learned via different settings) and gauge OPE-quality via the Pearson correlation between OPE estimated returns and the actual return (over our 20 policies). We observe higher correlation for FQE + A (0.37 on average) over FQE (0.01 on average) in the majority of tasks, demonstrating the usefulness of our regularizers. The usefulness of our regularizers thus extend beyond batch RL and carry to the off-policg evaluation setting.
'The results for CQL are taken from the official author provided codes [https://githut com/ sviralkumar2907/ /CQL of [29]. The published CQL codes are used to produce results for all but Adroit and FrankaKitchen where the codes are not available. For these latter domains, we simply use the CQL results reported in the paper of [29].

---

Index
lask Name
SAC
BC
BRAC-F
BRAC-V
BEAK
BCQ
CQL
A 0 1)
CDC
hnffeheeted-mmmionn
29.6
2.1
23.5
28.1
25.5
2.25
16.71
32.8
27.36
hnllshoetab-mediim
40.97
36.1
44.0
45.6
38
51
46.05
2
halfcheetah medium replay
36.47
38.1
15.6
45.9
36.2
34.79
12.77
22.72
11.74
helfehectch--ee ceceetttddniii-epppeeeee
3.78
35.8
43.8
45.3
51.7
69.64
39.18
7.12
59.64
-0.41
107.0
3.8
83.
5
hupper random
0.8
9.8
11.1
12.0
9.5
10.6
10.37
1.58
14.76
pupper-medium
0.81
23.0
31.2
32.3
47.6
65.07
51.79
0.58
60.39
hopper medium replay
8
mnnge--ndium-eepet
8.96
119.9
1.1
0.8
4.0
108.
112.46
18.07
86.9
9
hopper-xpert
0.8
100.0
6.6
3-7
110.3
75.52
109,997
1.27
102.75
11
walker 25 medium
0.51
6.6
72.7
81.3
$1.2
52.03
71.03
0.31
82.15 13
12
walker 28--maiim---ee
0.04
11.3
0.3
0.9
10.8
13.67
19.56
3.81
22.96
medium expent
14
walker walker22-experr expert
0.05
125.7
0.2
0.00
J06.1
87.59
103.6
87.54
15
ont cntmarc-umsee
0.0
65
50
70
73
88.52
90.12
22.22
91.85
16
0.0
55
70
61.11
11.11
62.59
0.0
0.0
40.74
18
anemaze medium diiverse
0.0
0
0
0.0
40.71
0.0
40.74
19
play
0.0
0
1.85
0.0
0.0
11
21
pen humam
-1.15
314
8.1
0.6
1.
66.88
37.5
-3-43
73.19
22
pen-cloned
-0.64
56.9
-25
26.5
50.56
39.2
-3.4
49.18
hamnerhumma
1.5
24
hammercluned
0.27
0.8
0.3
0.3
0.3
0.38
2.1
0:26
2 27
25
doar-human
-0.34
0.5
0.33
-.3
0.3
0,05
9.9
0.16
4.02
27
relocate- haman
-0.31
0
0.3
00.3
0.3
0.04
0.2
0.73
28
eclecatsctoned
00.11
0.1
0.3
0.3
0.2
0,228
01.1
0.15
01.24
complete
30
kitchen partial
0.0
33.8
13.1
9.26
49.8
0.0
42.5
31
kithhe-miiaa
0.0
47.5
47.2
18.43
51
0.0
42.87
Total Score
116.28
984.4
383.4
434.5
844.0
1060044
1245./2
173.67
18996.99
p-value
7.06007
1.6e-03
5.3c-07
3.96-06
T.1e-04
6.1c-04
3.6c-02
1.8c066
Table 1: Return achieved in deployment of policies learned via different batch RL methods. The return in cach environment here is normalized using (12) as originally advocated by [14]. For cach method: we perform head-to-head comparison against CDC across the D4RL tasks, reporting the p-value of a (one-sided) Wilcoxon signed rank test [61] that compares this method's return against that of CDC (over the tasks). Here = 0 y = is variant Algorithm without our penalties where proves that major performance gains for CDC stem from our novel pair of regularizers.
6
Discussion
Here we propose a simple and effective algorithm for batch RL by introducing a simple pair of regularizers that abate the challenge of learning how to act from limited data. The first constrains the value update to mitigate extra-overestimation error, while the latter constrains the policy update to ensure candidate policies do stray too far from the offline data. Unlike previous work, this paper highlights the utility of simultaneous policy and value regularization in batch RL. One can envision other combinations of alternative policy and value regularizers that may perform even better than the particular policy/value penalties used in CDC. That said, our CDC penalties are particularly simple to incorporate into arbitrary actor-critic RI. frameworks, and operate synergistically as illustrated the ablation studies. Comprehensive experiments on standard offline continuous-control benchmarks suggest that CDC compares favorably with state-of-the- methods for batch RL, and our proposed penalties are also useful to improve offline policy evaluation. The broader impact of this work will hopefully be to improve batch RL performance offline applications, but we caution that unobserved confounding remains another key challenge in real-world data that was not addressed in this work.
10

---

References
[1] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning, In International Conference Machine Learning. 2020.
[2] Y. Altun and A. Smola. Unifying divergence minimization and statistical inference via convex duality. In International Conference on Computational Learning Theory, pages 139-153. Springer, 2006.
[3] A. Antos, R. Munos, and C. Szepesvari. Fitted Q-iteration in continuous action-space MDPs. In Advances in Neural Information Processing Systems, 2007.
[4] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29, pages 1471 1471-1479, 2016.
[5] R. E. Bellman. Dynamic Programming. Princeton University Press, 1957.
[6] D. P. Bertsekas and S. Shreve. Stochastic optimal control: the discrete-time case. Athena Scientific, 2004.
[7] Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism fixed-dataset policy optimization. In International Conference on Learning Representations, 2021.
[8] L. Busoniu, R. Babuska, B. De Schutter, and D. Ernst. Reinforcement learning and dynamic programming using function approximators, volume 39. CRC press, 2010.
[9] R. Y. Chen, S. Sidor, P. Abbeel, and J. Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv: 111.110060000000 2017.
[10] I. Csiszár and P. Shields. Information theory and statisties: tutorial. Foundations and Trends in Communications and Information Theory, 1(4):417-528, 2004. ISSN 1567-2190. doi: 10. 00.151/01000000044
[11] G. Dulac-Amold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning. In ICML Reinforcement Learning for Real Life (RLARealLife) Workshop. 2019.
[12] R. Fakoor, P. Chaudhari, and A. J. Smola. P3O: policy-on policy-off policy optimization. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, page 371, 2019.
[13] R. Fakoor, P. Chaudhari, and A. J. Smola. Ddpg++: Striving for simplicity in continuous-control off-policy reinforcement learning. 1511:9, 2020.
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. wxXh:2204.00729, 2020.
[15] J. Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang. M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. L. Paine. Benchmarks for deep off- off-policy evaluation, 2021.
[16] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, 2018.
[17] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062, 2019.
[18] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-matt q-learning operator for simple yet effective offline and online rl. arXiv:2007.1 1091, 2021.
[19] O. Gottesman, J. Futoma, Y. Liu, Parbhoo, Celi, E. Brunskill, and F. Doshi-Velez. Inter- pretable off policy evaluation in reinforcement learning by highlighting influential transitions. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3658-3661 PMLR, 2020.

---

[20] C. Gulcehre, S. G. Colmenarejo, Z. Wang, J. Sygnowski, T. Paine, K. Zolna, Y. Chen, M. Hoff- man, R. Pascanu, and N. de Freitas. Regularized behavior value estimation, 2021.
[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv:1801.01200 2018.
[22] H. V. Hasselt. Double Q-learning, In Advances in Neural Information Processing Systems 23, pages 2613-2621, 2010.
[23] H. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'1,, page 2094-2100. AAAI Press, 2016.
[24] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac- Arnold, I. Osband, Agapiou, J. Z. Leibo, and A. Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR, abs/1704.03732, 2017.
[25] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline RL? arXiv preprint arXiv:2012. 15085, 2020.
[26] R. Kidambi, Rajeswaran, P. Netrapalli, and T. Joachims. Morel Model-based offline reinforcement learning. wrXiv:2005.0595// 2020.
[27] V. Konda and Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 1008-1014. MIT Press, 2000.
[28] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. 1906009906000499 Nov. 2019.
[29] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-Learning for Offline Reinforce- ment Learning. arXiv:2006.04779 June 2020.
[30] A. Kuznetsov, Shvechikov, A. Grishin, and D. Vetrov, Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5556-5566. PMLR, 13-18 Jul 2020.
[31] M. G. Lagoudakis and R. Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1 1111-11-4,, 2003.
[32] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-leamning In International Conference on Learning Representations, 2019.
[33] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-learning In International Conference on Learning Representations, 2020.
[34] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In M. Wiering and M. van Otterlo, editors, Reinforcement Learning: State-of-the-Arz pages 45-73. Springer, 2012.
[35] H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3703-3712. PMLR, 2019.
[36] Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. 0rXiv:2005.01643 May 2020.
[37] Y. Li. Deep reinforcement learning: An overview. arXiv preprint 1 2017.
[38] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
12

---

[39] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1264-1274. Curran Associates, Inc., 2020.
[40] G. Matheron, N. Perrin, and O. Sigaud. The problem with ddpg: understanding failures in deterministic environments with sparse rewards. arXiv preprini 1rXiv:1911.1 2019.
[41] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, 2015.
[42] A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997. ISSN 00018678.
[43] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179, 2018.
[44] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 2721-2730. PMLR, 06-11 Aug 2017.
[45] T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055, 2020.
[46] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage weieatteeewwegghedd regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00077, 2019.
[47] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-99) 1991.
[48] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. USA, 1st edition, 1994. ISBN 0471619779.
[49] S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In G. J. Gordon, D. B. Dunson, and M. Dudík, editors, AISTATS, volume 15 of JMLR Proceedings, pages 627-635. JMLR.org, 2011.
[50] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
[51] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv::1077: 170..06347 2017.
[52] N. Siegel. J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[53] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387-395. PMLR, 2014.
[54] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. den Driessche, J. Schrittwieser, I. Antonogiou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, Lillicrap, M. Leach, K. Kavukcuoglu. T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016.
[55] D. Silver, T. Hubert, J. Schritwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanetot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018. ISSN 0036-8075. doi: 10.
13

---

[56] C. Szepesvári. Efficient approximate planning in continuous space markovian decision problems.
Al Communications, 14(3):63-176,, 2001.
[57] Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence policy improvement. In
International Conference Machine Learning, pages 2380-2388. PMLR, 2015.
[58] S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School, pages 255-263. Lawrence Erlbaum, 1993.
[59] Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[60] Wang. Novikov, K. Zolna, T. Springenberg, S. Reed, B. Shahriari, N. Siegel, Merel, C. Gulcehre, N. Heess, and N. de Freitas. Critic regularized regression. 2ti::2000.11:3:3 2020.
[61] Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-33, 1945.
[62] R. J. Williams and J. Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-26,, 1991.
[63] Y. Wu, G. Tucker, and O. Nachum. Behavior Regularized Offline Reinforcement Learning. 1911.
2019.
[64] Yu, G. Thomas, L.. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv:2005. /3239, 2020.
14


Reviews:
[{"review_id": "qVKpkCYhmZh", "reviewer": "Reviewer_dkx5", "summary": "This works proposes two regularization techniques that address severe extrapolation errors in offline RL. In various D4RL datasets, the method shows competitive performance compared to the previous state-of-the-art, outperforming it on average performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The motivations behind the two regularization terms are solid.\n- The proposed method performs well empirically.\n\nCons\n- It is a bit unclear why the policy regularization and the Q-value regularization are both required. Conceptually, the policy regularization term will mitigate the Q-value overestimation, as OOD actions will be less sampled from equation (2). Likewise, the Q-value regularization term will lead the policy to be close to the behavior policy as the term encourages the dataset actions to have the maximum Q-value given a state.\n- The hyperparameter setting is not fully specified. I strongly recommend the authors to specify the value of $\\lambda$ and $\\eta$, the two core parameters, used for **each** dataset. This is especially important for offline RL methods since evaluating the learned policy is difficult without online deployment.\n- Also, the hyperparameter search range seems to be a bit arbitrary (e.g., {.1, .2, .5, .6, .8, 1, 5, ...} for $\\eta$). Can the authors provide some results with regard to the hyperparameter sensitivity?\n\nOther comments\n- A concurrent work [1] also proposes to use a behavior cloning term for offline RL. While this does not hurt the originality of this paper as both works are concurrent, it is interesting that this paper and [1] have very different conclusions about the behavior cloning term. While Table S3 from this paper suggests the behavior cloning alone is not sufficient to train offline RL policies, [1] claims the behavior cloning term works well for all datasets (at least for D4RL Gym). Can the authors provide some intuitions on why the results are so different?\n\n[1] A Minimalist Approach to Offline Reinforcement Learning, Fujimoto et al., arXiv 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n- Listed above\n\nSocietal Impact\n- N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "n-wmZ3Q2yJw", "reviewer": "Reviewer_4YAR", "summary": "In this paper the authors identify two areas, namely policy over-“exploration” and value overestimation, where instabilities can arise in offline RL.\n\nUsing this knowledge, the authors propose two corresponding regularizations to ameliorate this issue and show competitive performance compared with state-of-the-art benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* I found the paper clearly written, and it motivates and identifies the issues that arise in offline RL well.\n* The described approaches to fixing the identified extrapolation problems in offline RL are simple to follow and should be straightforward to implement onto existing algorithms.\n\nCons:\n* Fundamentally there is not enough novelty introduced in this paper, and nor is it framed in a way that facilitates its distinguishing elements. For instance, the general discourse around extrapolation in offline RL is well-discussed in many prior works, so the paper is not the first to identify these issues (see [1,2], for instance). Regarding the actual ameliorations themselves, again variations of these have appeared in prior work; for instance the critic regularization is very similar to CQL, with two small modifications; a max is taken over the sampled actions (see Equation 2 of [3]) and a rectifier function that ensures positivity. Similarly, the actor regularization is already well discussed in [2,4], except a reverse-KL is chosen here to avoid learning a BC prior policy (or discriminator in a Fenchel dual form). My issue is that there is very little acknowledgement that the methods *heavily* lean on this prior work, and no ablations are provided to help understand whether the minor aforementioned modifications help or hinder compared to the original work (runtime, performance, etc.). Whilst I appreciate these two approaches (i.e., CQL + reverse KL policy constraint) haven’t been combined in this way before, that is not enough to merit acceptance. I think if a significant rewrite was done, framing it with a more explicit comparison with prior work (rather than just citing it in related work), that would improve my likelihood of accepting this paper.\n* There are not enough seeds; 3 seeds is significantly below what is expected for the proprioceptive versions of MuJoCo benchmarks, and most current literature will use 5 and upwards, including more expensive model-based approaches.\n* Important details, such as offline training epochs/iterations are omitted. The authors identified that OPE is important, and that in offline RL, performance can degrade with increasing training epochs due to over-extrapolation. However when reporting performance, did the authors simply stop training when they found the best online performance on the oracle environment, or did they select the best policy under their FQE critic as alluded to in Section 5.1?\n* The results are not that compelling; it is true that on the benchmarks tested, CDC does appear the most performant, but in line with the seed issue above, it is unclear if this performance is truly universal. Furthermore, recent algorithms such as [5] aren’t included, nor are any model-based benchmarks, which are known to perform better than model-free equivalents (e.g., [6,7]).\n\nComments/Nits:\n* The use of the word ‘exploration’ in the actor constraint is not ideal (i.e.,’exploration penalty’). The whole point of offline RL is that no exploration is possible, so claiming that a constraint reduces exploration when we literally have no interaction is confusing. I agree that if this algorithm was used in standard online RL, the constraint would likely reduce exploration, but this is precisely the point; exploration is the act of sampling those new states/trajectories through interaction. Using the term exploration in this pure offline problem setting gives the false impression we are able to do this sampling.\n\nConclusion:\nOverall the paper is well written, and has nice elements to it, such as relative algorithmic simplicity, and clear problem framing of issues in offline RL. However, the actual details are relatively incremental compared to prior work (which isn’t made altogether clear in the manuscript and narrative), and indeed much of the discourse in this paper has been highlighted by previous work as well. I therefore believe this paper is below the level required for acceptance, and have given clear points in the cons section on how the authors’ should update their paper in order to move my position more towards acceptance.\n\n[1] Off-Policy Deep Reinforcement Learning without Exploration, Fujimoto et al.\n\n[2] Behavior Regularized Offline Reinforcement Learning, Wu et al.\n\n[3] Conservative Q-Learning for Offline Reinforcement Learning, Kumar et al.\n\n[4] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, Kumar et al.\n\n[5] Offline Reinforcement Learning with Fisher Divergence Critic Regularization, Kostrikov et al.\n\n[6] COMBO: Conservative Offline Model-Based Policy Optimization, Yu et al.\n\n[7] MOReL : Model-Based Offline Reinforcement Learning, Kidambi et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "4wXh00wAGoB", "reviewer": "Reviewer_meGC", "summary": "This paper presents two constraints for batch reinforcement learning. One is a Q regularizer; the other is a policy regularizer. The Q-value regularizer penalizes overestimation by ensuring out-of-distribution, policy-impacting Q values aren't too much larger than other Q values for a given state. The policy regularizer penalizes exploration by penalizing the divergence between the behavior and target policies.\n\nThese changes were tested on a suite of RL benchmarks using both ablation and comparison to other batch RL approaches. The performance of CDC was often superior.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presented some very interesting results, but the limited number of runs (3) in the empirical section is a concern. The results seem novel and interesting, but can I rely on them with so few runs? I was impressed by how well written and clear the entire paper was. I have not worked through the appendix in detail to assess the theoretical foundation, but the theory described in the paper was framed well. I would love to see this paper published so I can share it and try it, but I think it needs more runs first.\n\nRegarding **originality**, I'm not familiar enough with the field of batch RL to know how original it is. But the authors do an excellent job of describing how the two regularizers are novel, yet grounded in existing work. \n\n**Quality** is my biggest concern. I was originally concerned about the number of runs, but the authors pointed out batch RL requires fewer runs than online RL. The lack of error bars in Figure 2 leaves me concerned about whether any differences are significant.\nI believe the claims are supported by theory. However, the majority of the theory was relegated to Appendix D, which I did not review.\n\nRegarding **clarity**, the paper was very well written. It had a great flow between sections, it justified its problem, and it gave me the background I needed as I needed it. It was enjoyable to read, and I left with a deeper understanding of both the problem and where CDC fit as a solution. I believe enough information was provided to reproduce the results, though I hope code will be provided in the final submission.\n\nRegarding **significance**, I believe these results are valuable and likely to be of benefit to the field. The results of these regularizers in both batch RL and offline evaluation are very interesting.\n\n### Specific Notes\n\n*Update: Many of these were resolved*\n\n* In Figure 2, I'd like to see **error bars or confidence intervals**. Checklist 3c claims they were included, but they are not. (p-values are not the same as error bars and are not a good substitute.) I realize this might make the plots too busy (and your plots look great!) but it gives me more confidence in the results than a p-value. Update: the authors provided learning curves for CDC vs BCQ. That gave me more confidence in this point.\n* I wasn't sure how to interpret **Figure 1**, including its units (e.g. is it normalized? an aggregate return or specific environment?) and how it was created. Plots on the first page are unusual, but this one clarified a point in the introduction, so it *might* be appropriate here. However, I don't recall seeing more details about it, even in the appendix, which is a problem.\n* In **Figure 3**, I'm curious why some tasks are so much better or worse. When I look at Figure 2b and the task names in Table S1, it seems like CDC does best when the source data is poor. In cases when expert data is available, it's often not the best. I'd love to see more focus on this, since it's important in understanding when to pick it instead of another method. However, this could also be completely random; once again, adding error bars and more runs would make a big difference.\n* There's a **typo** on line 351. I believe it should say, \"For CDC, total normalized return...\"\n* The **ordering of Figures 2a and 2b** is the opposite of the text. Lines 350-356 discuss 2b, and then lines 357–363 discuss 2a. This was slightly jarring.\n* Checklist 3d said the **total amount and type of compute** were included, but I couldn't find the *amount* of compute. How many hours did experiments take to complete?\n\n### Update\n\nThe authors ran with extra seeds and pointed out that number of seeds for batch RL doesn't need to be as high as in online RL. And they  provided learning curves for CDC vs BCQ. That gave me more confidence in the (lack of) Figure 2 error bars.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "2dpmLV-EhRB", "reviewer": "Reviewer_ZoHU", "summary": "This paper proposes two regularizers for actor-critic RL methods to make them applicable for offline RL setting when only offline data is available. One regularizer is added to Q learning step in order to alleviate overestimation problem and one regularizer is added to policy optimisation step to avoid sampling out of distribution (OOD) actions. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nTo the best of my knowledge in prior works either a regularizer is added to policy optimization step or in critic learning step and not in both steps. So to me the main contribution of the paper is considering regularizing both policy evaluation and policy  optimisation steps at the same time. The other claimed contributions such as choice of the regularizer in each step is minimal given that adding BC loss to policy optimisation loss is not new and also the Q-penalty added to policy evaluation being explored in various prior works. Overall i think studying the dynamics of combining these high-level ideas from prior works is valuable.\n\nQuality:\n\nI think this paper does not support the idea of adding regularizers to both policy evaluation and policy optimisation steps well enough. There are some intuitive reasoning why the role of these two regulaizers are orthogonal but i'd like to see more principled reasoning that can explain what is the relation between these two regulaizers and in which condition both can have a same effect. This is crucial as we dont want to add a redundant hyper parameter to our RL algorithm.  I'm asking this because intuitively i think a regularizer on policy to stay close enough to the data should be able to avoid exploiting over-estimated Q values without a need for an extra regularizer in policy evaluation step. \n\nI'm a bit confused why two methods are employed to avoid overestimation here: \n    1- one by using ensemble of q networks (equation 5) \n    2- the other one by adding the proposed regularizer. \nHow these two parallel methods influence eachother?\n\nAdding a BC loss to RL loss makes a lot of sense but this has been proposed before in other prior works; for example see  SAC+BC in [1]. The question remains here is that how to choose the trade-off. It seems to me the RL loss and BC loss can be in a very different scale. How to deal with this issue?\n\nAdding Q penalty to policy evaluation step is similar to what CQL does with the difference that here a different regularizer is proposed. Looking at the results in Table S2 it seems CQL outperforms the proposed method with \\lambda=0.0 . This bring me to the question that why authors did not choose to use the CQL regularizer? In general it is inconclusive that the  choice of regularizer  for policy evaluation step is a good one compare to prior works.\n\nOverall i think paper could do a better job in supporting the claims.\n\nclarity:\n\nThis paper is written clearly and quite educational.\n\n[1] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement\nlearning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "authors adequately has discussed the possible limitations and societal impact of their work in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have carried out an extensive discussion on several points here, with several common themes being raised:
1) "Pitch" of the novelty with respect to the multitude of similarly motivated approaches; 
2) Need for further explanation or analysis on the core technical advance: combining value and policy regularization; and 
3) Further seeds, ablations, and experimental details.

With their responses, I believe the authors have nicely addressed 3), and with their reply text as well as several image links provided. The submitted method is reported to often be a top-performer on important benchmarks, and that the experiments have been done rather exhuastively relative to the recent competition in this field.

The discussion also offers hope that point 1) can be addressed, but this is harder to judge without seeing a new revision of the paper. Point 2) may be the most challenging and difficult to assess (concrete analysis of the real performance of deep RL is at an early stage overall). Ablations show "the combination works best", but I did agree with the reviewers that an expanded and unified discussion about how the two constraints may interact, would be valuable. Is it possible to explain how they are the right choices not independently but in combination?

In summary, this is an intriguing and valuable paper, but the writing enhancements requested are rather large, making it just slightly below threshold in the competitive NeurIPS2021 field. 

Decision: Accept (Poster)