Continuous Doubly Constrained Batch Reinforcement Learning
Authors: Rasool Fakoor
Jonas Mueller
Kavosh Asadi
Pratik Chaudhari
Alex Smola
Venue: NeurIPS 2021 Poster (2021)
Keywords: batch reinforcement learning
overestimation bias
extrapolation
offline reinforcement learning
batch rl
offline rl

Abstract: Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of $32$ continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.

Continuous Doubly Constrained Batch Reinforcement Learning
Rasool Fakoor', .onas Mueller', Kavosh Asadi', Pratik Chaudhari 1.2, Alexander J. Smola *Amazon Web Services, 2 University of Pennsylvania fakoor@amazon.com com
Abstract
Reliant on too many experiments learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and value-constrain that discourages overly optimistic estimates. Over a comprehensive set of 32 continuous action batch RL benchmarks, our approach compares favorably to state-of-the- methods, regardless of how the offline data were collected.
Introduction
Deep RL algorithms have demonstrated impres- sive performance in simulable digital environ- ments like video games [41, 54, 55]. In these settings, the agent execute different policies and observe their performance. Barring a few ex- amples [37], advancements have not translated quite as well to real-world environments, where it is typically infeasible experience millions of environmental interactions [11]. Moreover, in presence of an acceptable heuristic, it is in- appropriate to deploy an agent that learns from scratch hoping that may eventually outperform the heuristic after sufficient experimentation.
Daanket Aebee (oun)
Average Q-values
6.25
8.50
0.35
150
Herations
0.C C.)
6.7 130 122
Figure 1: Batch RL with CDC vs. No CDC. Left: Stan- dard actor-critic overestimates Q-values whereas CDC estimates are well controlled. Right: Wild overestima- tion leads to worse- porse-performing policies whereas CDC performs well.
The setting of batch offline RL instead offers policies for real-world applications [34, 57]. Batch RL is widely applicable because this setting does not require that: a proposed policy be tested through real environment interactions, or that data be collected under a particular policy. Instead, the agent only has access to a fixed dataset D collected through actions taken according some unknown behavior policy mp- The main challenge in this setting is that data may only span small subset of the possible state-action pairs. Worst yet, the agent cannot observe the effects of novel out-of-distribution (OOD) state-action combinations that, by definition, are not present in D.
pertinent framework to learn performant
A key challenge stems from the inherent uncertainty when learning from limited data [28, 36). Failure to account for this can lead to wild extrapolation [17, 29] and over/under-estimatioo bias in value
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

estimates [22, 23, 32, 58]. This is systemic problem that is exacerbated for (OOD) state-actions where data is scaree. Standard temporal difference updates to Q-values rely on the Bellman optimality operator which implies these updates. As Q-values are updated with overestimated targets, they become upwardly biased even for state-actions well-represented in DD In turn, this can further increase the upper limit of the extrapolation errors at OOD state-actions, which forms a vicious cycle of extrapolationiinfal overestimation eextra-overestimaaiion for short) shown in Figure 1. This extra-overesimmatio is much more severe than the usual overestimation bias encountered in online RL [22. 58]. As such, we critically need to constrain value estimates whenever they lead to situations that look potentially 'too good to be true', in particular when they occur where a policy might exploit them.
estimates tend to dominate
Likewise, naive exploration can lead to policies that diverge significantly from TT. This, in turn, leads to even greater estimation error since we have very little data in this un(der)-explored space. Note that this is not reason for particular concern in online RL: after all, once we are done exploring a region of the space that turns out to be less promising than we thought, we simply update the value function and stop visiting or visit rarely. Not so in batch RL where we cannot adjust our policy based on observing its actual effects in the environment. These issues are exacerbated for applications with large number of possible states and actions, such as the continuous settings considered in this work. Since there is no opportunity to try out a proposed policy in batch RL, learning must remain appropriately conservative for the policy to have reasonable effects when it is later actually deployed. Standard regularization techniques are leveraged in supervised learning to address such ill-specified estimation problems, and have been employed in the RL setting well [12, 50, 62).
This paper adapts standard off-policy actor-critic RL. the batch setting by adding a simple pair of regularizers. In particular, our main contribution is to introduce two novel batch-RL regularizers: The first regularizer combats the extra-overestimation bias in regions that are out-of-distribution. The second regularizer is designed to hedge against the adverse effects of policy updates that severly diverge from (a(ss)) The resultant method, Continuous Doubly Constrained Batch RL (CDC) exhibits state-of-the-ard performance across 32 continuous control tasks from the D4RL benchmark [14] demonstrating the usefulness of our regularizers for batch RL.
2 Background
Here S is the state space, A C TR.d is a (continuous) action space, T S x A x S - R. encodes transition probabilities of the MDP, (4o denotes the initial state distribution, r(s, a) is the instantaneous reward obtained by taking action E A in state E S, and 7 E [0, 1] is discount factor for future rewards.
Consider an infinite-horizon Markov Decision Process (MDP) [48],
Given a stochastic policy m(ass), the sum of discounted rewards generated by taking series of actions ae ~ π x11(st) corresponds to the return R; = Ee - "r(s,, a;) achieved under policy æ,, Q²=(s, a), is defined as the expected return starting at state s, taking a, and acting according to π thereafter, (=(ssa) = E, Exc-Taar 1E yrr, (So, ag) = (s, a)). (T, a) obeys the Bellman equation [5]:
л. The action-value function (Q-value for short) corresponding
(s, r(s, a) + yE -BB-TT(laa))) (1s (s',aa))
(1)
Unlike in online RL, no interactions with the environment allowed here, so the agent does not have the luxury of exploration. D is previously collected via actions taken according to some unknown behavior policy To ma(a)s). In this work, we assume D consists of 1-step transition: {(s,, a,, Ti. of = where no further sample collection is permitted. In particular, our method, like [60, 63), only needs a dataset consisting of a single-step transitions and does not require complete episode trajectories. This is valuable, for instance, whenever data privacy and sharing restrictions prevent the use of the latter [34]. is also useful when combining data from sources where the interaction still in progress, e.g. from ongoing user interactions.
We aim to learn optimal policy x² that maximizes the expected return, denoting the corresponding Q-values for this policy as Q' QF" Q" is the fixed point of the Bellman optimality operator [5]: TQ' (s. a) = r(s. a) + ((, [max, (ss,''')) One way to learn x² is via actor-critic methods [27], with policy no and Q-value Qo. parametrized by and respectively.
Learning good policies becomes far more difficult in batch RL as depends on the quality/quantity of available data. Moreover, for continuous control the set of possible actions is infinite, making

---

nontrivial to find the optimal action even for online RL. One option is to approximate the maximization above by only considering finitely many actions sampled from some x. This leads the Expected Max-Q (FMaQ) operator of Ghasemipour et al. [18]:
TQ a) == r(s, a) + ((1s.a) max (s' ab) {of}}
(2)
Here dg. ~ [To(-ss') for l = L,.., NN, i.. the candidate actions are drawn IID from the current (stochastic) policy rather than over all possible actions. When drawing only single sample from T this reduces to the standard Bellman operator (in expectation). Conversely, when N + oo and me has support over A, this turns into the Bellman optimality operator. We learn Q by minimizing the standard 1-step temporal difference (TD) error.
That is, we update at iteration
O₂ f- argmin B(s.))DD (Qunss.)) TQQ, (8, a)) 27
(3)
Throughout, the notation Es (a.D)) denotes an empirical expectation over dataset D, whereas expec- tations with respect to are taken over the true underlying distribution corresponding to policy Next, we update the policy by increasing the likelihood of actions with higher Q-values:
Φ f argmax E (s
Qo. (s. a)]
(4)
using off-policy gradient-based updates [53]. Depending on the context, we omit t from Qo and To.
2.1 Extrapolation-Iniae Inflated Overestimation
When our Q-values are estimated via function approximation' (with parameters 0), the Q-update can be erroneous and noisy [58]. Let Qu, (s,a) denote the estimates of true underlying Q" a) values at iteration of batch RI. algorithm that iterates steps (3) and (4), with To, denoting the policy that maximizes Qa For proper learning method, we might hope that the estimation error, ER := Q9, (s. a) - Q* a). has expected value = 0 and variance o > 0 for particular states/actions (the expectation here is over the sampling variability in the dataset D and stochastic updates in our batch RL algorithm). However even in this desirable scenario, Jensen's 's inequality nonetheless implies there will be overestimation error OE == E [max. Qo, (s, a)] max, Q" (a s) 2 0 for the actions currently favored by To. Here the expectation is over the randomness of the underlying dataset D and the learning algorithm. OE will be strictly positive when the estimation errors are weakly correlated and will grow with the ER-variance a [32, 58]. Under the Bellman optimality or EMaQ operator, these inflated estimates are used as target values in the next Q-update in (3), which thus produces Q (s, a) estimate that suffers from overestimation bias, meaning it is expected to exceed the true Q value even if this was not the case for initial estimate Qo, (s, a) [13, 16, 22, 23, 30, 33, 58].
In continuous batch RL, ER may have far greater variance (larger σ) for OOD states/actions poorly represented in the dataset D, our function approximator Qo, may wildly extrapolate in these data-scarce regions [4]. This in turn implies the updated policy Te. will likely differ significantly from To and favor some action à argmax, Qo, (s, a) that is OOD [17]. The estimated value of this OOD action subsequently becomes the target in the Q-update (3), and its OE will now be more severe due to the larger a [7]. Even though we only apply these Q-updates to non-OOD (s,a) a) E DD whose ER may be initially smaller, the severely overestimated target values can induce increased overestimation bias in Q+.:1 a) for (s,a) E D. In vicious cycle, the increase in Q (s,a) a) for (s, a) E D can cause extrapolated Qe... estimates to also grow for OOD actions (as there is no data to ground these OOD estimates), such that overestimation at s, a E D is further amplified through additional temporal difference updates. After many iterative updates, this extra-ooverssimaatin can eventually lead to the disturbing explosion of value estimates seen in Figure 1.
Several strategies address overestimation [13, 16, 22, 23, 30, 33]. Fujimoto et al. [17] proposed straightforward convex combination of the extremes of an estimated distribution over plausible Q values. Given a set of estimates Q0, for j = 1,..MM, they combine both the maximum and the minimum value for a given (s, a) pair:
(s,5 = min Qa a) + (1 - v) max (s,
(5)
'hiiee overestimation bias has been mainly studied in regard to function approximation error [13, 16, 23, 33, 58), Hasselt [22] shows that overestimation also arise in tabular MDPs due noise in the environment.
3

---

Here E (0. 1) determines how conservative we wish to be, and the min/max are taken across M Q-networks that only differ in their reight-initializttinn but are otherwise (independently) estimated. For larger > 0.5, Q may be viewed as a lower confidence bound for Q" where the epistemic uncertainty in Q estimates is captured via an ensemble of deep Q-networks [9].
3 Methods
Our previous discussion of extra-overestimation suggests two key sources of potential error in batch RL. Firstly, policy learned by our algorithm might be too different from the behavior policy, which can lead to risky actions whose effects are impossible to glean from the limited data. To address this, we propose to add an exploration-penalty in policy updates that reduces the divergence between our learned policy & and the policy my that generated the data. Secondly, we must restrict overestimation in Q-values, albeit only where it matters, that is, only when this leads to a policy exploiting overly optimistic estimates. As such, we only need to penalize suspiciously large Q-values for actions potentially selected by our candidate policy Th (e.g. if their estimated Q-value greatly exceeds the Q-value of actually observed actions).
3.1 Q-Value Regularization
While sequential interaction with the environment is a strong requirement that limits the practical applicability of online RL (and leads to other issues like exploration vs. exploitation), has one critical benefit: although unreliable extrapolation of Q-estimates beyond the previous observations happens during training, it is naturally corrected through further interaction with the environment. OOD state-actions with wildly overestimated values are in fact likely to be explored in subsequent updates, and their values then corrected after observing their actual effect.
In contrast, overesstttsttnnionnnnn is far more severe issue in batch RL, where we must be confident in the reliability of our learned policy before it deployed. The issue can lead to completely useless Qeestimates. The policies corresponding to these wildly extrapolated Q-function will perform poorly, pursuing risky actions whose true effects cannot be known based on the limited data in D (Figure shows an example of how extra-overetimmation can lead to the disturbing explosion of Q-value estimates).
To mitigate the key issue of extra-overestimaation in Qe(s, a), we consider three particular aspects:
An overall shift in Q-value is less important. A change from, say Qo a) to Qo a) + c(s) changes nothing about which action we might want to pick. As such, we only penalize the relative shift between Q-values.
An overestimation of Qo(s, â) which still satisfies Qa a) < Qo(s, a) for well-established a, s E ID will not change behavior and does not require penalization.
Lastly, overestimation only matters if our policy capable of discovering and exploiting
We use these three aspects to design penalty for Q-value updates to be more pessimistic [7, 25).
2
( a) = (a ( (+1)) Qo(s, à) - Q0(s,a)
max
where a E D. We can see that the first requirement is easily satisfied, since we only compare differences Qe(s, a) - Qo a) for different actions, given the same state s. The second aspect is addressed by taking the maximum between 0 and Qo(s. a) Qa(s, a). As such, we do not penalize optimism when it does not rise to the level where it would effect a change in behavior. Lastly, taking the maximum over actions drawn from the æ rather than from the maximum over all possible actions ensures that we only penalize when the overestimation would have observable consequences. As such, we limit ourselves to a rather narrow set of cases. As a result, we add this penalty to the Q-update:
0x < argmin B(S,a)~D (0otss - TQo,
A(s,a)
Anatomy of the extra-overestimation penalty A. Our proposed A penalty (6) mitigates extra- overestimation bias by hindering the learned Q-value from wildly extrapolating large values for OOD

---

state-actions, Estimated values of actions previously never seen in (known) state E D are instead encouraged to not significantly exceed the values of the actions whose effects we have seen at Note that the temporal difference update and the extra-overestimation penalty A in (7) are both framed on a common scale as a squared difference between two Q-functions.
How A affects / becomes evident through its derivative:
V VoA(s,a)= - {
( VoQo(s,,) - Volo(s,a)) i
(8)
otherwise
Here = mag Qo(s, âx) again taken N actions sampled from our current policy π, and ε = - a). A only affects certain differel-diferrences where Q-values of (possibly OOD) state-actions have higher values than the (s,a) E D. In this case, A not only reduces (s ä) by an amount proportional to e, but this penalty also increases the value of the previously-observed action (s a) to the same degree. A thus results in a value network that favors previously observed actions. We will generally want to choose a large conservative value of r in applications where we know either: that the behavior policy was of high-quality (since its chosen actions should then be highly valued), or that only a tiny fraction of the possible state-action space is represented in D, perhaps due small sample-size restricted behavior policy (since there may be severe extrapolation error).
3.2 Policy Regularization
In batch RL, the available offline data can have varying quality depending on the behavior policy Te used to collect the data. Since trying out actions is not possible in batch settings, our policy network instead updated to favor not only actions with the highest estimated Q-value but also the actions observed in D (whose effects can be more certain of). Thus we introduce an exploration penalty
to regularize the policy update step: & <- argmax & EewD (-11)))))) Qo a) X D(Tb: To)-
In principle, various f-divergences [10] or Integral Probability Metries [42] could employed in D(.,-).
In practice, we limit our choice to quantities that do not require estimating the behavior policy T&. This leaves us with the reverse Kl-ddivegeence and IPMs in Hilbert Space [2]. we further restrict ourselves to distances that do not require sampling from क.. then only the reverse KL--divergence remains. We thus estimate
(TT, no) EL Et To (a|s)] E (- F (a[s)]
--Rawm (a
log (a,ls)
(10)
whenever a, TT(sss). This is exactly what happens in batch RL where we have plenty of data drawn from the behavior policy, albeit no access to its explicit functional form. Note the first entropy
term in (9) can be ignored when we aim to minimize the estimated KI in terms of To (as will be done in our exploration penalty). Using (10), we can efficiently minimize an estimated reverse KI. divergence without having know/estimate ma or sample from Tg.
Lemma argmaax Esgan 1a,awa/[Qoss,, a)] - A A. ((T)(()))) (o(()))) is given by
(a(s))
Qo(s,a) a)
(T(sa))=
Z exp
D is the forward KL divergence (iteerenee_KKrrt(lll)s....)
(a(a)s)
ns(s)a)=
if the reverse KL divergence = KT(mo(-)s) Qo(s, a)A
where Z E R is normalizing constant in each case. Lemma shows that using either forward or reverse KL-divergence as an objective, we recover Fe = n5 in the limit of X xo. This is to be expected. After all, in this case use the distance in distributions (thus policies) as only criterion, and we prefer reverse KL to avoid having to estimate Ty. CDC thus employs the following policy-update (where the reverse KL. is expressed as a log- likeeloood as in (10)
f- argmax E, B, P amm (11s) (0(s.a)) a)] + A (s,a)) (ags)(a(s)
(11)
5

---

The exploration penalty helps ensure our learned ve is not significantly worse than Th, which is far from guaranteed in batch settings without ever testing an action. If the data were collected by fairly random (subpar) behavior policy, then this penalty (in expectation) acts similarly to maximum-entropy term. The addition of such terms to similar policy-objectives has been shown to boost performance in RL methods like soft actor-critic [21].
Note that our penalization of exploration stands in direct contrast to online RL methods that specifi- cally incentivize exploration [4, 44]. In the batch RL, exploration is extremely dangerous as it will only take place during deployment when policy is no longer being updated in response to the effect of its actions. Constraining policy-updates around an existing data-generating policy has also been demonstrated as reliable way to least obtain an improved policy in both batch [17. 63] and online [51] settings. Even moderate may be too much ask for with data of limited size or coverage of the possible mate-actions). Reliable improvement is crucial in batch settings as we cannot first test out our new policy.
can often be extremely valuable (the optimal policy
Remark 1 (Behavioral cloning occurs as X oc) Regularized policy updates with strong regular- ization (large X) is in the limit imitation learning. In fact, this is the well well-known likelihood based behavioral cloning algorithm used by [47].
If the original behavior policy ni was optimal (e.g. demonstration by a human-expert), then behavioral cloning should be utilized for learning from D [43]. However in practice, data are often collected from a subpar policy that we wish to improve upon via batch RL rather than simple imitation learning.
3.3 CDC Algorithm
Algorithm I Continuous Doubly Constrained Batch RL
Furnished with the tools for Q-value and policy regularization proposed in previous sections, introduce CDC in Algorithm 1. CDC utilizes
I: Initialize policy 5u and Qs: [Qo,]==
2: Initialize Target Qs: (Qo; 03 + 0,1341
3: for in [I..... T) do
an actor-critic framework [27] for continuous actions with stochastic policy To and Q-value Qo. parameterized by & and / respectively. Our major additions to that A penalty that mitigates overestimation bias by reducing wild extrapo- lation in value estimates and the exploration penalty (log Tÿ)) that discourages the estimated policy from straying to OOD state-actions very different from those whose effects we have ob- served in D.
4: Sample = {(s.a,
~
5: For each s, E B: sample N actions (ax Ran
<a(+))), 1o())=11 mo(-1s")
6: Qo value update:
u(s'r) ()),r):=ro := + of
To (s. a4) given by Eq
(,(r,) = ([ max Qu, (x. - Qu, (v.00) )
0g +- argmin E [ (2.)) - 2(()r:))
+ n A5(, for 1,..,
Although the particular form of CDC presented in Algorithm I optimizes a stochastic policy with the off-policy updates of [53] and temporal difference value-updates using (3), we empha- size that the general idea behind CDC can be utilized with other forms of actor-critic updates such as those considered by [12. 16, 21]. In practice, CDC estimates expectations of quanti- ties introduced throughout via mini- estimates derived from samples taken from D, and each optimization is performed via a few stochastic gradient method iterates.
7: Fy policy update:
> (o)ls + 22
argmax
8:
Update Target Networks:
8
+ +
(1
()] Vj E M
end for
To account for epistemic uncertainty due the limited data, the value update in Step 6 of Algorithm I uses 2, from (5) in place of Qg. In CDC,
simply utilize the same moderately conservative value of 0.75 used by [17], since are not purely relying on the lower confidence bound Qa to correct all overestimation. For this reason, CDC is able to achieve strong performance with a small ensemble of M = 4 Q-networks (used throughout this work), whereas [18] require larger ensembles of 16 nentwwrrks and an extremely conservative V/ = in order to achieve good performance.
To correct oxeretoiverttiimatioo within each of the M individual Q-networks, Algorithm actually applies separate extra-overestimation penalty A specific to each Q-network. The steps of our proposed CDC method are detailed in Algorithm 1. In blue, we highlight the only modifications CDC makes to standard off- off-policy actor-critic framework that has been suitably adapted for continuous batch RL via the aforementioned techniques like EMaQ [18] and lower-confidence bounds for Q-

---

values [17]. Throughout, we use 7 0 & À = 0 to refer to this baseline framework (without our proposed penalties), and note that majority of modern batch RL methods like CQL [29], BCQ [17], BEAR [28], BRAC [63] are built upon similar frameworks.
Although each of our proposed regularizers is modular, we emphasize that they complement each other: the Q-Value regularization mitigates extra-overestimation error while the policy regularizer ensures candidate policies do not stray too far from the offline data. Ablation studies show that the best performance is only achieved through simultaneous use of both regularizers (Figure 2a, Table S1). Note that CDC is quite simple to implement: each penalty can be added to existing actor-critic RL frameworks with minimal extra code and the addition of both penalties involves no further complexity beyond the sum of the parts.
be used independently and their implementation
Theorem For Qo in (5), let TCnC : Qo. - To.,, denote the operator corresponding to the Qo- updates resulting from the tth iteration of Steps 6-7 of aAlgorithm I. TTDc is Los contraction under standard conditions that suffice for the ordinary Bellman operator to be contractive [3, 6, 8, 56).
The proof and formal list of assumptions are in Appendix D.1. Together with Banach's 's theorem, the contraction property established in Theorem above guarantees that our CDC updates converge to a fixed point under commonly-assumen conditions that suffice for standard RL algorithms to converge [31]. Due to issues of (nonconvex) function approximation, it is difficult to guarantee this in practice or empirical optimality of the resulting estimates [38, 40]. We do note that the addition of our two novel regularizers further enhances the contractive nature and stability of the CDC updates when n7 A > by shrinking Q-values and policy cetton-probabilitee toward the corresponding values estimated for the behavior policy (i.e. values computed for observations in D). Our CDC penalties can thus not only lead to less citty-eetrapplated batch estimates, but also faster (and more stable) convergence of the learning process (as shown in Figure 1, where Standard actor-critic refers to Algorithm where 1)
0).
Let T& E 11 be the policy learned by CDC, 7 denote discount factor, and denote the sample size of dataset D generated from Th- Also let (xx) represent the true expected return produced by deploying policy πt in the environment. Under mild assumptions listed in Appendix D, there exist constants r", ,Cx, V such that with high probability 2 1-
Theorem 2
J(TT) 2 J(Tx) -( - + -V--log 8)/n
r"
Appendix D.2 contains proof and descriptions of the assumptions in this result. Theorem 2 assures us of the reliability of the policy Fs produced by CDC, guarantceing that with high probability ne will not have much worse outcomes than the behavior policy Te. where the probability here depends on the size of the dataset D and our choice of policy regularization penalty À (the constant Cx is decreasing function of A). In batch settings, expecting to learn the optimal policy is futile from limited data. Even ensuring any improvement at all over an arbitrary T. is ambitious when we cannot ever test any policies in the environment, and reliability of the learned Tı is thus major concern.
Let Ogag E [max, Qo(s, a)] - max, Q' a) be the overestimation error in actions favored by an agent ag. Here Qa denotes the estimate of true Q-value learned by ag, which may either use CDC (with n > 0) or a baseline version of Algorithm I with 0 (with the same value of A). Under the assumptions listed in Appendix D).3, there co-exist constants Lı and To such that
Theorem 3
OEeDe S L1 11 S OEnaseline
This theorem (proved in Appendix D.3) underscores the influence of the 1) parameter in terms of containing the overestimation problem in offline Q-learning. Mitigating this overestimation, which can be done using non-zero 77, can ultimately lead into better returns as we show in the experimental section. In particular, CDC achieves lower overestimation by deliberately underestinating Q-values for non-observed state-actions (but it limits the degree of downward bias as described in Remark 2). Buckman al. [7], Jin al. [25] prove that some degree of pessimism is unavoidable to ensure non-catastrophic deployment of batch RL in practice, where it is unlikely there will ever be sufficient data for the agent to accurately estimate the consequences of all possible actions in all states.
Remark 2 (Pessimism is limited in CDC) Extreme pessimism leads to overly conservative policies with limited returns. The degree of pessimism in CDC remains limited (capped once A = 0), unlike lower-confidence bounds which can become arbitrarily pessimistic and hence limited in their return.
7

---

4 Related Work
Aiming for practical framework to improve arbitrary existing policies, much research has studied batch RL [34, 36] and the issue of overestimation [22, 23, 58). [26, 64] consider model-based approaches for batch RL, and [1] find ensembles partly address some of the issues that arise in batch settings, To remain suitably conservative, a popular class of approaches constrain the policy updates to remain in the vicinity of TT, via, e.g.. distributional matching [17], support matching [28, 63], imposition of a behavior-based prior [52], or implicit constraints via selective policy-updates [46, 60]. Similar to imitation learning in online setting [24, 43, 47, 49], many of such methods need to explicitly estimate the behavior policy [17. 18, 28]. Although methods like [46, 60] do not have an explicit constraint the policy update, they still can be categorized
policy constrained- based approach as the policy update rule has been changed in a such a way that it selectively updates the policy utilizing information contained in the Q-values. Although these approaches show promising results, policy-constraint methods often work best for data collected from a high-quality (expert) behavior policy, and may struggle to significantly improve upon highly suboptimal Th- Compared to the previous works, our CDC does not need to severely constrain candidate policies around Ty. which reduces achievable returns. Even with a strong policy constraint, the resulting policy is still affected by the learned Q-value, thus we still must correct Q-value issues. Instead of constraining policy updates, [29] advocate conservatively lower-bounding estimates of the value function. This allows for more flexibility to improve upon low-quality т-- [39] considers pessimistic and conservative approach to update Q-value by utilizing the marginalized state-action distribution of available data. and literature, demonstrating these address complementary issues of the batch RL problem and are both required in a performant solution.
Our proposed CDC algorithm is inspired by ideas from both the
Experiments
In this section, we evaluate our CDC algorithm against existing methods 32 tasks from the D4RL benchmark [14]. We also investigate the utility of individual CDC regularizers through ablation analyses, and demonstrate the broader applicability of our extra-overestimation penalty to off-policy evaluation in addition to batch RL. Our training/evaluation setup exactly follows existing work [14, 17, 28, 29J. See Appendices A, B, and C for a complete description of our experimental pipeline.
Setup. We compare CDC against existing batch RL methods: BEAR [28), BRAC-V/P [63], BC [63], CQL [29]. BCQ [17], RBVE2 [20] and SAC [21]. This covers a rich set of strong batch RL methods ranging from behavioral cloning to taltue-oonntrinnd--aaaee
pessimistic methods, with the exception of SAC. SAC is a popular off-policy method that empirically performs quite well in online RL, and is included to investigate how online RL methods fare when applied to the batch setting. Note that CDC was simply run on every task using the same network and the original rewards/actions provided in the task, without any manual task-specific smoothing. Moreover, all these baseline methods also utilize an ensemble of Q networks as in (5).
Results. Figure 2b and Table illustrate that CDC performs better than the majority of the other batch RL methods the D4RL tasks. Across all 32 tasks, CDC obtains normalized return of 1397, whereas the next-best method (CQL) achieves 1 1245. In head-to-head comparisons, CDC generates statistically significantly greater overall returns (Table 1). Unsurprisingly, behavioral-cloning (BC) works well on tasks with data generated by an expert Tr. while the online RI. method, SAC, fares poorly in many tasks. CDC remains reasonably competitive across all tasks, regardless of the environment or the quality of Th (i.e. random vs. expert).
Next we perform a comprehensive set of ablation studies to gauge the contribution of our proposed penalties in CDC. Here we run additional variants of Algorithm without our penalties (i.e. 17 = A 0), with only our estra-overestimation penalty (A = 0), and with only our exploration penalty (1) = 0). Figure 2a and Tables SI show that both penalties are critical for the strong performance of CDC, with the oxtra-overestimatiio penalty A being of greater importance than exploration (see also Figure 2a). Note that all our ablation variants still employ the lower confidence bound from (5), which alone clearly does not suffice correct oetresoverstiintioon.
Comparison between CDC and the concurrently-proppeee RBVE method [20] is relegated to Section A.I.
8

---

D0C 7-08
OOL
BTACA
DPAGP
-20
-20
-00
-100
12
05 Taska
10
14-15
16 Taaka
(b)
Figure 2:
Difference in (normalized) return achieved by various algorithms vs CDC in 32 D4RL tasks. X-axis colors indicate environments (sce Table 1), and points below the line ( (-) indicate worse performance than CDC. Figure 2a shows that fixing 1) or À to zero (i.c. omitting our penalties) produces far worsc returns than CDC (sec also Table S1). This ablation study proves that major performance gains for CDC stem from our novel pair of regularizers, as the only difference between CDC and these ablated variants either 1) or or both are set to zero in Algorithm (all other details are exactly the same). Figure 2b compares CDC against existing batch RL algorithms, where CDC overall compares favorably to each other method in head-to-head comparisons (see also Table 1). Note these figures can be compared to each other well.
1.00
5.1 Offline Policy Evaluation
Correlation Coefficient
0.50
The true practical applicability of batch RL re-
mains however hampered without the ability to do proper algorthnnhhypeppramettrr selec- tion. Table shows that no algorithm universally dominates all others across all environments or In practice, it is difficult to know which technique will perform best, un- less can do proper offline policy evaluation (OPE) of different candidate policies before their actual online deployment [15, 45).
0.00
-0.25
-0.90
FGE
-0:75
behavior-policee
Figure 3: How well OPE estimates correlate with ac- tual return achieved by 20 different policies for each D4RL task. Due unmitigated overestimation, FQE estimates correlate negatively with true returns 15 of 32 tasks (using A in FQE reduces this to 4).
Tasks
OPE aims to estimate the performance of a given policy under the same setting considered here, with offline data collected by unknown behavior policy [35, 45]. Beyond segorithn//yperpaaameee comparison, OPE is often employed for critical policy-making decisions where environment interac- tion is no longer an option, e.g., sensitive healtheare applications [19]. One practical OPE method for data of the form in D is Fitted Q Evaluation (FQE) [35]. To score given policy æ, FQE iterates temporal difference updates of the form (3) using the standard Bellman operator from (1) in place of EMaQ. After learning an estimate QT, FQE simply estimates the return of via the expectation of Q (sss a) over the initial state distribution and actions sampled from æ.
However, like batch RL, OPE also relies on limited data and thus can still suffer from severe Q-value estimation errors. To contain the overestimation bias, we regularize the FQE temporal difference updates with our A penalty, in a similar manner to (7). Figure 3 compares the performance of throughout) against the standard unregularized FQE. Here
A A-penalization of FQE (with r we use both OPE methods to score 20 different policies (learned via different settings) and gauge OPE-quality via the Pearson correlation between OPE estimated returns and the actual return (over our 20 policies). We observe higher correlation for FQE + A (0.37 on average) over FQE (0.01 on average) in the majority of tasks, demonstrating the usefulness of our regularizers. The usefulness of our regularizers thus extend beyond batch RL and carry to the off-policg evaluation setting.
'The results for CQL are taken from the official author provided codes [https://githut com/ sviralkumar2907/ /CQL of [29]. The published CQL codes are used to produce results for all but Adroit and FrankaKitchen where the codes are not available. For these latter domains, we simply use the CQL results reported in the paper of [29].

---

Index
lask Name
SAC
BC
BRAC-F
BRAC-V
BEAK
BCQ
CQL
A 0 1)
CDC
hnffeheeted-mmmionn
29.6
2.1
23.5
28.1
25.5
2.25
16.71
32.8
27.36
hnllshoetab-mediim
40.97
36.1
44.0
45.6
38
51
46.05
2
halfcheetah medium replay
36.47
38.1
15.6
45.9
36.2
34.79
12.77
22.72
11.74
helfehectch--ee ceceetttddniii-epppeeeee
3.78
35.8
43.8
45.3
51.7
69.64
39.18
7.12
59.64
-0.41
107.0
3.8
83.
5
hupper random
0.8
9.8
11.1
12.0
9.5
10.6
10.37
1.58
14.76
pupper-medium
0.81
23.0
31.2
32.3
47.6
65.07
51.79
0.58
60.39
hopper medium replay
8
mnnge--ndium-eepet
8.96
119.9
1.1
0.8
4.0
108.
112.46
18.07
86.9
9
hopper-xpert
0.8
100.0
6.6
3-7
110.3
75.52
109,997
1.27
102.75
11
walker 25 medium
0.51
6.6
72.7
81.3
$1.2
52.03
71.03
0.31
82.15 13
12
walker 28--maiim---ee
0.04
11.3
0.3
0.9
10.8
13.67
19.56
3.81
22.96
medium expent
14
walker walker22-experr expert
0.05
125.7
0.2
0.00
J06.1
87.59
103.6
87.54
15
ont cntmarc-umsee
0.0
65
50
70
73
88.52
90.12
22.22
91.85
16
0.0
55
70
61.11
11.11
62.59
0.0
0.0
40.74
18
anemaze medium diiverse
0.0
0
0
0.0
40.71
0.0
40.74
19
play
0.0
0
1.85
0.0
0.0
11
21
pen humam
-1.15
314
8.1
0.6
1.
66.88
37.5
-3-43
73.19
22
pen-cloned
-0.64
56.9
-25
26.5
50.56
39.2
-3.4
49.18
hamnerhumma
1.5
24
hammercluned
0.27
0.8
0.3
0.3
0.3
0.38
2.1
0:26
2 27
25
doar-human
-0.34
0.5
0.33
-.3
0.3
0,05
9.9
0.16
4.02
27
relocate- haman
-0.31
0
0.3
00.3
0.3
0.04
0.2
0.73
28
eclecatsctoned
00.11
0.1
0.3
0.3
0.2
0,228
01.1
0.15
01.24
complete
30
kitchen partial
0.0
33.8
13.1
9.26
49.8
0.0
42.5
31
kithhe-miiaa
0.0
47.5
47.2
18.43
51
0.0
42.87
Total Score
116.28
984.4
383.4
434.5
844.0
1060044
1245./2
173.67
18996.99
p-value
7.06007
1.6e-03
5.3c-07
3.96-06
T.1e-04
6.1c-04
3.6c-02
1.8c066
Table 1: Return achieved in deployment of policies learned via different batch RL methods. The return in cach environment here is normalized using (12) as originally advocated by [14]. For cach method: we perform head-to-head comparison against CDC across the D4RL tasks, reporting the p-value of a (one-sided) Wilcoxon signed rank test [61] that compares this method's return against that of CDC (over the tasks). Here = 0 y = is variant Algorithm without our penalties where proves that major performance gains for CDC stem from our novel pair of regularizers.
6
Discussion
Here we propose a simple and effective algorithm for batch RL by introducing a simple pair of regularizers that abate the challenge of learning how to act from limited data. The first constrains the value update to mitigate extra-overestimation error, while the latter constrains the policy update to ensure candidate policies do stray too far from the offline data. Unlike previous work, this paper highlights the utility of simultaneous policy and value regularization in batch RL. One can envision other combinations of alternative policy and value regularizers that may perform even better than the particular policy/value penalties used in CDC. That said, our CDC penalties are particularly simple to incorporate into arbitrary actor-critic RI. frameworks, and operate synergistically as illustrated the ablation studies. Comprehensive experiments on standard offline continuous-control benchmarks suggest that CDC compares favorably with state-of-the- methods for batch RL, and our proposed penalties are also useful to improve offline policy evaluation. The broader impact of this work will hopefully be to improve batch RL performance offline applications, but we caution that unobserved confounding remains another key challenge in real-world data that was not addressed in this work.
10

---

References
[1] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning, In International Conference Machine Learning. 2020.
[2] Y. Altun and A. Smola. Unifying divergence minimization and statistical inference via convex duality. In International Conference on Computational Learning Theory, pages 139-153. Springer, 2006.
[3] A. Antos, R. Munos, and C. Szepesvari. Fitted Q-iteration in continuous action-space MDPs. In Advances in Neural Information Processing Systems, 2007.
[4] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29, pages 1471 1471-1479, 2016.
[5] R. E. Bellman. Dynamic Programming. Princeton University Press, 1957.
[6] D. P. Bertsekas and S. Shreve. Stochastic optimal control: the discrete-time case. Athena Scientific, 2004.
[7] Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism fixed-dataset policy optimization. In International Conference on Learning Representations, 2021.
[8] L. Busoniu, R. Babuska, B. De Schutter, and D. Ernst. Reinforcement learning and dynamic programming using function approximators, volume 39. CRC press, 2010.
[9] R. Y. Chen, S. Sidor, P. Abbeel, and J. Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv: 111.110060000000 2017.
[10] I. Csiszár and P. Shields. Information theory and statisties: tutorial. Foundations and Trends in Communications and Information Theory, 1(4):417-528, 2004. ISSN 1567-2190. doi: 10. 00.151/01000000044
[11] G. Dulac-Amold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learning. In ICML Reinforcement Learning for Real Life (RLARealLife) Workshop. 2019.
[12] R. Fakoor, P. Chaudhari, and A. J. Smola. P3O: policy-on policy-off policy optimization. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, page 371, 2019.
[13] R. Fakoor, P. Chaudhari, and A. J. Smola. Ddpg++: Striving for simplicity in continuous-control off-policy reinforcement learning. 1511:9, 2020.
[14] J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. wxXh:2204.00729, 2020.
[15] J. Fu, M. Norouzi, O. Nachum, G. Tucker, Z. Wang, A. Novikov, M. Yang. M. R. Zhang, Y. Chen, A. Kumar, C. Paduraru, S. Levine, and T. L. Paine. Benchmarks for deep off- off-policy evaluation, 2021.
[16] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR, 2018.
[17] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pages 2052-2062, 2019.
[18] S. K. S. Ghasemipour, D. Schuurmans, and S. S. Gu. Emaq: Expected-matt q-learning operator for simple yet effective offline and online rl. arXiv:2007.1 1091, 2021.
[19] O. Gottesman, J. Futoma, Y. Liu, Parbhoo, Celi, E. Brunskill, and F. Doshi-Velez. Inter- pretable off policy evaluation in reinforcement learning by highlighting influential transitions. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3658-3661 PMLR, 2020.

---

[20] C. Gulcehre, S. G. Colmenarejo, Z. Wang, J. Sygnowski, T. Paine, K. Zolna, Y. Chen, M. Hoff- man, R. Pascanu, and N. de Freitas. Regularized behavior value estimation, 2021.
[21] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv:1801.01200 2018.
[22] H. V. Hasselt. Double Q-learning, In Advances in Neural Information Processing Systems 23, pages 2613-2621, 2010.
[23] H. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'1,, page 2094-2100. AAAI Press, 2016.
[24] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac- Arnold, I. Osband, Agapiou, J. Z. Leibo, and A. Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR, abs/1704.03732, 2017.
[25] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline RL? arXiv preprint arXiv:2012. 15085, 2020.
[26] R. Kidambi, Rajeswaran, P. Netrapalli, and T. Joachims. Morel Model-based offline reinforcement learning. wrXiv:2005.0595// 2020.
[27] V. Konda and Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 1008-1014. MIT Press, 2000.
[28] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. 1906009906000499 Nov. 2019.
[29] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-Learning for Offline Reinforce- ment Learning. arXiv:2006.04779 June 2020.
[30] A. Kuznetsov, Shvechikov, A. Grishin, and D. Vetrov, Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5556-5566. PMLR, 13-18 Jul 2020.
[31] M. G. Lagoudakis and R. Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1 1111-11-4,, 2003.
[32] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-leamning In International Conference on Learning Representations, 2019.
[33] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-learning In International Conference on Learning Representations, 2020.
[34] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. In M. Wiering and M. van Otterlo, editors, Reinforcement Learning: State-of-the-Arz pages 45-73. Springer, 2012.
[35] H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3703-3712. PMLR, 2019.
[36] Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. 0rXiv:2005.01643 May 2020.
[37] Y. Li. Deep reinforcement learning: An overview. arXiv preprint 1 2017.
[38] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
12

---

[39] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch off-policy reinforcement learning without great exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1264-1274. Curran Associates, Inc., 2020.
[40] G. Matheron, N. Perrin, and O. Sigaud. The problem with ddpg: understanding failures in deterministic environments with sparse rewards. arXiv preprini 1rXiv:1911.1 2019.
[41] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, 2015.
[42] A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997. ISSN 00018678.
[43] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179, 2018.
[44] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 2721-2730. PMLR, 06-11 Aug 2017.
[45] T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055, 2020.
[46] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage weieatteeewwegghedd regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00077, 2019.
[47] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-99) 1991.
[48] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. USA, 1st edition, 1994. ISBN 0471619779.
[49] S. Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In G. J. Gordon, D. B. Dunson, and M. Dudík, editors, AISTATS, volume 15 of JMLR Proceedings, pages 627-635. JMLR.org, 2011.
[50] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
[51] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv::1077: 170..06347 2017.
[52] N. Siegel. J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[53] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387-395. PMLR, 2014.
[54] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. den Driessche, J. Schrittwieser, I. Antonogiou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, Lillicrap, M. Leach, K. Kavukcuoglu. T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016.
[55] D. Silver, T. Hubert, J. Schritwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanetot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018. ISSN 0036-8075. doi: 10.
13

---

[56] C. Szepesvári. Efficient approximate planning in continuous space markovian decision problems.
Al Communications, 14(3):63-176,, 2001.
[57] Thomas, G. Theocharous, and M. Ghavamzadeh. High confidence policy improvement. In
International Conference Machine Learning, pages 2380-2388. PMLR, 2015.
[58] S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School, pages 255-263. Lawrence Erlbaum, 1993.
[59] Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[60] Wang. Novikov, K. Zolna, T. Springenberg, S. Reed, B. Shahriari, N. Siegel, Merel, C. Gulcehre, N. Heess, and N. de Freitas. Critic regularized regression. 2ti::2000.11:3:3 2020.
[61] Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-33, 1945.
[62] R. J. Williams and J. Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-26,, 1991.
[63] Y. Wu, G. Tucker, and O. Nachum. Behavior Regularized Offline Reinforcement Learning. 1911.
2019.
[64] Yu, G. Thomas, L.. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv:2005. /3239, 2020.
14


Reviews:
[{"review_id": "pFgy4eSsqR4", "reviewer": "Reviewer_XtLh", "summary": "The authors propose Continuous Doubly Constrained (CDC) Batch RL. It adds two straightforward penalties, a policy constraint and a value-constraint. The policy constraint prevent the policy from taking actions that are not supported by the dataset, and the value-constraint prevents the value from being overly optimistic. Both penalties improve performance on the offline RL benchmarks compared to the baseline, but the two penalties combined work substantially better. They also perform competitively against several other offline RL baseline agents (BC, BCQ, BEAR, BRAC-V, BRAC-P, CQL).", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality:\nThe method is a novel combination of well-known techniques. Specifically, it combines policy-constraint offline RL methods and value-constraint offline RL methods. Both have been prevalent in the literature, and this paper shows that there is a clear advantage to combining both approaches. It is clear to me how this method differs from previous contributions, and related work is adequately cited.\n\n## Quality:\nOverall I think the paper is technically sound, and its main claims are well supported.\nIt shows through experimental results that combining policy-constraints and value-constraints has a clear advantage across a well established benchmark for offline RL. And shows that combined constraints outperforms a number of baseline agents. \n\nThe main weakness in my mind is the authors introduce a new policy-constraint and a new value-constraint, but it is not clear the policy-constraint is better than existing policy-constraint methods e.g. BCQ, BRAC, or that the value-constraint is better than existing value-constraint methods e.g. CQL. \n\nIndeed the best PC method BCQ gets 1060, while theirs gets 299, and the best VC method CQL gets 1245, while theirs gets 555.\n\nIs that correct? I am comparing results across tables in the appendix so I am not sure if my comparisons are fair.\n\nCan the authors explain why they think their PC and VC methods perform so much worse than the baselines? I think it may be interesting to make these comparisons in the main body of the paper, and clarify the discrepancies.\n\n## Clarity:\nOverall I think the paper is clearly written and well organized. And I think an expert reader would be able to reproduce its results.\n\n## Significance:\nI think the results are important. Offline RL is an important research direction and these results show that policy and value constraints can be combined to get better performing algorithms. In that sense, it addresses a difficult task in a better way than previous work.\n\n## Additional Questions:\nOn line 79 you mention \"In particular, we do not even require that complete episode trajectories have been logged [59, 62].\"\nCan you clarify why CRR and BRAC require complete episode trajectories?\n\nIn Section 5.1 you show results for Offline Policy Evaluation. Your FQE baseline on the D4RL tasks performs quite poorly in comparison to the FQE baseline in [this benchmark paper](https://arxiv.org/abs/2103.16596), which also uses policies trained on the D4RL datasets. Do you have any idea what might account for this difference?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact are adequately addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "jNh2z4KaYBH", "reviewer": "Reviewer_jjjy", "summary": "The paper proposes a new algorithm for batch Reinforcement Learning, which differs from existing solutions by the addition of an additional term to the critic loss, aimed at penalising overestimation as well as adding a behavior-cloning-style term to the actor loss, aiming at keeping generated actions close to the distribution of data in the batch. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The is a good fit for the NeurIPS community in terms of scope. It shows good empirical results. The paper is clearly written, although there are some minor omissions (see “minor points” below).\n\nMy main concerns are the following.\n1. The novelty of the contribution isn't huge - the paper is an incremental extension of previous techniques.\n2. Since the paper seems to be mainly an empirical contribution, the ability to reproduce the results is critical. However, the authors do not provide the source code and the information provided in the appendix is limited to pseudocode and a list of hyperparameters. \n3. The paper discusses both forward and reverse KL for the policy regularization but only uses the reverse KL. It is not clear what the benefit of including the forward KL in Lemma 1 is (it makes the paper harder to read because one is kept wondering when it will be used).\n4. The proofs are a bit too informal. For example, line 774 in the appendix reads “From standard empirical process theory used throughout statistical learning [85], we have that the following bound simultaneously holds”. This does not really qualify as including a full proof as per 2b in the checklist. Is is OK to reference standards results from sources external to the paper, but they should be fully identified (i.e. Lemma [number] by [other author]). This makes the math difficult to follow or check. Overall, I think the paper could benefit from making the theoretical results less general, but more complete.\n\nOverall, due to the strength of the empirical results, despite these concerns, I score this paper as a weak accept.\n\nMinor points:\n- line 76: is -> are;\n- line 112: EMAQ -> EMaQ;\n- line 121: non-OOD -> in-distribution;\n- Eq 10: the \\propto sign is used in an unconventional way (it normally denotes scaling);\n- line 297: learned by ag (what does it mean?);\n- Bibliography: [31] and [32] are the same paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the checklist, the authors say that assessment of societal impact does not apply. It is true that the impact of this RL algorithm is similar to others measured against the same benchmarks, but I feel an assessment should have been included. One ethics-related aspect of batch RL that I think merits discussion would be its ability to learn policies interacting with humans without involving actual human subjects (which can be used both for good and bad purposes).", "review_text": "", "extra_scores": {"ethical_concerns": "I did not identify ethical issues with this submission.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "_ENjAdl9rKL", "reviewer": "Reviewer_T9dA", "summary": "The paper tackles the classical problem of batch RL. As a starting point it takes an actor-critic method with Q-function, plus EMAQ Q update and low-confidence bounds with ensembles of Q functions. Authors introduce two methods for regularizing the model:\n- Extra-overestimation penalty, which discourages, for a given state, actions not present in the dataset to have higher Q-value than the actions present there\n- Exploration penalty, encouraging high KL between behavioral and learned policy\n\nThe resulting method is tested on the standard benchmark: D4RL and the results compared against a number of contemporary batch RL methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is written very clearly, and authors provide convincing intuitions behind the methods they propose.\n2. The experimental section is impressive: authors compared their method against a number of baselines and provided reasonable ablations of their work.\n3. The results themselves are not groundbreaking, but good enough for an accepted paper.\n4. Authors provide theoretical guarantees (Theorem 2, 3) about their method which feel strong. I haven't been able to verify they are correct, but they seem theoretically plausible (rewards not much worse than behavioral policy, with the bound tightening the stronger the regularization and the bigger the dataset).\n5. In Theorem 1, authors claim that their Q-update operator T is a contraction and Q-function converges under \"commonly-assumed conditions\". It's not clear to me what these are; the readers would appreciate it if authors stated them explicitly, at least in the Appendix. Assuming that the authors mean the conditions mentioned in sec. 2.3.2 of Busoniu et al. (2010), they require that \"all the state-action pairs are (asymptotically) visited infinitely often\", which is by definition not the case for batch RL, making the Theorem 1 useless.\n6. Authors fail to cite a relevant work: Gulcehre et al. [\"Addressing Extrapolation Error in Deep Offline Reinforcement Learning\"](https://openreview.net/forum?id=OCRKCul3eKN), where a very similar regularization term:\n$$\\sum_{\\hat{a}\\in{a_1,\\ldots, a_N}}[Q(s, \\hat{a}) - Q(s, a) - v]_+^2$$\nis used in the same context and with the same motivation as the authors' extra-overestimation term. I think it would be appropriate to refer to this work, and ideally compare to it experimentally, especially given a similar structure to the paper (a regularizer for Q, another method for keeping the policy close to the behavior one). It seems one could mix&match between the parts proposed by these two papers and it may be interesting to see results of such an ablation.\n\nOverall, I find the paper sound, and suggest acceptance.\n\nSmaller comments:\n\n- In 182, $\\varepsilon$ is not defined. Is it meant to be $Q(s, \\hat{a}) - Q(s, a)$ ?\n- EMAQ is written inconsistently in 91 vs 112, 266\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IOnS-2m4bgQ", "reviewer": "Reviewer_JguC", "summary": "This paper introduces two batch-RL regularizers in order to solve the wild exploration and overestimation problem. The experiments on 32 tasks on D4RL demonstrate that the proposed method is better than the majority of batch-RL methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Originality}$: The method is new and it innovatively proposes three principles in regularizing Q values. However, the framework of CDC is still similar to that of BRAC, regularization on both value and policy. \n\n$\\textbf{Quality}$: The overall quality of the paper is satisfying, there are solid experimental results and statistical tests supporting the advantages of CDC. The theoretical foundation of CDC is also provided.\n\n$\\textbf{Clarity}$: I'm confusing about the relation between the two regularizers proposed in the paper, are they independent components? If one of the regularizers is replaced by the technique used in other offline RL algorithms, how will the overall performance change? I think the authors need to clarify this.\n\n$\\textbf{Significance}$: The performance of CDC is shown to be significantly better than other offline RL algorithms and the ablation study demonstrates the necessity of each component. The design principles of the Q value regularizer are meaningful but I'm unclear about the motivation of the policy regularizer. In BRAC dual format, the policy regularizer is also free of the estimation of behavior policy $\\pi_{b}$, I think one ablation study is needed to show the advantage of the policy regularizer in CDC", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed all of them.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have proposed an algorithm for (offline) batch RL that leverages a pair of regularisers, one constraining the policy and the other constraining that value, that lead to improved performance across a broad set of continuous RL benchmarks.

The reviewers agree that the results are convincing, well-supported by theoretical analysis and overall clearly presented. There is some discussion whether the individual regularisers themselves are especially novel w.r.t. prior work, but they are evidently complementary and yield convincing empirical performance. The authors have gone to noteworthy lengths to address all reviewer feedback, including running new ablations (yielding consistent results) and expanding their literature review to capture methods published around the submission deadline.

I fail to identify any reviewer concerns that have not been sufficiently addressed by the authors' detailed responses, and I agree with the reviewers' assessment that this is a high-quality submission well-suited for publication at NeurIPS.

We thank the authors for their submission and effort engaging with the review process.

Decision: Accept (Poster)