Offline Reinforcement Learning from Heteroskedastic Data Via Support Constraints
Authors: Anikait Singh
Aviral Kumar
Quan Vuong
Yevgen Chebotar
Sergey Levine
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline RL
support constraints
heteroskedastic data

Abstract: Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. 
We show that existing popular offline RL methods based on distribution constraints fail to learn from data with such non-uniform change in the variability of demonstrated behaviors, often due to the requirement to stay close to the behavior policy to the same extent across the state space. We demonstrate this failure mode both theoretically and experimentally. Ideally, the learned policy should be free to choose per-state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning and show that support constraints emerge when doing so. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method CQL (ReDS) is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation. 

Under review as a conference paper at ICLR 2023
OFFLINE REINFORCEMENT LEARNING FROM HETERO- SKEDASTIC DATA VIA SUPPORT CONSTRAINTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto highway, some drivers merge quickly, effi- ciently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, show that typical offline RL methods, which are based on dis- tribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution conservative Q-learning (CQL) obtain an approximate support constraint formulation. The reweighted distribu- tion is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
INTRODUCTION
Recent advances in offline RI. (Levine et al., 2020; Lange et al., 2012) hint at exciting possibili- ties in learning tigh-perfrmmin policies, entirely from offline datasets, without requiring danger- ous (Gareıa & Fernández, 2015) expensive (Kalashnikov et al., 2018) active interaction with the environment. Analogously to the importance of data diversity in supervised learning (Deng et al., 2009), the practical benefits of offline RL depend heavily the coverage of behavior in the offline datasets (Kumar et al., 2022). Intuitively, the dataset must illustrate the consequences of a diverse range of behaviors, that an offline RL method can determine what behaviors lead to high returns, ideally returns that are significantly higher than the best single behavior in the dataset.
We posit that combining many realistic sources of data can provide this kind of coverage, but doing so can lead to the variety of demonstrated behaviors varying in highly non-uniform ways across the state space, i.e. heteroskedastic datasets. For example, a dataset of humans driving cars might show very high variability in driving habits, with some drivers being timid and some more aggressive, but remain remarkably consistent in critical states (e.g.. human drivers are extremely unlikely to swerve in an empty road or drive off a bridge). A good offline RL algorithm should combine the best parts of each behavior in the dataset- e.g., in the above example, the algorithm should produce a policy that is as good as the best human in each situation, which would be better than any human driver overall. At the same time, the learned policy should not attempt to extrapolate to novel actions in subset of the state space where the distribution of demonstrated behaviors is narrow (e.g.. the algorithm should not attempt to drive off a bridge). How effectively can current offline RL methods per-state basis how closely to stick to the behavior policy?
selectively choose
Most existing methods (Kumar et al., 2019; 2020; Kostrikov et al., 2021b;a; Wu et al., 2019; Fu- jimoto et al., 201 8a; Jaques al., 2019) constraint the learned policy to stay close to the behavior policy, so-called "distributional constraints". Our first contribution consists of empirical and theo- retical evidence demonstrating that distributional constraints insufficient when the heteroskedas-

---

Under review as a conference paper at ICLR 2023
Start
LEGEND
A
Wammd palicy
heteroskedastic data
U UDD
Goal
policy
chocen
(6) Variability in behavior depends
(b) AWR stuck
CQL stuck
(d) CQL (ReDS) succeeds
Figure 1: Failure mode of distributional constraints. In this navigation task, offline RI. algorithm must find path from the start state the goal state indicated in (a). The offline dataset provided exhibits non- uniform coverage at different state, e-g.. in the state marked as "B" located in a wide room more uniform action distribution, whereas the states in the narrow hallways exhibit a more narrow action distribution. This is akin to how the behavior of human drivers varies in certain locations ("B"), but is very similar in other situations ("A"), To perform well, algorithm must stay close to the data in the hallways ("A"), but deviate significantly from the data in the rooms ("B"), where the data supports many different behaviors (most are not good). AWR and CQL become stuck because they stay too close to the bad behavior policy in the rooms, c.g. the left and right arrows near State B in Fig (b) and (c). Our method, CQL (RedS), learns to ignore the had behavior policy action in state B and prioritizes the good action, indicated by the downward arrow near State B in (d).
ticity of the demonstrated behaviors varies non-uniformly across states, because the strength of the constraint is aale-aggnstic, and may be overly conservative some states even when it is not con- servative enough at other states. We also devise mcasure of heteroskedasticity that enables us to determine if certain offline datasets would be challenging for distributional constraints.
Our second contribution is simple and mneoreticalll-motiiann against reweighted version of the behavior policy give rise to support constraints. That is, the return-maxinge maxumizaiiiiziion optimization process can freely choose state-by-ste policy should stay close to the behavior policy, so long as the learned policy remains within the data support. We show that it is particularly convenient to instantiate this insight on top of conservative Q-lcarning (CQL) (Kumar et al., 2020), a recent offline RL method. The new method, CQL (ReDS), only changes minimally the form of regularization, design decisions employed by CQL and inherits existing hyper-parameter constraints methods variety of tasks with more heteroskedastic distributions.
observation: distribution constraints
how much the learned
parameter values. CQL (ReDS) attains better performance than recent distribution
PRELIMINARIES ON DISTRIBUTIONAL CONSTRAINTS OFFLINE RL
The goal in offline RL is find the optimal policy in a Markov decision process (MDP) specified by the tuple M = (S, A, T, r, Ao, 7). A denote the state and action spaces. T's' a) and r(s, a) represent the dynamics and reward function. (s) denotes the initial state distribution. 7 E (0, denotes the discount factor. The goal is to learn a policy that maximizes the return, denoted by (T) = - 7'r(s, at)]. We must find the best possible policy while only having access to an offline dataset of transitions collected using a behavior policy TT. D {(s, a, r, s')).
Offline RL via distributional constraints. Most offline RL. algorithms regularize the learned policy π from querying the target Q-function on unseen actions (Fujimoto et al., 2018a; Kumar et al., 2019), either implicitly or explicitly. For our theoretical analysis, we will abstract the behavior of distributional constraint offline RL algorithms into a generic formulation following Kumar et al. consider the problem where must maximize the return of
(2020). As shown in Equation the learned policy π (in the empirical MDP) (()). while also penalizing the divergence from Ng: max Egga+
[[(T) aD(, aD(TTTT)(s)) (s) (generic distributional constraint)
(1)
where denotes a divergence between the learned policy π and the behavior policy Ty at state s. Conservative Q-learning. (Kumar et al., 2020) enforces the distributional constraint on the policy implicitly. To see why this is the case, consider the CQL objective, which consists of two terms:
min a (EanDaa [Q0 - -a (Qo(s,
ExnS (Qu(s, BT ((s)
(2)
8
R(0)
where BTTT Q a) is the Bellman backup operator applied to a delayed target Q-network, Q: BT Q a) r(s, a) + | (Q(') a'). The second term (in blue) is the standard TD error (Lillicrap et al., 2015; Fujimoto et al., 2018b; Tuomas Haarnoja & Levine, 2018). The first term R() (in red) attempts to prevent overestimation in the Q-values for out-of-distribution (OOD)
2

---

Under review as a conference paper at ICLR 2023
actions by minimizing the Q-values under distribution pl(ass,, which is automatically chosen to pick actions with high Q-values Qo(s. a). and counterbalances by maximizing the Q-values of the actions in the dataset. Kumar et al. (2020) show that Equation 2 gives rise to a pessimistic Q-function that modifies the optimal Q function by the ratios of densities, π rTaass) a (als) at given state-action pair (s, a). More formally, the Q-function obtained after one iteration of TTD-leamnin is given by:
T(als) TTa (a|s)
(ss, :
=
The Q function is unchanged only if the density of the learned policy и matches that of the behavior policy TT. Otherwise, for state-action pairs where π (a]s) (a)s))) Eq. 3 increases their Q values and encourages the policy и to assign more mass to the action. Vice versa, if iTaas) > TT (als), Eq. 3 encourages the policy to as- sign fewer mass to the action a.
Robatic
Locomation
G
In Eq. 3 a is a constant, and hence the value function is modified by the ratio of probabilities to the same extent at all possible state-action pairs. Similarly, the a in Eq. I weights the two terms in the objective equally. Such de- sign choices can be sub-optimal when the learnt policy should stay close to the behavior policy in some states, but not others. We next elaborate on this intuition both theoretically and experimentally.
Figure 2: Our method outperforms of-
fline RL methods based on distribution constraints on a diverse set of domains.
3 WHY DISTRIBUTION CONSTRAINTS FAIL WITH HETEROSKEDASTIC DATA
Realistic sources of offline data are often heteroskedastic. Heteroskedasticity is where for certain regions of the state space, the observed behaviors assign the most probability mass to a few actions, while in other regions, the behaviors are more uniform. This is because datasets are typically gen- erated by multiple policies, each with its own characteristics under a variety of different conditions. For instance, driving datasets are typically collected from multiple humans (Ettinger et al., 2021), and many robotic manipulation datasets are collected by multiple scripted policies (Mandlekar et al., 2021) or telcoperators (Ebert et al., 2021) that aim to solve variety of tasks.
3.1 A DIDACTIC EXAMPLE
To understand why distributional constraints insufficient when learning from heteroskedastic data, we present a didactic example. Motivated by the driving scenario in the introduction, we consider a maze navigation task shown in Fig. 1. The task is to navigate from the position labeled as "Start" to the position labeled as "Goal" using five actions at every possible state (L: <-, R: ->, U: f. D: 4, No: No Op), while making sure that the executed actions do not hit the walls of the grid.
Heteroskedastic dataset. In order to collect a heteroskedastic dataset, we consider a mixture of several behavior policies that attain a uniform occupancy over different states in the maze. However, the dataset action distributions differ significantly in different states. The induced action distribution heavily biased to move towards the goal in the narrow hallways (c.g., the behavior policy moves upwards at state A)). In contrast, the action distribution is quite diverse in the wider rooms. In these rooms, the behavior policy often selects actions that do not immediately move the agent towards the goal (e.g., the behavior policy at state B), because doing so does not generally hit the walls as the rooms are wider, and hence the agent is not penalized. On the other hand, the agent must take utmost precaution to not hit the walls in the narrow hallways. More details are in Appendix A.
Representative distribution constraints algorithms such as AWR (Peng et al., 2019; Nair et al., 2020b) and CQL. (Kumar et al., 2020) fail to perform the task, as shown in Figure 1. To ensure fair comparison, we tune each method to its best evaluation performance using online rollouts. The visualization in Figure 1 demonstrates that these two algorithms fail to learn reasonable policies because the learned policies match the random behavior of the dataset actions too closely in the wider rooms, and therefore are unable to make progress towards the Goal position. On the other hand, as shown in Appendix A, utilizing lower amount of conservatism is unable to prevent the policies from hitting the walls in the narrow hallways. In contrast, the method we propose in this paper, indicated as "CQL (ReDS)", effectively traverses the maze and attains a success rate of 80%.
3

---

Under review as a conference paper at ICLR 2023
3.2 CHARACTERIZING THE CHALLENGES WITH DISTRIBUTIONAL CONSTRAINTS
Having seen that distribution constraints can fail in certain scenarios, we now formally characterize when offline RL datasets is heteroskedastic, and why distribution constraints may be ineffective in such scenarios. Similar to how standard analyses utilize concentrability coefficient (Rashidinejad et al., 2021), which upper bounds the ratio of state-action visitation under a policy d' (s, a) and the dataset distribution Js, i.e,, max5,a d (s, a)/ p(s. a) S C", we introduce new metric called differ- ential concentrability, which measures dataset heteroskedasticity. All proofs are in Appendix B.
Definition 3.1 (Differential concentrabilityy). Given a divergence D over the action space, the dif- ferential concentrability of a given policy æ with respect to the behavioral policy T is given by:
[NV D(T,
27
D(T, (82) (u(82)
Caf
E S1.8yrd
((SS)
Eq. 4 measures the variation in the divergence between a given policy π T(ass) and the behavior policy Ta(ass) weighted inversely by the density of these states in the offline dataset (i.e., u(s) in the denominator). For simplicity, let us revisit the navigation example from Section 3.1 and first consider a scenario where (s(s) = Un. If there are states where π chooses actions that lie on the fringe of the data distribution (e.g., in the wider rooms), as well as states where the policy chooses actions at the mode of the data distribution (e.g., as in the narrow passages), then Cäi would be large for the policy т learned using distributional constraints. Crucially, Cān would be small even the learned policy π deviates significantly from the behavior policy Tg. such that ((T, (s) is large, but [TT, (s,( D(T, (sa)) is small, indicating the dataset is not heteroskedastiic
We will show in Section 5 that arbitrary policy checkpoints т generally attain a low value of Cā on offline datasets from non-heteroskedastic datasets such as those D4RL (Fu et al., 2020). We now use differential concentrability to bound both the improvement and deprovement of π w.r.t. Ta for distribution constraint algorithms using the framework of safe policy improvement (Laroche al., 2017; Kumar et al., 2020). We show that when Căn is large, then distribution constraints (Eq. 1) may not improve significantly over na even for the best value for the weight a:
Theorem 3.1 (Informal: Limited policy improvement via distributional constraints.). Wh..p. 2 1- for any prescribed level of safety Ç. the maximum possible policy improvement over choices of a, maxa (J(T)) [TT]]] SCT, where ç+ is given by:
h" (a) 71
CIV log
ISILAI
dif
:+:= max a
S.I.
Badda [D( (T) SC, (5)
(1
7)2
(D]
1-y
where h1" is a monotonically decreasing function af and h(0)= = 0(1)
Theorem 3.1 quantifies the fundamental tradeoff with distribution constraints: to satisfy given (-safety constraint in problems with larger C. we would need a larger a. Since the maximum policy improvement Ç+ is upper bounded by h-" (a),, the policy may not necessarily improve over the behavior policy if is large. On the flip side, if we choose to fix the value of a to be small in hopes to attain more improvement in problems where C is high, we would end up compromising on the safety guarantee as Ç needs to be large for a small a and large Cā Thus, in this case, the policy may not improve over the behavior policy reliably. Note that larger Căa need not imply large E.-.- [DTT, TT) (s)] because the latter does not involve u(ss). Chi also measures the dispersion of D(T, TD) (s), while the latter performs mean over states. In addition, Theorem 3.1 characterizes the maximum possible improvement with an oracle selection of a, though is not feasible in practice. Thus, when Cän is large, distribution constraint algorithms could either not safely improve over Tī or would attain only a limited, bounded improvement.
SUPPORT CONSTRAINTS As REWEIGHTED DISTRIBUTION CONSTRAINTS
Thus far, we have seen that distribution constraints can be ineffective with hetcroskedastic datasets. If we can impose the distribution constraints such that the constraint strength can be modulated per state, then in principle, we can break the upper bound on performance shown in Theorem 3.1.
Our key insight is that by reweighting the action distribution in the dataset before utilizing a dis- tribution constraint, we can obtain a method that enforces a per-state distribution constraint, which corresponds to an approximate support constraint. This will push down the values of actions that
4

---

Under review as a conference paper at ICLR 2023
CQL Icletribution const)
CQL (ReDS)
Q
Legend
(support const)
Quatie
Quetoon
parting
(3)
nettaverr pa
Qance
Quiaer
aroppes
bad astioons
B
Ied actions
(5)
(6)
(5)
(6) 2
Qake anripetto
(a) Support constraints is preferable to distribution constraints
(b) The strength of the constmantts is modulated per state
Figure 3: Comparison between support and distributional constraints: (a) Left: CQL pushes down the Q-
function under the policy (red) (1), while pushing up the function under the behavior policy тт. This means that the Q-values for bad actions can go up (2). (a) Right: contrast, ReDS re-weights the data distribution to
push down the values of bad actions (3), alleviating this shortcoming (4). (b) Left: the two modes (5) and (6) of the behavior policy Iy have similar Q values. The reweighted distribution (in red) pushes down the values of the two modes similar extent. The constraint requiring the learned policy stay distributionally close the behavior policy is strong in this state (not an issue given the two modes have similar values). (b) Right: The reweighted distribution pushes down less (7) the mode having clearly higher Q-value. The distributional constraint is thus only weakly enforced in this case, allowing the learned policy to pick out the better mode.
are outside the behavior policy support, but otherwise not impose a severe penalty for in-support ac- tions, thus enabling the policy to deviate from the behavior policy by different amounts at different states. Consider the generic distribution constraint shown in Eq. 1. Rather than having a distribution constraint between and Te, if we can impose a constraint between and reweighted version of TH where the reweighting is state-dependent, then we can obtain an approximate support constraint. Let the reweighted distribution be x= Intuitively, if (-Iss) is within the support of the (-)s))), then one can find a reweighting x= (-Is) such that D(T, x (s) = O, whereas if (-Is)) is not within the support of (-sss, then (TT, (T, x (s) still penalizes π when π chooses out-of-support actions, since no reweighting -² can put non-zero probability on out-of-support actions. This allows us to handle the failure mode from Section 3: at states with wide behavior policy, even with large a, is not anymore constrained to the behavior policy distribution, whereas at other "critical" states, where the behavior policy is narrow, a large enough a will constrain x (-|s) to stay close to (=(sss). We call this approach Reweighting Distribution constraints to Support (ReDS).
4.1 INSTANTIATING THE PRINCIPLE BEHIND REDS
How can we derive a reweighting of the behavior policy to instantiate an approximate support con- straint? One option is reweight T3 into x² and enforce the constraints between π and x" explicitly, by minimizing D(T,, r,, implicitly, by pushing up the Q-values under n' instead of TT. anal- ogously to CQL. Both choices are problematic, because the error in estimating the behavior policy can propagate, leading to poor downstream RL performance, et al., 2020b; Ghasemipour et al., 2021). For the case of CQL, this issue might be especially severe if we push up the Q-values under T" because then the errors in estimating the behavior policy might lead to severe over-estimation.
demonstrated in prior work (Nair
Instead, we devise an alternative formulation for ReDS that re-weights the learned policy such that applying a distributional constraint on this re-weighting imposes support constraint. In this formu- lation, we instead push down the Q-values under x= To still enable approximate support constraints between ı and nT. we define Th as mixture distribution of the learned policy and the reweighted behavior policy as shown below. We then demonstrate how "e can modify CQL to enable approxi- mate support constraint-s while re-using all existing algorithmic components.
+ (T)() (=(++s)))
(6)
where g(-) is monotonically decreasing function. As Fig. 3a depicts, the second term in Eq. 6 increases the probability of actions that are likely under the behavior policy, but are less likely under the learned policy (due to g being a decreasing function). We will show in Lemma 4.1 that utilizing π in CQL enforces an approximate support constraint on л. Utilizing a mixture of π and reweighted k g is crucial, allowing for two important properties. Firstly, the inclusion of n in the mixture ensures that the learned policy ι is penalized for deviating outside the support of the behavior policy, Tg. Second, the inclusion of the re-weighted behavior policy Ta g(a)s) allows to assign more density to good actions that are within the support of TT even if these actions have lower densities under mg. This ensures that n is constrained to lie within the support of na, but is
5

---

Under review as a conference paper at ICLR 2023
not constrained to the behavior policy distribution. Section 4.2 illustrates theoretically why pushing down the Q-values under Eq. 6 approximates a support constraint in terms of how modifies the resulting Q-values. We must pick g to devise complete offline algorithm. A practical choice that reuses existing components in CQL. is to subsume Ty and g into one distribution, denoted by (additional justification below). More concretely, x² now mixes and p (the equal weight of 1/2 is an arbitrary choice and any non-zero weight on p can be used):
x (-Is) 2 ( + pP((ls)
ReDS modifies the regularizer (0) in CQL (Fqq.) to utilize the mixture distribution p from Eq. 7, leading to the following modified regularizer:
[Qoss, a)] + 1
E R(0: p) = D.aDT
E
[Q( a)]
E 5,a-D
[Qo a))
(8)
2 2s~D.amp
We will show that Equation 8 imposes support constraint in Lemma 4.1. Subsuming Th and g into P means the optimal should satisfy p(-(s) = TT(-|s) 9 (T(-)))). Under the MaxEnt RL framework (Levine, 2018). policy probabilities are the exponentiated advantages: iTaas) exp(A(s. a)) exp(Q(s, a) ((s)). As such, we can utilize the advantage function (derived from Q-function) directly to train p. instead of using the policy. That is, p(-(s) = < g(exp(A(s, a))). As shown in prior work on reward wwwihe-wegghted and advantage-we weithageeweighed policy learn- ing (e.g., AWR (Peng et al., 2019)), we can solve for this distribution using a weighted maximum log-likelihood objective. We choose g(x) = 1/x, such that g(exp( (((s, (A(s,))) = exp(-A(s, a)) (which is a decreasing function). As is common with such weighted regression methods (Peng et al., 2019), we add temperature into the exponent, yielding the following objective for p.
P (+ss) = arg max Pe
Bg-DDanmp(le [log Pv (als) exp (-Aa(s. a)/T]]
(9)
The crucial difference between this objective and standard eeraattg--weihhter updates is the differ- ence of the sign. While algorithms such as AWR aim to find an action that attains high advantage while being close to the behavior policy, and hence, uses a positive advantage, we utilize the nega- tive advantage to mine for poor actions that are still quite likely under the behavior policy. The final objective for the Q-function combines the regularizer in Eq. 8 with a standard TD objective:
min JQ( R(00p)
B BanN'D [(oos..)) (B((ss,))BB((ss.) (ss
(10)
We name the new algorithm CQL (ReDS), illustrated in Algorithm Description using Python-syntax is in Appendix C. CQL (ReDS) first per- forms one gradient descent step on Ps using Eq. 9, then performs one gradi- ent step on the Q-function using the objective in Eq. 10, then performs one gradient descent step the policy no. and repeats.
Algorithm 1 CQL (ReDS) pseudo-code
1: Initialize Q-funttion. Qa. policy, me, and distribution Py:
2: for step (l.....N N} do
3: Update PV: with the objective Eq. 9:
Vi := w + no V. E [log pe (als) exp(- Aop(-Ao(s,,)) a)(,
"..a~D"
4: Train the Q-function using (e) in Eq. 10:
0. == 0,-1 n7 Vo)
5: Improve policy Te with SAC-style update:
di = dc-1 + 7/x V V...aaD (o(ss log T (a|s))
4.2 THEORETICAL ANALYSES OF CQL (REDS)
Next, analyze CQL (ReDS), showing how learning using the regularizer in (7) modifies the Q-values, providing justification behind our choice of the distribution in the previous section.
Lemma 4.1 (Per-state modification of Q-values.). Let g represents g (T TT-ss). J. In tabular setting, the Q-function obtained after one TD-learning iteration using the objective in Eq. 10 is:
Qo(s, == BBQsss) - Talss) + TT(aas) 2 (als)
(11)
(
where BT Q(s, a) is the Bellman backup operator applied to delayed target Q-network.
Eq. 39 illustrates why the modified regularizer in Eq. 8 leads "soft" support constraint whose strength modulated state Since g is monotonically decreasing function of æ, for state-action pairs where πt (als)) has high values, g is low and therefore the values for such state-action pairs

---

Under review as a conference paper at ICLR 2023
underestimated less. Vice versa, for state-action pairs where ((ass) attains low values, g is high to counter-acts the low r(als) values. Also, since (aaa))) appears in the denominator, for out-of-
support actions, where T = 0, π (als)) must also assign 0 probability to the actions for the Q values to be well defined. We can further formalize the objective for ReDS as follows:
Lemma 4.2 (CQL (ReDS) objective.). Assume that for all policies π E II, ((, a), ((aas)
> 0.
Then, CQL (ReDS) solves the following optimization problem:
max J(T) REIT
a
EE an
D(T, D(T, ap)(s) +
lg (T x (alss) {Tp((as)) > 03] (12)
2(1 111-7)
(T) corresponds to the empirical return of the learned policy, i.e., the return of the policy under the learned Q-function. The objective in Lemma 4.2 can be intuitively interpreted as follows. The first term, (T,, H (s), is a standard distributional constraint, also present in naive CQL, and it aims to penalize the learned policy π if it deviates too far away from T3. ReDS adds an additional second term that effectively encourages π to be "sharp" within the support of the behavior policy, enabling it to put its mass on actions that lead to a high J(r).. This second term allows us control the strength of the distributional constraint per state: at states where the support of the policy is narrow, i.e., the volume of actions such that T (als) 2 E is small, the penalty in Equation 12 reverts to a standard distributional constraint by penalizing divergence from the behavioral policy via D(,, Te TT)(s) as the second term is non-existent. At states where the policy Ty is broad, the second term counteracts the effect of the distributional constraint within the support of the behavior policy, by ι to concentrate its density on only good actions within the support of Th with the same multiplier a. Thus even when
we need to set a to be large in order to stay close to na (-|s) at certain states (e-g.. in the narrow hallways in the didactic example in Sec. 3.1), D(T, ( is not heavily constrained at other states.
EXPERIMENTAL EVALUATION
The goal of our experiments is to understand how CQL (ReDS) compares to distributional constraint methods when learning from heteroskedastic offline datasets. In order to perform our experiments, we construct new heteroskedastic datasets that pose challenges representative of what we would expect to see in real-world problems. We first introduce tasks and heteroskedastic datasets that evaluate on, and then present our results compared to prior state-of- the-art methods. We also evaluate ReDS on some of the standard D4RL (Fu et al., 2020) datasets which are not heteroskedastic in and find that the addition of ReDS, as expected, does not help, nor does it hurt on those tasks.
5.1
COMPARISON ON D4RL. BENCHMARK
Our motivation for studying heteroskedastic datasets is that heteroskedasticity likely exists in real- world domains such as driving, manipulation where datasets are collected by multiple policies that agree and disagree at different states. While standard benchmarks (D4RL (Fu et al., 2020) and RLUnplugged (Gulcehre et al., 2020)) include offline datasets generated by mixture policies, e.g- the "medium-expert" generated by two policies with different performance, these policies are them selves trained via RL methods such as SAC, that constrain the entropy of the action distribution at each state to be uniform. To measure their heteroskedasticity. we utilize a tractable approximation to Chr used in our theoretical analysis: the standard deviation in the value of D(T, (s) across states in the dataset, using fixed policy obtained by rumning CQL. We did not Cāi directly, as it requires estimating the state density of the data distribution, which is challenging in continu- ous spaces. Observe in Table 4a that the standard deviation is relatively low for the D4RL. antmaze datasets. This corroborates our intuition that D4RL. datasets are significantly less heteroskedastic.
5.2 COMPARISONS ON CONSTRUCTED HETEROSKEDASTIC DATASETS
Heteroskedastic datasets. To stress-test our method and prior distributional constraint approaches, we collect new datasets for the medium and large mazes used in the antmaze navigation tasks from D4RL: noisy sy datasets, where the behavior policy action variance differs in different regions of the maze, representative of user variability in navigation, and b ased datasets, where the behavior policy admits systematic bias towards certain behaviors in different regions of the maze, represen- tative of bias towards certain routes in navigation problems. Table 4a illustrates that these datasets significantly more heteroskedastic compared to the D4RL datasets.
Using these more heteroskedastic datasets, we compare CQL (ReDS) with CQL and IQL (Kostrikov et al., 2021a), which are recent popular methods, and two prior methods, BEAR (Kumar et al., 2019) and EDAC (An et al., 2021), that also enforce support constraints. For each algorithm, including ours, we utilize hyperparameters directly from the counterpart tasks in D4RL. Due to lack of an
7

---

Under review as a conference paper at ICLR 2023
Datasct
std
max
Task & Dataset
EDAC
BEAR
CQL
IQL
Ours
noisy (Ours)
18
253
medium-noisy
55
44
73
biased (Ours)
9
31
medium- biased
0
73
48
74
diverse (D4RL)
large-noisy
0
42
39
53
play (D4RL)
2
13
large-biased
0
50
41
45
(a) the new antmaae datasets (Ours) are signifi- cantly more heteroskedastic compared to the stan- dard DARL datasets. We measure heteroskedas- ticity using across states in the offline datasct.
(b) (R)(() outperforms prior offline RI. methods including state- oftte-oo-thh-e art methods (IQL), and prior support constraint methods(BBAR, EDAC) three four scenarios when learning from betersskedastic datasets in the antmaze task. The im- peovement over prior methods larger when learning from no ay datasets, which more heteroskedactic, shown Tablc 4a, compared to the blased datasets.
max of [15, (s)
Table 1:
Our method out-performs prior methods in the antmaze navigation task when learning from het- eroskedastic datasets, indicating the importance of support constraints relative to distributional constraints. Table 2: Performance comparison with recent offline RL algorithms on the D4RL benchmark.
Dataset
BC
10%BC
DT
AWAC
Onestep RI.
TD3-BC
COMBO
CQI.
IQI.
(Ours)
ballicheetal anedium replamy
36.6
36.6
40.5
38.1
44.6
55.1
45.5
52.3
hopper- medium- replay
18.1
75.9
82.7
37.2
97.5
60.9
89.5
95.0
94.7
101.5
walker2d-medine 2ddemddummmeplll halfchoctal-modiun--oeee
26.0
62.5
66.6
27.0
49.5
81.8
56.0
77.2
73.9
85.0
55.2
92.9
86.8
42.8
93.4
90.7
90.0
91.6
86.7
89.5
hopper-medium- medium--eeeee expert
52.5
110.9
107.6
55.8
103.3
98.0
111.1
105.4
91.5
110.0
walker2d medium expert
107.5
109.0
108.1
74.5
113.0
110.1
103.3
108.8
109.6
112.0
locomotion total
|| 295.9
491.8
488.4
277.8
486.1
523.5
500.6
550.3
effective method for offline policy selection (see Fu et al. (2021)); we utilize oracle checkpoint selection for every method. We compute the mean and standard deviation of the performance across 3 seeds. Table 4b illustrates that the largest gap between CQL (ReDS) and prior methods appears on the noisy datasets, which are particularly more heteroskedastic as measured in Table 4a.
We also compare CQL (ReDS) with recent offline RL algorithms on D4RL, including DT (Chen et al., 2021 1), AWAC (Nair et al., 2020a), onestep RL (Brandfonbrener al., 2021), TD3+BC (Fuji- moto & Gu, 2021b) and COMBO (Yu et al., 2021). Table 2 shows that CQL (ReDS) obtains similar performance as existing distributional constraint methods and outperform BC-based baselines. This is expected given that the D4RI. datasets exhibit significantly smaller degree of heteroscedasticityy as previously explained. Also, a large fraction of the datasets is trajectories with high returns. BC using the top 10% trajectories with the highest episode returns already has strong performance (10%BC).
The previous results compares CQL (ReDS) to baselines tasks where the MDP states are low- dimensional vectors. This section compares CQL (ReDS) to baselines in vision- tasks.
Visual robotic manipulation. We consider two manipulation tasks. In the "Pick & Place" task, the algorithm controls a WidowX robot to grasp an object and place it into tray located at test location, directly from raw 128 x 128 x28x3 images and sparse 0/1 reward signal. The dataset consists of behavior from suboptimal grasping and placing policies, and the positions of the tray in the offline dataset very rarely match the target test location. The placing policies exhibit significant variability, implying that these datasets are heteroskedastic under our definition. We also consider "Bin Sort" task, where a WidowX robot is controlled to sort two objects into two separate bins. Here heteroskedacity is introduced by when sorting the objects into the desirable bins. Similar to the Pick & Place domain, the placing policy exhibits significant variability, and shows an the object being placed in the incorrect bin (e.g.. recyclable trash thrown into the non-recyclable bin). However, the grasping policy is narrow, grasping the object with low variability. More details in Appendix D.
Table illustrates the results on these tasks. We utilize oracle policy selection analogous to the antmaze experiments from Table 4b. Table 3 shows that CQL (ReDS) outperforms CQL attaining success rate of about 15.1% for the visual pick and place task, whereas CQL only attains 6.5% suc-
Table 3: CQL (RcDS) CQL the manipulation tasks. CQL (ReDS) outperforms CQL significantly when learning from more heteroskedastic datasets, as measured by our estimates of Ca the standard-deviation and the maximum of D(T,, (s) across states.
Task
CQL
CQL (ReDS)
std D(x, (x)s))))s)
max D(r, (s)
Pick & Place
6.5 1 0.4
15.1 1 0.4
48.7
307.4
Bin Sort (Easy)
31.2 0.3
31.4 : 0.3
7.9
81.6
Bin Sort (Hard)
+: 0.2
23.1 +
59.6
988.3
8

---

Under review as a conference paper at ICLR 2023
cess. While these performance numbers might appear low in an absolute sense, note that both CQL and ReDS do improve over the behavior policy, which only attains a success rate of 4%. Thus offline RL does work on this task, and utilizing ReDS in conjunction with the standard distributional con- straint in CQI. does result in a boost in performance with this heteroskedastic dataset. For the "Bin Sorting", our method outperforms CQL by 3.5x when learning from more heteroskedastic datasets. This indicates that the effectiveness of our method in settings with higher hetcroskcdasticity.
Atari games. We collect data on 10 Atari games from multiple policies that behave differently at certain states, while having similar actions otherwise. We consider a case of two such policies, and a harder scenario of five. We evaluate the performance of CQL (ReDS) on the Atari games using the evaluation metrics from prior works (Agarwal et al., 2020; Kumar et al., 2021), denoted IQM.
Figure shows that in both testing scenarios: with the mixture of two policies (top figure) and the mixture of five policies (bottom figure), CQL (ReDS) outperforms CQL in aggregate.
IQM
Policies Policess
0.75
0.90
1.05
1.20
To summarize, our results indicate that incorporating CQL (ReDS) outperform common distributional constraints algo- rithms when learning from heteroskedastic datasets.
0.45
0.60
0.75
NormalizedScorm Score
RELATED WORK
(BuDS)
Offline Q-learning methods utilize mechanisms to prevent backing up unseen actions (Levine et al., 2020), by applying an explicit behavior constraint that forces the learned policy to be "close" to the behavior policy (Jaques et al., 2019; Wu et al., 2019; Peng et al., 2019; Siegel et al., 2020; Wu et al., 2019; Ku- mar al., 2019; Kostrikov et al., 202 Wang et al., 2020;
Performance of CQL (ReDS) vs CQL. on the IQM normal- ized score and the mean normalized score over ten Atari games, for the case two (top) and five (bottom) policies. Fujimoto & Gu, 2021a), or by learning a conservative value function (Kumar et al., 2020; Xie al., 2021; Nachum et al., 2019; Yu et al., 2021; 2020; Rezaeifar et al., 2021; Jin al., 2020; Wu et al., 2019). Most of these offline RL methods utilize distributional constraint, explicit (e.g.. TD3+BC (Fujimoto & Gu, 2021a)) or implicit (e.g.. CQL (Kumar et al., 2020)). and our empirical analysis of representative algorithms from either family indicates that these methods struggle with heteroskedastic data, especially those methods that use an explicit constraint.
Figure 4:
Some prior works have also made a case for utilizing support constraints instead of distribution constraints, often via didactic examples (Kumar et al., 2019; Kumar, 2019; Levine et al., 2020), and devised algorithms that impose support constraints in theory, by utilizing the maximum mean discrepancy metric (Kumar et al., 2019) or an asymmetric f-divergences (Wu et al., 2019) for the policy constraint (Wu et al., 2019). Empirical results on D4RL (Fu et al., 2020) and the analysis by Wu et al. (2019) suggest that support constraints are not needed, as strong distribution constraint algorithms often have strong performance. As we discussed in Sections 3.2 (Theorem 3.1 indicates that this distributional constraints may not fail when Cā is small, provided these algorithms are well-nuned.) and 4, these benchmark datasets policies that are equally wide at all states and centered on good actions (e.g., Antmaze domains in (Fu et al., 2020), control suite tasks in Gulcehre et al. (2020)) and hence, do not need to modulate the distributional constraint strength. To benchmark with heteroskedastic data, we developed some novel tasks which may be of independent interest beyond this work.
not heteroskedastic, as they collected from
DISCUSSION
In this work, we studied the behavior of distributional constraint offline RL algorithms when learning from heteroskedastic datasets, a property we are likely encounter in the real world. Naive distribu- tional constraint algorithms can be highly ineffective in such settings both in theory and practice, as they fail to modulate the constraint strength ptr-state. We propose ReDS, a method to convert distributional constraints into support-based constraints via reweighting, and validate it in CQL. A limitation of ReDS is that it requires estimating the distribution Ps to enforce a support constraint, which brings about its some additional compute overhead. Devising approaches for enforcing sup- port constraints that do not require extra machinery is an interesting direction for future work. Un- derstanding if support constraints are less sensitive hyperparameters or are more amenable to cross-validation procedures is also interesting. Extending ReDS to other Q-learning based, and even model-based offline RL algorithms is also an interesting direction for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning-Voie -ollue 70, pp. 22-31. JMLR. org, 2017.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning (ICML), 2020.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-Based Offline Rein- forcement Learning with Diversified Q-Ensemble. arXiv e-prints, art. 1 October 2021.
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL with- out off- policy evaluation. CoRR, abs/2106.08909, 2021. URL https: /arxivvvvvv.//////a///aaa//aaatv org/ abs/ 2106. 08909.
Pablo Samuel Castro, Subhodeep Moïtra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
// //arxiv. baizg/abs/1122 06110.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.0 0134,, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A acage-scale hi- erarchical image database. In 2009 IFFE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13320 2021.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan. Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp.
2021.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Q- learning algorithms. arXiv preprint arXiv: 10250, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072111 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In International Confer- ence on Learning Representations, 2021. URL https: /ttp:////ppeeeeeeeeeeem net wttpr//wwwweee/tuummee forum?id= kWSeGReHvF8.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint urXiv:2106.0668 2021a.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. CoRR, abs/2106.06860, 2021b. URL https: ptttpp://axxxv.ooot http://aaaw....oooa////a/....//. 06860.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. arXiv preprint 1rXiv: 2018a.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. In International Conference on Machine Learning (ICML), pp. 1587-1596, 8b.
10

---

Under review as a conference paper at ICLR 2023
Javier Garcia and Fernando Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-188,, 2015.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning. pp. 3682-3691. PMLR, 2021.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. un- plugged: Benchmarks for offline reinforcement learning. 2020.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint urXiv:1907.004550 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. pessimism provably efficient for offline arXiv preprint arXiv:20/2. 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651- 673, 2018.
lyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2 10.06169, 2021a.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021b.
Aviral Kumar. Data-driven deep reinforcement learning.
(tlgg2211//22/ /0//2//05//er.../// 2019. BAIR Blog.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Sys- tems, pp. 11761-1177,, 2019.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. arXiv e-prints, art. arXiv:1906.00949, June 2019.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. 2019. URL http:: /tttt://arxiv..oo abs/ 190.. 00949.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- naatetrprramtnnizzationn inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep- resentations, 2021. URL https: /ttp//pppeeeeeeeeeeeeeeeeeeeeeee
wwwetteeouurneeee wow.mmiino0bnnnnoo h fXU.
Aviral Kumar, Joey Hong. Anikait Singh, and Sergey Levine. Should run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022. URL htw.t://///// wttptttsss//ppeeeeeeeeeee http://pppeeeeeeeeeeeeeeeeeeeeee net forr/foruuae forum?
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe policy improvement with baseline bootstrapping. arXiv preprint arXiv:1712.06924 2017.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint arXiv: 1805. 2018
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto-
rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643. 2020.
11

---

Under review as a conference paper at ICLR 2023
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint S.KKv::10090097711 2015.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Fei-Fei Li, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin. human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning. 2021. URL https: httts::/openreviiee hpwp//peeeeeeeeeeeee
What matters in learning from offline
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: 191.000744 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offlin datasets. CoRR, dbs/2006.09359) 2020a. URL https: http://aaxvvvvvo abs/2006. 09359.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020b.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine Advintage-weightted regression: Simple and scalable ooflpo reinforcement learning. arXiv preprint arXiv:1910.001177, 2019.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learningg A tale of pessimism. arXiv preprint 27X1v:2103.12021, 2021.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. arXiv preprint arXiv:2106.06431, 2021.
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Be- havioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
Avi Singh, Albert Yu, Jonathan Yang. Jesse Zhang, Aviral Kumar, and Sergey Levine Cog: Connecting new skills to past experience with offline reinforeement learning. arXiv preprint arXiv:2010.14500, 2020.
Kristian Hartikainen George Tucker Sehoon Ha Jie Tan Vikash Kumar Henry Zhu Abhishek Gupta Pieter Abbeel Tuomas Haarnoja, Aurick Zhou and Sergey Levine. Soft actor-critic algorithms and applications. Technical report, 2018
Ziyu Wang, Alexander Novikov, Konrad Žołna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006.15:1 15134, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1r 1336, 2019.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-coosister pessimism for offline reinforcement learning. Advances neural information processing systems, 34, 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201.13422 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
12

---

Under review as a conference paper at ICLR 2023
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-base policy optimization. arXiv preprint arX0022102.08363, 2021.
13

---

Under review as a conference paper at ICLR 2023
Appendices
DETAILS OF THE DIDACTIC NAVIGATION EXAMPLE FROM SECTION 3.1 A
AWR Visitations
Temperature
Temperature
Temperature 10
Figure 5: Results of running AWR on the gridworld maze for a variety of temperature values annotated with state- nace-occcy values. Observe that AWR is unable reach the goal across wide range of temperatures for the AWR hyperparameter. Even though for some of these hyperparame- ters, such of its state visitation in the narrow hallways this case (e.g., the final narrow hallway it is able to reach to) indicating that it gets stuck. Increasing the temperature since now it does not even reach this hallway with as high of an occupancy.
0.1, it is able to traverse quickly into the third region, it does spend larger fraction
1.0 does not solve it either,
COL Visitations
Alpha
Alpha
Alpha 10
Figure 6: Results of running CQL on the gridworld maze for variety of temperature values. Observe that CQL is unable reach the goal and gets stuck for all the values studied.
In this section, we present some details regarding the navigation example we considered in Sec- tion 3.1. To create this example, we modified the gridworld code from Fu et al. (2019) (code taken from: https:/ / thub. com/ just in fu/ d agnos i ng glea rn ig) to create the cor- gllearnii
responding gridworld maze. We utilize 24 16 gridworld as shown in the figures below, larger than the x 8 or 16 x 16 gridworlds studied in this repository in the past. We utilize a smooth rep- resentation of the observation space. This is constructed by first sampling random Gaussian feature vectors from 50 for each grid cell (each grid cell a state). Smoothing of these features vectors is in Fu et al. (2020). This observa-
then done locally, following the protocol for frrid-**smoothor tion type presents a challenge for Q-learning algorithms that may often generalize incorrectly. This is because aliasing occurs with the predictions of nearby states that exhibit very different dynamics
14

---

Under review as a conference paper at ICLR 2023
but not different observation vectors. The agents gets sparse binary 0/+1 reward: +1 is attained only when the agent reaches the goal (marked in black), starting from the start location (marked in green).
The behavior policy in each part of the maze is based on a mixture of different policies. In the wider rooms of the maze, of the policies is uniform policy, that uniformly chooses every action at state. The second policy is a biased policy, that drives the agent away from the goal. In the narrow hallways, the behavior policy deterministically drives the agent towards the goal. For wide rooms, in contrast, a bias exists for actions taken in the direction away from the goal. This bias was set to be 0.8. This means that for rooms where you need to exit the wide passage by going down, the action of going up was selected 80% of the time by the behavior policy and each of the other action was randomly sampled with 20% probability. The opposite decision was taken for rooms where the goal was towards the top of the wide passage where 80% of the time the down action was selected.
Result visualizations. We now present some visualizations of the policies learned by various meth- ods: AWR, CQL and CQL (ReDS). Since the reward values are binary and sparse, return curves for AWR and CQL are not as informative, since they attain a return in any episode, even they make some progress. Therefore, we present the results in the form of state-visitation density plots under rollouts from the learned policy. Observe in Figures 5 and 6 that neither AWR or CQL are able to actually successfully traverse the maze, and get stuck in it. Note that T and a control the strength of the distributional constraint in the methods AWR and CQL respsectively. Varying the temperature hyperparameter for AWR and for CQL does modify the density in the narrow hallways, but utilizing too small of a parameter leads the agent to more frequent crashes in the narrow hallways. This makes the agent spend a higher fraction of its visitation in one such narrow region, whereas higher 7 or a does not even reach the third narrow hallway with a high-enough visitation.
REDS State Visitations
Figure Results running CQL. (ReDS) on the gridworld. Note that CQL (ReDS) does actually succeed at solving the task.
On the other hand, the method we propose in this paper, ReDS, when combined with CQL is able to successfully traverse this maze, as shown in Figure 7.
Analysis. These experiments are consistent with our expectations. When distributional constraints are faced with highly heteroskedastic action distributions at subsequent states, they failed to perform good actions at these consecutive states. A strong distributional constraint leads to the agent being too close to the behavior policy, whereas a weaker constraint fails to identify good actions at states where the behavior policy is narrow.
A.1 WHY AND WHEN Do DISTRIBUTIONAL CONSTRAINTS FAIL IN SCENARIOS WITH HETEROSKEDASTIC DATA?
In this section, we shall discuss why distributional constraints are especially worse than support constraints in scenarios with heteroskedastic data. Let's first consider a simple single-state bandit problem. In this simple one-state problem, must find single optimal action by using offline
15

---

Under review as a conference paper at ICLR 2023
data. At first, it might appear that when faced with wide behavior policy, distributional constraints would clearly fail since they would not deviate far away from the behavior policy. However, note that can, in principle, control the strength of the distributional constraint quite effectively. To see this concretely, consider applying CQL. (Equation 2) to single-state bandit problem. The Q-values on actions not observed in the training dataset will be clearly pushed down to -oo. In addition, choosing non-zero will allow us to precisely control how close the action taken by the learned policy is the best in-support action vs how close the learned policy is to the behavior policy. Therefore, for an optimally chosen value of a, distributional constraint methods such as CQL should already work well in the single-state setting.
by carefully choosing the strength of the distributional constraint,
However, the precise challenge with distributional constraints arises in the multi-state setting, where finding single value of a that can work well at all states might be can be exceedingly challenging in practice. This is the setting with heteroskedastic data that our paper considers, including in our didactic example above. In such a setting, a strong distributional constraint is able to take desirable action when the behavior policy is narrow (for example in the narrow room). However, the action distribution has wider support in the wider room and the strong distributional constraint would not take a desirable action in this setting due to the high entropy of the behavioral distribution in this setting. In contrast, with weaker distributional constraint, bad dif-iistrinttion action may be taken where the behavior policy is narrow which can lead to instability in the policy. Note that these challenges are not, however. present in the single-state scenario. Our method CQL (ReDS) can tackle the problem in this setting by modulating the strength of the constraint per state. This allows the agent to stay close the behavior distribution in the narrow room as well as learning to output the most desirable in-support action in the wider room.
A.2 HETEROSKEDASTIC DATA VS EXPLORATORY DATA
There is a distinction between exploratory data (Yarats et al., 2022) and heteroskedasticity. In the context of offline RI, exploratory data typically refers to being able to cover all possible state- action pairs, as uniformly as possible. On the other hand, heteroskedasticity is used to refer to disproportionate coverage: when the action distribution in the offline data is more uniform at certain states, but very narrow at others. Thus, a heteroskedastic dataset does not have high coverage in the sense of observing all possible state- pairs.
Mathematically, while the notion of coverage is typically quantified by the concentrability coeffi- cient which denotes the worst case density ratio between the distribution of the training dataset and the distribution of the learned policy, as shown below:
d= d'((s. a)
CTT max
8,8
ass
where y(ss a) denotes the state-action distribution induced by the dataset, while d= (s, a) denotes the state-action visitation distribution of policy т. max, C is generally expected to be smaller when
ulss, a) is more uniform. In contrast, we quantify the notion of heteroskedasticity by the variation
in density ratios (in this case, action density ratios, =(als) across the state space in Equation 4 (a(s))))
(reproduced below for convenience).
27
(())) ((Ss)
D(7, (s²) (122)
CHT
Even if the dataset has high coverage (i.e., has a low concentrability coefficient, C*), it can attain a higher Ca because of higher variation in the density ratios across state spaces.
Out of all methods considered in Yarats et al. (2022), we note that the methods classified as "Data" (APT, ProtoRL) attempt to maximize state coverage, the methods classified as "Knowledge" (ICM, Disagreement, RND) attempt to find surprising state-action pairs and visit them, and the methods classified as "Competence" (SMM, DIAYN, APS) that maximize the mutual information between the state distribution attained by the learned policy and the learned skills also implicitly optimize for state coverage. In effect, all of these methods aim to visit all acte-action pairs (or all states), which is akin to the standard definition of concentrability, whereas heteroskedastic data corresponds distinctly to disproportionate action coverage different states.
16

---

Under review as a conference paper at ICLR 2023
B PROOFS
In this appendix, will provide proofs for the various theoretical results in the main paper: The- orem 3.1. We will first discuss some preliminaries and notation, then present the proof for Theo- rem 3.1, and finally the remaining results.
B.1 NOTATION AND PRELIMINARIES
Let (a|s) denote the behavior policy. Note that the dataset, D, is generated from the marginal state-action distribution of T.. i... D ~ d (a)) Define d= as the state marginal distribution introduced by и under the empirical MDP defined by the transitions in the dataset. Let DeoL (p. q) denote the following distance between two distributions p(x) and ((x) with equal support X::
(p, = Σx(x)
((x)
p(x) ((x)
-1).
We drop the subsrcipt "CQL" from DOOL for clarity. (Kumar et al., 2020) showed that when opti- mizing the generic distributional constraint algorithm shown in Equation 1, the resulting policy x² attains a high probability safe-policy improvement guarantee, i.e., (+*) 2 J(T - G. where ( is:
c=o ( (1-
[V
D(T*, (s) + [D(s)]
F g-
8 += [ "M""",,"))) (s)( (13)
We can further express [D(s] [)], The first term in Equation 13 corresponds to the decrease in performance due to sampling error and this term is high when the learned policy x² visits low density states under the dataset distribution (i.e., (ss) is small) and when the divergence from the behavior policy T is higher under these states. We will use this safe policy improvement guarantee in our proofs.
B.2 FORMAL RESTATEMENT THEOREM 3.1
Theorem B.1 (Formal version of Theorem 3.1), For any prescribed level of safety iee. the learned policy satisfies J(x) (TT)) 2 -G. the maximun possible policy improvement over choices of oz:
max [J(ma) ( [[TT]]] sCt
where Ç+ is given by:
Ci log
S|AA
dif
E Baairo [D(Ta; Ts)ss))
5t := max ht (a) (1 (1-
s.l.
(1 7)²
[D]
1-7
==((a]
(14)
where is a monotonically decreasing function of a. and h(0) = const.
Intuition behind the theorem: We will show in the proof of this theorem that g(a) a monoton- ically increasing function of and g(0) 0. This means that if the value of Of is large for all policies ग, the value of g Cimee - (where e' is the constant and subsumes the term containing 5) is also large. Further, note that h² (a) is decreasing in This means that the larger the value of a, the smaller the value of h² (ax), and hence, smaller the value of ç+. This means that Cji increases, needs take larger values satisfy the safety constraint, and this reduces the value maximal improvement, +
Conversely, to attain larger improvement 5 if we choose smaller for problem where C large for all policies, then, we must give up on the safety guarantee, and would be large. This means that the learned policy n. might substantially degrade beyond the behavior policy.
17

---

Under review as a conference paper at ICLR 2023
B.3 PROOF OF THEOREM 3.1
In order to prove this result, we we will utilize basic algebraic inequality mentioned below in Lemma B.1.
Lemma B.1 (Algebraic variation), Given any N positive real numbers, 21, 22, :.. NN.
N
Vzi
Ci Z I(==-vVz))
(15)
(N 1) K2j
1=1
Proof. For every 24, define yı Jz. Then, the difference between the two sides in the equation above is given by:
Σ 2
1
( + v3 2y:yi) Σ 2 N 1
1/8
1
2yc
(16)
(N
1) icj
N
N
0,
(17)
N
K
where the first step follows by rearranging y/ and yj. and the final step follows by noting that y 2 for all The other inequality follows trivially by noting that vi, are positive, and applying the standard formula for sum of squares.
We will also require a Lemma that allows us to upper bound the performance difference ((x) (Tx) terms of the metric DeoL (7, (T) that appears in the safe policy improvement guarantee in Equation 13.
Lemma B.2 (Tight upper bound on policy improvement.). Assume that the reward function r(s, a) of the MDP is bounded such that Vs, r(s, a) E [- Rmmax Rmax]. For any two policies and ng we have the following:
(1-
1
) Ea-dr (((T(1ss,,==((s))) Rmax.
SO
(18)
Proof. The core of the proof of this lemma relies on the fact that for any given function v(x) over some space x, we can upper bound, A, (p, q) == Ex [u(x)] - [xxg[[(x) in terms D(p, q). To show this, we expand this expression:
(x( = Σ (p(x) ((x))= u(x)
(19)
[p(x)
(x))
(xx) -1) 1) v(x)
(20)
Σ
((x) (((x)
(((x) -1) u(x)+
Σ
((x)
[((x)
1 u(x)
((x)
x:P0o <l,((x)20
Σ
q(x) (p(x) -1 v(x)
Σ
g(x)
[p(x)
((()) -1) v(x).
((x)
2(x)
<(
Each of the four terms above can be bounded independently follows: for the first two terms, multiply by pix) the third term is clearly negative, and the final term is upper bounded by x(x)'
18

---

Under review as a conference paper at ICLR 2023
multiplying by 1 1+ p(x).
q(x)
p(x)
p(x)
p(x)
Σ
((x)
u(x) 5
Σ
((x)
1 u(x)
(21)
[(x)
g(x)
q(x)
x: 2|w) ((ल) (2)
x: p(x) q(x) 2x))
[((x)
p(x)
p(x)
Σ
(x)
m(x) S
Σ
q(x)
-1) f(x)
(22)
((x)
((x)
((x)
X: p(Xم
v(K] <122
v(x) 000)
<1,((x20
(P(X)
Σ
(xx)
1 v(x) 0
(23)
((x)
zلاع [() (1,0(
(p(x)
[p(x) g(x) -1) (1 (1+ p(x)
Σ
((x)
q(xx -1) v(x)
Σ
q(x)
v(x).
g(xx)
2(x) (l
m6a) 7(87
(1))
(24)
Finally, for each of these terms, we can now upper bound v(() by its maximum absolute value, |x(x)] S vo, and combine the terms to get the following bound on (p(p, q):
((x) 1 1) + +0+ M1 ((x)
p²
(p( S vo Σ p(x)
Σ
((x))
(25)
q(x)
x==(x>>0
<L,())<
p² (x)
v/0
(26)
((x)
where Equation 26 follows from using the fact that for the case when p(x)/q(x) > 1 but v(x) S 0, p(x)
p(x)
1) > 0, and hence it upper bounds the RHS of Equation 23. For the last case, where
((x)
x:
p(x)
v() O, we note that Ex. Ptx) <1,x(x)<o ((x) E m0x) <1, p(x), and hence vtx)
g(x)
90x)
the upper bound on this term in Equation 26 follows. To complete the argument note that DOoL exactly takes the form obtained in the final equation, and hence:
(p. S vo D)p.g).
We can now use this result to bound the return differences, by using standard results for bounding the performance difference between policies (Achiam et al., 2017: Schulman et al., 2015) in terms of (1-a)² x (TT, π&). At the core of these results is a bound on Ea [f(s]][(s [a]]-a-ra((s))) (f a)), and hence the result proven above directly applies. This proves the required result.
B.4 ADDITIONAL TECHNICAL LEMMAS
In this section, we will provide proofs of two technical lemmas, that allow us to conclude the proof of Theorem 3.1. For these lemmas, we will consider generie optimization problem,
max f(x) + a g(x),
(27)
where g(x) > 0 for any x, and a can only take non-negative values.
Lemma B.3 (Value of g(x) as a function of a), Let be the value x that maximizes Equation 27 for a given fixed value of a. Then the following statements hold:
1. For any > 8 0. g(x)) 2 g g(x3).
2. For any a Iv
3 N 0, ag (xx) ) 2 3g(x)))
Proof. For any given o, x satisfies the following inequality:
4X' f(x²) ag(x)) 2 f(x') + ag(x').
(28)
19

---

Under review as a conference paper at ICLR 2023
Using the above relation, can write down two inequalities relating xa and x3:
f(xa) + ag(x") 2 f(x3) + ag(x3))
(29)
f(x3) + Bg(33) 2 f(xx)) + Bg(x"))
(30)
Now, adding the two inequalities above, and cancelling the terms fxx)) + f(x3) from both sides, we obtain:
(a 3) g(xx) 2 (a ((a-B))(xp) ((x3)
(31)
Since 3 is non-negative, it either equal to O, in which case g(x²) = g(x3) or it is positive, in which case, g(x;,) 2 g(x]). Combining these two cases, we get the desired result in (I). For proving the second part (2), note that we can write the difference of the two sides as:
ag(x),) 3g(x3) agg(x)) - ag(x3) (g(x))) Bg(xx))
(32)
= a [g(xt) 9(x3)] + (a - 3))((}})
(33)
2 20+0=0 0,
(34)
where the last inequality follows from the fact that g(x) and B.
We will now provide a proof for Theorem 3.1.
Theorem B.2 (Theorem 3.1 restated). W.h.p. 2 1 for any prescribed level of safety Ç. the maximum possible policy improvement over choices of a, ( - J(Tz) S S5 where 5 is given by:
C
C1
a
: max h" (a) 71-
S.t.
E 7
Andt [D(Ta, TT)(s) S.
(1 -1-98 7)2
[D]
(35)
where h² is a monotonically functionn of a, and h(0) = 0(1)
Proof of Theorem B.2. To prove this theorem, we will apply Lemma B. on T;
D(5=(-1s:) (1(3)) and combine it with the safe policy improvement guarantee for behavior regu- ((a)) larization methods that admit updates of the form shown in Equation 1.
First, we note by applying Lemma B. I in its expectation form that:
(Ba-an
D(a. (5)
E vod sa
(is, p(#1)
Dia. [a]l [(22)
where the term on the RHS of the above equation corresponds C
Now we can plug this into the safe-policy improvement guarantce to obtain the resulting result follows. Note that the first term in the bound in Equation 13 can be lower bounded using the differential concentrability as discussed above, and therefore, we get the following lower bound on
(
1
C.To oE
5
0
(1 (1-
SNiTT [D(Ta, [s)(s)]
(36)
|D]
which is exactly the same the expression for the constraint in Theorem 3.1.
Next we provide an upper bound on the maximal improvement that can be possible, in terms of D(T.; Rg). For this, we will utilize Lemma B.2, and we can directly upper bound J(m.) J(ma)
as follows:
J(ma) - ((Tp) S (1 (1- E Butto (T (s)(s)] Rmaxx
(37)
Finally, we express this upper bound terms of a.
Now, note that the RHS in Equation 37 depends E E [D( T (s)], which is directly the
term that a generic distributional constraint algorithm minimizes (Equation 1). We wish to under- stand how this term evolves function of
20

---

Under review as a conference paper at ICLR 2023
Now will invoke Lemma B.3 to understand the behavior of the term above when solving the optimization problem in Equation 1. To do so, consider any a, a' and note that f(x) J(z) and g(x) = -E an snnd [T, (s)[ Now applying Lemma B.3, Part (I), we note that:
E B- (8) (8) [D(T (s)] & or (Tar, (s)(
(38)
for a' I This means that can upper bound this quantity by function ht (a) that is monoton- ically decreasing in a.
Therefore, the maximal improvement is upper bounded by: ha)a) ( (1-1))) which completes the proof of this theorem.
B.5 PROOF OF LEMMA 4.1
Lemma B.4 (Lemma 4.1 restated) Per-state modification of Q-values.). Let 9 represents (T (T-Mi (-1s)). The Q-function obtained after one TD-learning iteration using the objective in Eq. 10 is:
T(als) + TT(aalsg 2 (als)
(39)
(
where B ((s, a) the Bellman backup operator applied delayed target Q-network.
Recall from Section 2 that the objective of CQL consists of two terms
1 min a (BEAD (Qa(s, a)] ER (Q,(s a))) + Eg a'
(Qo(s.a) -BB(s,a)BBrQ(ss Q(ss,a))
(40)
2
R(0)
where ET (s, a) is the Bellman backup operator applied to delayed target Q-network. In tabular setting, the Q-function obtained after one iteration of TD-learning using the objective function in Eq. 40 is given by:
T TTass) (a(s)
a=(
a
(41)
In CQL, the result in Eq. 41 is obtained by setting the derivative of the objective in Eq. 40 with respect to the Q-values to 0, and solve for Qo(s, a) (Kumar et al., 2020).
We now restate the new regularizer introduced by ReDS, and the new objective function for the Q-function.
(2 (1 (1 [Q)(s +
1 E [Q((s E (Qo(s.
R(0:p)
(42)
2nDDamp
min - 1 Baa.s' (B
(43)
Notice that the main difference between the original CQL objective in Eq. 40 and the new objective in Eq. 43 is the distribution with which we push down Q values. The objective Eq. 40 pushes values down under the learned policy æ, whereas the objective in Eq. 43 pushes Q values down
under mixture of и and p. i.e. 1 1 does not contain the values, its gradient with respect the values is 0. Additionally, since the 1 mixture ग + 70 plays the same role in Eq that и plays in the objective function in Eq. 40, we therefore can obtain the solution for the Q-values after updating the Q-function using the objective
+ 7 Since p is parameterized by a neural network whose input
21

---

Under review as a conference paper at ICLR 2023
function in Eq. 43 simply by replacing in Eq. 41 with the mixture. That is, in tabular setting, after updating the Q-function using the objective Eq.43, the Q-values are:
π (a|s) + p
Qa(s,a) = as) -a
(a)()))
<(als) + p(als)
BB* a
2 (a|s) (
(aass) + p(als) 2
= -B*Q(s, a) a
2
QB I(aas) + TT [a(a)s) 2n 2T (als)
2T
since p subsumes ne 9, giving the desired result.
B.6 PROOF OF LEMMA 4.2
Intuition: Comparison of CQL and CQL (ReDS) objectives We will first intuitively compare the objectives in CQL and CQL (ReDS) to understand where the difference arises from. We write down the CQL (ReDS) objective below:
max J(T) well
E Swit p(r,, (())) +. E
(TT= > он]
(44)
2(1 7)
a>x((1e)
and now the CQL objective:
max [(]] xell (1 7) E Ba [D(T, (s)]
(45)
Observe that the only difference between CQL and ReDS stems from the fact that while the regular- izer in CQL only optimizes the policy to stay close to the behavior policy, the regularizer in ReDS minimizes an additional term E, π (() x(ass) This term attempts to make the behavior policy more "sharp", as g is monotonically decreasing function of its argument, by preventing to be less sharp states where T is broad. This enables и to find the action in the dataset support that maximizes the learned return [Tr, even when T is broad.
To prove Lemma 4.2, we will consider the following abstract update form for the policy evaluation version of CQL (ReDS), that obtains the next Q-function iterate Qu+1:
min a anrrr IQ(s, a)] Q
(((s.a)) + 2.
[(Q(s.a) - QU), (46)
Lemma B.5 (CQL (ReDS) restated more completely.). CQL (ReDS) solves the following optimiza- tion problem, when a is large enough:
max J(TT) 2(1 - 7) E E [(T, (s) + Ba (s) [g (7 [ 2 0}]
Proof. For proving Lemma 4.2, we follow an argument similar the proof of Theorem 3.1 from Kumar et al. (2020). By differentiating the above objective w.r.t. Q. we note that the ReDS + CQL objective above exhibits the following effective Bellman backup
1)
Vs, E D, QK+i a) == Qx)
(47)
Ta(a|s)
This backup is equivalent to running pessimistic RL with reward bonus equal to (a(s)) - -1). and therefore, the policy obtained by maximizing the resulting Q-function can -a (a(a)))
22

---

Under review as a conference paper at ICLR 2023
be expressed as:
max [(x) - a -I- E sndd Baar( (
1
(T)() {
π (als)
(48)
max J(xx - a 2(1 - 7) E sn [D(T, TT)
a 2(1 7)
E B.a [T)a((ass) > 0}g( (T π (a|s))]
(49)
We retain the notion support (a(s)(a)s) > 0} because ( in the denominator only cancels out for actions within the support of the behavior policy, as for all other actions, this term the term would be ill-defined as [a(a)s) = 0 appears in the denominator. Hence this term in the above equation cannot up over actions not in the behavior policy in the second term (these actions will be pushed down to have -00 Q-values). If we just cancelled the Fy in the denominator, note that g(T [Tass) ) not guaranteed be actions a, where (a(ass) 0. Hence, we must retain the indicator function to explain this result properly. Instead, for the case of standard CQL, the divergence D(T, (on,TT)(( (s) term prevents from putting non-zero density on actions where (T(aas) in CQL, or else D(T, (s) oc, meaning that the Q-value for such would be -o0.
C IMPLEMENTATION DETAILS OF CQL (REDS)
In this section, we will provide implementation details about our algorithm, CQL (ReDS). The pseudo-code in Algorithm 1 illustrates the different update steps of our algorithms. In addition, we provided detailed python-like algorithm description for ease of implementation. This can be found below in Section C.
Most of the components of Algorithm 1 are straightforward and follow the same convention, training update and, as we will discuss, hyperparameters as the CQL algorithm. This includes training the policy TT. and for the most part training the critic Qs. The main difference in the update for CQL (ReDS) is utilizing the mixture of π and in the CQL regularizer. For obtaining Pv. we utilize standard udvantage-weighten et al., 2020b; Peng et al., 2019). Following these prior works, we also clip the argument to the exponent between minimum range and maximum range to be numerically stable:
training update, following the papers (Kostrikov et al., 2021a; Nair
(g P (als) exp |clip (-A7(s a)/T, mmin: 0
(50)
In our experiments, we chose min = - 10 and o max = 5 across all the tasks and domains we study. These details are standard in training ad weeinttge-weighted algorithms.
C.1 DETAILED ALGORITHM DESCRIPTION FOR CQL (REDS)
Algorithm 1 provides the pseudo-code for CQL (ReDS). We provide the detailed description of how each update step in Algorithm 1 is implemented using Python syntax based on the Py PyTrrch Framework in this section. We include 3 code listings below, illustrating the update steps for the parametric Q-functions, the policy and the learnt distribution
23

---

Under review as a conference paper at ICLR 2023
Listing 1:
Training Q networks given batch of data, corresponding step 3 in Algorithm
q-data = critic (batch [ ooberrvations I, batch [ "acttoon']
next_dist actor (batchb "extoosststsss
next-pi.actions: next_log-pis next_dist saxple ()
target qval = target critic (batch (batch[""bsrrvationn" "bbservations') ],
target_qval batch rewards next-pi_actions)
self gamma (I batch 'dones') }) target-qval
td_loss mse_loss (q-data, target-qval)
importance sampling term num_samples
assume is normalized between [-1, 1] candom_actions uniform-san -amplessmmeeeeeemmmmee (nummsaample batch [ actions" [llp min=-1, max=1) random pi = 0.5 ** batch actions 'I. shape[-1]]
dist = actor (batch "'obeervation")) pi-actions, log-pis
sample (num_samples)
rho rho batch (arcch['bbbeeeee "observations") ']) rho log probs- rist sample (num num-samples)
q-rand_is critic (batch "observations') "J.
random_actions) - random_pi
q-pi-is critic (batch oobservationss '].
pi actions) log pis
q rho is critic (batch observations" rho_actions) log-probs.rho
cat-q = concatenate ( (q-rand_is.... q-pi-is, new.axis=True) cat_q logsumexp (cat-q. axis=-1)
cat q rho logsumexp( q rho is. axis=-1)
average between rho and pi tesse = 0.5 (cat-q + cat-q-rho)
reds llos id loss + (epush.down.term..uk
q-data) mean () cql-alpha)
critic critic.optimizer .ereerreeerrree grad ()
reds_loss backward ()
critic.optimizer. step (
24

---

Under review as a conference paper at ICLR 2023
Listing 2: Training the policy (or the actor) given batch of data (step 4 in Algorithm 1)
Identical to CQL
return distribution of actions
dist = actor (batch observations
# sample actions with associated log probabilities pi actions, log pis = d sample()
calculate value of actor actions
qpi critic (batch observations" I, actions)
qpi = qpi minmmiinxxis)
same objective as CQL (kumar et )l.) actor loss (log pis self alpha qpi ) mean ()
optimize loss
actor-optimize gero-grad ()
actor_loss backward ()
actor.optimizer. step ()
Listing 3: Training the Pe distribution given batch of data (step 5 in Algorithm 1)
AWR style update to find rho
sample policy actions for advantage calculation dist actor (batch observations ')) pi actions. log pis = dist. sample
calculate advantage qdata = critic (bathh observations"] (batchh""oserrvttooon"""
batch aatch"'atioons "actions") })
value = critic (batch observations 'I actions)
advantage = (qdata - value. min (0)) mean
awr style clipping aiipeedaavvntage clip (advaataatgg//sell self temperatur,, min=-10, max=5)
find log ra rho dist rho batch [ obserrvations' 'I)
log tog-prob_rho ris_dist log-prob (batch actions" '])
# Advaniage Weighted Log Probabilities is the loss for rho rho_loss = rhosloss rho_loss ()
* log-prob-rho)
rho optimizer.. zero grad (ackward rho.optimizer.ster () (t)
25

---

Under review as a conference paper at ICLR 2023
D TASK AND DATASET DESCRIPTIONS
In this section, we will describe the various tasks we introduce in this paper. We also provide qualitative descriptions of these tasks here.
Heteroskedastic antmaze navigation. We introduce four new antmaze datasets which exhibit two different dataset distributions each for the medium and large mazes from D4RI, (Fu al., 2020). We reuse the layouts of the mazes directly from D4RL, but modify the data collection protocol. For the noisy datasets, given an observation from the environment, we first compute the action that would have been taken by the D4RL behavior policy, and then add Gaussian noise to the action. While this alone is not much harder, crucially, note that the variance of this added Gaussian noise differs depending on the location of the Ant in the 2D Maze. In addition there is a small bias added the D4RL behavior policy, but this bias is dominated by noise. We present the noise standard deviations (indicated "Noise") and the bias added (indicated "Bias") for this dataset function of different location intervals in the maze in the left part of the Figure 8 below.
Noise and Bias in animazee noisy datasets
Noise and Bias antimaze- biased datasets
Rise
Rie
1:5
3.5
0.5
Noite
Dis Hobe
Joulbom of the ant the mace
positton of ant the made
Figure 8: The distribution of noise and bias in the heteroskedastic antmaze datasets as a function of the x-position of the ant in the maze. While the noisy datasets primarily add noise, the biased datasets also add significant bias beyond the noise. The value of given bar is the variance of the noise bias added in the region between the x-position for that bar, and the next one.
For the bi ased datasets, in addition to adding location-depender Gaussian noise to the action computed by D4RL behavior policies, we add a strong bias the action (see Figure 8 (right)). Cru- cially note that the direction of this bias (i.e., the sign) changes based on the location of the Ant in the 2D maze, which mimics the scenario studied in our didactic navigation example in Section 3. 1. In summary, because in some 2D regions of the maze, the values of the noise and bias added to the actions are larger, while in other 2D regions, they are smaller, the new offline datasets contain more heteroskedastic data distribution, where an optimal learned policy must deviate away from the data distribution much more in certain regions, whereas much lesser in other regions, which would correspond to increase in the differential concentrability. This is demonstrated quantita- tively in Table 4a. Thus, we expect that learning well on these tasks modulating the strength of the distributional constraints per state.
Visual robotic pick and place. We introduce a pick and place dataset, which exhibits a unique dataset distribution for robotic pick and place manipulation task, building on the framework from Singh et al. (2020). As shown in Figure 10, the robotic setup is a 6-DOF WidowX robot in front of a green bowl with 2 objects: a target object (the ball in this case) and distractor object (the can). The objective is to place the target object into the bin. The reward function is sparse, binary indicator of success, where a +1 reward is given when the object is placed in the bin. This task must be done from 128 x 128 x 3 raw visual observations, without access to either the robot state, or the state of the objects, which can change as the objects can roll on the surface.
Visual robotic bin sorting. We introduce bin sorting tasks, which are also built on the framework from Singh et al. (2020). As shown in Figure 10, the robotic setup is a 6-DOF WidowX robot in front of two identical white bins with 2 objects to sort The objective is to sort each object into its respective bins the target object in to the bin. The reward function is a sparse, binary indicator of success, where +1 reward is given when both objects are placed in their correct bins. This task must be done from 128 x 128 x 3 raw visual observations, without access to either the robot state, or the state of the objects, which can change the objects roll the surface.
26

---

Under review as a conference paper at ICLR 2023
Figure 9: Visualizing a sample trajectory for the visual plck-place robotic manipulation task. Here is an example successful trajectory in the dataset collected using a scripted policy. Th robot reaches for the target object (the green ball), lifts it, and places it inside of the green container.
Figure 10: Visualizing sample trajectories for the visual bin sorting robotic manipulation task. Here are two sample trajectories for the binsorting domain. Top: successful trajectory in the dataset. Here the robot places the white cylinder in the right bin and the blue ball in the left bin, succesfully sorting the objects into their respective bins. Bottom: A unsuccessful trajectory in the dataset. Here the robot places both objects in the same bin. This is unsuccessful an object was placed in the incorrect bin thereby not sorting them in a correct manner.
To collect heteroskedastir dataset, run data collection using hardcoded scripted control policies, whose success rate and variance can be controlled. Each trajectory in the dataset was collected in the following manner. For the first phase, the robot reaches towards the object without any bias and grasps it with a reasonable success rate. Here, though, the noise and stochasticity in the scripted data collection and the inaccuracies in the scripted policy make not succeed for every trial. During the second phase, where the robot places the object in the bin, there was a bias towards placing the object position in the workspace that does not correspond the target location of the bin for which the robot can attain a reward, and which the robot will observe during evaluation. For our experiments, this bias was 85%. This forces the data distribution be heteroskedastic: for the picking segment of this task, the behavior policy is centered around the desired optimal behavior i.e., grasping the object successfully, whereas for the placing segment, the behavior is biased towards carrying the object to the incorrect regions, requiring significant deviations from the behavior policy to succeed. An algorithm now required to have a non- non-uniform amount closeness to the behavior policy.
For the bin sorting domain, the easy domain had no bias and instead the scripted policy had additive gaussian white noise with a fixed variance which leads to low heteroskedacity.
Atari game playing, For the Atari tasks we consider in the paper, we devised a heteroskedastic data composition based on the DQN Replay dataset (Agarwal et al., 2020) which comprises of transitions found in the replay buffer of run of online DQN. Since this dataset consist of all the policies that the DQN agent produced over the course of training, and since Atari typically uses e-greedy exploration, where the value of e decays over time, different trajectories of this dataset are generated
27

---

Under review as a conference paper at ICLR 2023
from different behavior policies, that all have different levels stochasticity. Naturally, since the value of e decays over training and the performance of online DQN increases, the trajectories with higher return generally correlate with having lower stochasticity.
Given this information, we attempted to subsample a dataset that is heteroskcdastic. For this purpose, first divide the full replay buffer into N equal chunks, where chunk 0 consists to experience observed earliest in training, while the chunk N 1 consists of experience seen near the final parts of training. Then, we subsample 20% of the trajectories from each of these chunks independently to obtain an intermediate dataset that comes from multiplies policies, observed at different times while training online DQN. Then, for any given trajectory of length in the replay chunk i, we only retain the transitions occurring between time steps - [--(+1)xN] in our final dataset 1. and discard all the remaining transitions. This essentially means that the data closer to the initial states of the game comes from a good, less stochastic policy, whereas the data close to the final states of the game from a worse, highly stochastic policy. We develop two such datasets corresponding to N = 2 and N = 5 chunks. These chunks concatenated together construct the replay buffer of transitions that correspond to the 2 and 5 policy experiments seen in Section 5.
To see why this data is heteroskedastic, note that at different states of the game, we observe actions with different amounts of stochasticity and bias. This is because, as the game progresses, the effec- tive behavior policy induced by the offline dataset exhibits bias towards suboptimal actions (from the chunks that are earlier in DQN training) while also exhibiting substantial noise. The states that are closer to the initial states of the game, on the other hand, have an effective behavior policy that is primarily centered around good action, with very little noise. order to succeed, an offline RL. algorithm must have different amount of conservatism at different states.
In our experiments, we considered 10 games including several standard games, and this is a subset of games studied in prior work (Kumar et al., 2021). The games considered are: ASTERIX, BREAKOUT, Q' BERT, SEAQUEST, SPACEINVADERS, BEAMRIDER, MSPACMAN, WIZARDOF- WOR, JAMESBOND, PONG.
E EXPERIMENTAL DETAILS
For our experiments on the AntMaze domains, we built on the following open-source implemen- tation of CQL: https: / / ggithu........ com/ hot///yoogggggoee geng/ JaxCQL, for our visual robotic exper- iments, we utilized our own port of the following implementation from Singh et al. (2020) in Jax: https: htt://gitti thu. con/ aevisinght 99 cog, and for our Atari experiments, we use the offi- cial implementation of CQL built on Dopamine (Castro et al., 2018): https: / /github.... com/ avi ral kuma / /CLL/ tree / masttr/////eeaaaaaaaaaaaammwe aatari. For certain baselines (e.g EDAC, BEAR), we utilize the source implementation to stay consistent with the author's 's tested and tuned implemen- tation. We additionally verified the results for D4RL. benchmark for these tasks. We will summarize the hyperparameters in the next sections.
E. HYPERPARAMETERS FOR CQL (REDS)
Antmaze domains. For the AntMaze domains, we utilized a temperature parameter T 0.3 in our experiments (found by sweeping over 7 E [0.1, 1.0, 5.0)). for all the four dataset types in Table 4b. Every other hyperparameter was kept identical to CQL, which for the case of antmaze corresponds to applying the CQL regularizer R(0) with the dual version of CQL, where the threshold on the CQL regularizer is specified to be 1.0. Following CQL, we used 3-hidden layer critic and actor networks with layers of size 256, critic learning rate of 3e-4 and an actor learning rate of 1c-4. We utilized the Bellman backup that computes the target value by performing a maximization over target values computed for k = 10 actions sampled from the policy at the next state.
Atari domains. For our Atari experiments, we tuned the value of a in CQL (Equation 2) between two values 10.1, 0.2], and present the sensitivity results in Figure 4, and found that = 0.1 work better for CQL. We swept the value of r E [2.0, 5.0, 7.0] and report the sensitivity sweep in Fig- ure 11.
Visual Robotic Domains. For the visual pick and place domains, we follow exactly the same hyperparameters as the CQL implementation from COG (Singh al., 2020): a critic learning rate
28

---

Under review as a conference paper at ICLR 2023
of 3e-4, actor learning rate of le-4, using i = 4 actions from the policy for computing the target values for computing the TD error, and using &o = 4 actions to compute the log-sum-ext exp in CQL. For the value of swept 7 E [0.1, 1.0, 10.0, 100.0), and used t 1.0 for our experiments.
F ADDITIONAL ABLATION STUDIES
In this section, we present some results of an ablation study of the performance of CQL (ReDS) with respect to the temperature hyperparameter 7 that appears in Equation 9. Before discussing the results, let us intuitively aim understand the significance of this hyperparameter. When is extremely small we would expect to be distribution centered at the worst possible action, within the support of the behavior policy. When large, we would expect the learned Pe to be close to the behavior policy, since the exponentiated advantage term would essentially behave against a uniform distribution. Neither of these extremes are desirable, while the former does not behave much differently than distributional constraint (except that the Q-value at the action with the smallest Q-value the dataset support is not pushed up anymore), the latter also behaves like distributional constraint, but with just half the effective multiplier would therefore expect an intermediate to perform the best.
constraint
the CQL regularizer. We
To verify these insights, study the sensitivity of the performance of CQL (ReDS) with respect to a on the Atari datasets. Our results shown in Figure 12 confirm that indeed an intermediate value of 5.0 out of the tested values, [2.0, 5.0, 7.0] works the best.
IQM
Mean
ReDS + CQL, 0.1, T=7.0
ReDS + CQL, a= 0.1, 5=5.0
ReDS + CQL, 0= 0.1, T=2
COL, a=1
0.30
0.45
0.60
0.75
0.40
0.48 0.56 0.64
Normalized Score
Figure 11: Sensitivity of CQL (ReDS) to the temperature hyperparameter in Equation evaluated on the Atari game experiments with 5 policies. Observe that an intermediate value of temperature 5.0 works best
In addition, we study the sensitivity of the performance of CQL (ReDS) with respect to a on the Atari datasets. We report the performance for two different values of E {0.1, 0.2} from CQL (Equation 2) in Figure 12. Observe that CQL (ReDS) with given a outperforms base CQL for the corresponding Additionally note that the degradation in performance of CQL (ReDS) as increases is lesser than base CQL.
G ADDITIONAL BASELINE COMPARISON FOR HETEROSKEDASTIC ANTMAZE NAVIGATION
In this section we will provide additional baseline comparison for REDS with additional Offline RL methods: EDAC (An et al., 2021) and BEAR (Kumar et al., 2019).
G.1 HYPERPARAMETERS FOR EDAC
As done in al. (2021), we tune the method over two hyperparameters. The first hyperparam- eter is the ensemble size N which specifies the number of Q functions. The second parameter we consider 17, the weight of the ensemble gradient diversity term. Below in table 4, we show the values considered for each hyperparameter. There is significant overlap to these parameters with the ones used in the Mujoco Gym and Adroit Domains that the authors used. We utilized the publicly available code (https: / / q writhubb.. com/ enu-mll 1 ab / /EDAC) released by the authors of EDAC and were able to replicate the results they reported for the D4RL MuloCo Gym environments in An al. (2021).
29

---

Under review as a conference paper at ICLR 2023
IQM
Mean
ReDS CQL, 0.2
ReDS + CQL, 0.1
CQL, a= 0.2
CQL, a 0.1
0.75 0.90 1.05 1.20
0.8
1.0
1.2
Normalized Score
IQM
Mean
ReDS CQL, a
ReDS COL, mortr
CQL, a= 0.2
CQL, a a= 0.1
0.45
0.60
0.75 0.3
0.4
0.5 0,6
Normalized Score
Figure 12: Sensitivity of ReDS CQL to the temperature hyperparameter a Equation
We report the performance of CQL (ReDS) CQL on the IQM normalized score and the mean normalized score over ten Atari games, for the case two (top) and five (bottom) policies. We consider this performance for two different values of a {0.1, 0.2) in CQL (Equation 2). Observe that CQL (ReDS) with a given a outperforms base CQL for the corresponding a. Additionally note that the degradation in performance of ReDS (CQL) increases is lesser than base CQL..
Table 4: EDAC Hyperparameters
Hyperparameters
Values
N
10, 20, 50, 100
7
0, 1, 10, 50. 100, 1000
G.2 HYPERPARAMETERS FOR BEAR
As done Kumar et al. (2019), we tuned this method over two hyperparameters. The first is the Kernel Type of the MMD between the behavior policy Ty and the actor and found that Laplacian performed better. The second parameter considered is o. which needed for the Laplacian kernel as defined. Below in table 5, show the values considered for each hy- perparameter. There is significant overlap to these parameters with the ones used in the Mu- joco Gym and Adroit Domains that the authors used. We utilized the publicly available code (https: //github. com/ /rail- trailbbat -erbeeeeeeeeee peey/ddrl_evaa of BEAR and were able replicate the results they reported for the D4RL MuJoCo Gym environ- ments in Kumar et al. (2019).
ions) released by the authors
Table 5: BEAR Hyperparameters
Hyperparameters
Values
Kernel Type
Laplacian, Gaussian 10, 20, 50
30


Reviews:
[{"review_id": "xChc4cj8Gae", "reviewer": "Reviewer_HyhP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discovers that there may exist heteroskedasticity in realistic offline RL datasets, making current offline RL methods with distributional constraints suffer from performance degeneration. To address the problem, the paper proposes a novel method ReDS to convert distributional constraints into support-based constraints. Several new heteroskedastic datasets are constructed to demonstrate the superiority ReDS over current offline RL methods.", "summary_of_the_review": "I recommend acceptance", "strength_and_weaknesses": "The paper is well-written, theoretically motivated, and technically sound. I like Figure 1 which clearly illustrates the motivation of ReDS. However, I have two concerns.  \n\nFirst, how well does ReDS perform over non-heteroskedastic datasets? \n\nSecond, can ReDS be applied to offline datasets with a small number of samples? Do insufficient samples impose difficulty in estimating the distribution?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qpCjIHJ6irY", "reviewer": "Reviewer_d9Ww", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm called CQL (ReDS), which uses reweighted data distribution to make the algorithm act like having a support constraint instead of a distribution constraint. For most of the complex offline RL problems, dataset distribution will be heteroskedastic, i.e. will contain different variances of actions in each state, and having a distribution constraint, in this case, can lead to a limited performance because we can always be overly pessimistic or optimistic for certain states. To prevent this, this paper show a simple way to achieve a support constraint instead with a small modification from original CQL algorithm, by using a mixed distribution $\\pi^{re}$ instead of $\\pi$ used to regularize in CQL.\n\n$\\pi^{re}$ is half $\\pi$ and half $\\rho$, where $\\rho$ is some distribution of actions that is high when $\\pi_\\beta$ is high and $\\pi$ is low, and $\\rho$ will be low when $\\pi_beta$ is low and $\\pi$ is high. By having this additional $\\rho$ term, policy $\\pi$ of CQL(ReDS) can now be dramatically different from $\\pi$. Experiment results show that by having this modified regularization the algorithm improves much.", "summary_of_the_review": "Overall, I think the presentation quality is sufficient for being accepted. However, there are some points about the theory that is not clear to me, and I cannot recommend acceptance in its current form. I am ready to raise my score if those concerns are resolved.\n\n----------\nReading the authors' response, I decided to raise the score. Thanks for the detailed response!", "strength_and_weaknesses": "Strengths\n- High-quality descriptions with good examples\n- Extensive evaluations over many different domains\n\nWeaknesses\n- Theoretical results are hard to follow and somewhat misleading:\n\n1. Theorem 3.1 is especially hard to understand. While the policy improvement is shown with $\\zeta^+$, the constraint is over $\\zeta$, which makes it confusing because $\\zeta$ and $\\zeta^+$ do not seem to be related to each other in my opinion. And there is nothing about $\\delta$ even though the theorem holds under prob. $1-\\delta$. What is the variable that is constrained by the constraint? Is it $\\alpha$ that is determined by the constraint? If it is the case, $J(\\pi_\\alpha)$ is determined by $\\alpha$, which should not be used outside the maximization of eq (5). Overall, it is really hard to interpret the theorem. \n\n2. Why do we need to use $g(\\exp(A(s,a)))$ instead of $g(\\pi(a|s)$? Can't we directly use $g(\\pi(a|s))$ to weight the objective of eq (9)? It seems there isn't anything blocking us from doing such.\n\n3. Lemma 4.2 seems to be quite misleading. Authors argue that we can put the notion of support with $\\epsilon$ in the second term, but doesn't $\\pi_\\beta$ cancels out in the second term due to the definition of $\\rho$? And the argument authors made seems to be also able to be applied on CQL case, since we can say that CQL regularization can be arbitrarily high when $\\pi$ is high and $\\pi_\\beta$ is low. If it doesn't make CQL a support constraining algorithm, I think (ReDS) cannot be said to be a support constraining algorithm.\n\n- Minor point: it seems gradient signs $\\nabla_\\psi, \\nabla_\\phi$ are omitted in Algorithm 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PaSAj9x8At", "reviewer": "Reviewer_N7S6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a novel setting, i.e., learning policy from a heteroskedastic dataset with the demonstrated behaviors varies non-uniformly across states. For example, for a given maze environment and some non-optimal behavior policies, the states in the narrow hallways may exhibit a narrow action distribution, while more uniform actions for a wide room. In my view, this is true. In many cases, we cannot obtain the optimal policy with narrow action distribution for every state (if we can, we don’t need offline RL.). And suboptimal policies prefer to show different actions for a given state.\n\nThe authors propose empirical and theoretical evidence (Theorem 3.1) showing that explicit distributional constraints between the policy action and the (non-optimal) dataset actions may undermine the performance of the learned policy . \n\nThe authors try to achieve a support-based policy constraint to improve over the non-optimal behavior policies that generated the heteroskedastic datasets. That is, if the policy decision is within the support of the behavior policy, then the constraint should be zero. This constraint is weak than the distributional constraint. It thus holds the promise to derive a better policy on the states with more uniform action (non-expert) distributions. \n\nTo practically achieve the support constraint, the authors reweigh the behavioral policy into \\pi_{re} and implicitly minimize D(\\pi, \\pi_{re}, where\n$\\pi_{re}(\\cdot |s) = \\frac{1}{2}\\pi(\\cdot \\pi) + \\frac{1}{2}[\\pi_{beta}(\\cdot |s) \\cdot g(\\exp(A(s, \\cdot)))]$.\n\nThis work proposes an offline RL algorithm with support-based policy constraints. Although the evaluation is only based on the heteroskedastic dataset, I believe this pradgigm enjoys priority over the distributional constraint methods, such as the recent strong baseline algorithm, TD3+BC. \n", "summary_of_the_review": "See above", "strength_and_weaknesses": "## Pros\n- Achieving support-based constraints is well-motivated and elegant. In fact, there exist support-based constraint methods, e.g., BEAR. But MMD constraint can not be accomplished in practice. Thus it performs poorly on the medium-expert dataset (with a wider action distribution from the medium-level policy). This paper implicitly achieves support-based constraints without estimating a behavior policy, which is elegant. \n\n- The evaluation is conducted on four different types of datasets, including (1) D4RL Mujoco locomotion (*-medium-expert and *-medium-replay) datasets, (2) noisy/biased D4RL maze datasets, (3)the manipulation tasks, and (4) Atari games. The baseline algorithms, in my view, are also comprehensive.\n\n## Concerns\n- (Setting) The author says that the dataset for offline RL algorithms must illustrate the consequences of a diverse range of behaviors. As a result of this coverage, e.g., combining many realistic sources of data, algorithms should be able to learn from heteroskedastic datasets. But (1) I don't understand the logic of the argument. If such coverage benefits offline RL algorithms, why do we need to modify the existing offline RL algorithms for the dataset with good coverage? (2) Another concern is that recent work [1] shows that vanilla off-policy RL agents can outperform carefully designed ofﬂine RL algorithms with sufﬁciently diverse exploratory data. Could you please compare the heteroskedastic data and the diverse exploratory data? (3) Also, I am concerned about the motivation of the setting. The author clarifies the characteristics of heteroskedastic datasets, i.e., some dataset states correspond to narrow actions while others may be wider. But I can't easily see how this dataset relates to real-world applications.\n\n- (Algorithm) The authors build their proposed algorithm on top of CQL, with improvements focused on implicit support based constraint. The modification on the CQL regular term is described in Eq. (8), which involves sampling over the re-weighted policy, \\pi^{re}. I understand the idea that applying a distributional constraint on this re-weighting imposes a support based constraint. But I am not sure why the reweighted policy is deigned as a mixture of the learned policy \\pi and the \\rho, as described in Eq. 7. Could you please explain why the coefficient is designed as 0.5 and 0.5, or why not only use a \\rho? By the way, the sampling strategy of Eq. 9 and the third line of the algorithm seem to be inconsistent.\n\n- (Evaluation) The authors completed tests on three different datasets. Evaluations on the D4RL medium-expert and medium-replay datasets are really motivated, as these types of behavior policies can be common. But I think the dataset used in the second experiment, i.e., the noisy and biased D4RL antmaze dataset, is a bit contrived. \n\n- (Minor)It could be better if the authors add a discussion about the scope of the method. The problem considered in this paper can be well summarized in Figure 3: when the actions corresponding to the dataset states contain both expert and non-expert decisions, the support-based constraint method should be used. I think this is quite true, as actions from experts and non-experts can be different. However, when only non-expert dataset actions are available for some dataset states, how does the support-based constraint approach work? This can be true, as non-expert actions often result in a transfer to a non-expert state that differs from the state visited by expert behavior policy. Finally, I want to say that I don't think this discussion detracts from the contribution of this paper. It is entirely up to the authors to decide whether or not to give such a discussion.\n\n## Some typos.   \n- Page3, the first line, \\mu should be replaced with \\pi.\n- Page6, under equation 7, Eq.3 should be Eq.2.\n- Page6, under equation 11, Eq.30 should be Eq.11.\n- Page7, ‘constrain the entropy of … to be uniform’.\n\n\n[1] Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning. https://arxiv.org/abs/2201.13425", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject