ReDS: Offline RL With Heteroskedastic Datasets via Support Constraints
Authors: Anikait Singh
Aviral Kumar
Quan Vuong
Yevgen Chebotar
Sergey Levine
Venue: NeurIPS 2023 poster (2023)
Keywords: offline RL
support constraints
heteroskedastic data
TL;DR: We show that conventional distributional constraint RL algorithms are need with heteroskedatic datasets. We propose an offline RL method to handle such settings.

Abstract: Offline reinforcement learning (RL) learns policies entirely from static datasets. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy **to the same extent** across the state space. Ideally, the learned policy should be free to choose **per state** how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is theoretically motivated, and improves performance across a wide range of offline RL problems in games, navigation, and pixel-based manipulation.

ReDS: Offline Reinforcement Learning With Heteroskedastic Datasets via Support Constraints
Anikait Singh" 1.,, Aviral Kumar Quan Vuong', Yevgen Chebotar", Sergey Levine
UC Berkeley, Google DeepMind ("Equal contribution)
asap77720berkeley.oo
Abstract
Offline reinforcement learning (RL) learns policies entirely from static datasets. Practical applications of offline RL will incvitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due the requirement stay close the behavior policy to the same extent across the state space. Idcally, the learned policy should be free to choose per state how closely follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is theoretically motivated, and improves performance across a wide range of offline RL problems in games, navigation, and pixel-based manipulation.
conservative Q-
1 Introduction
Recent advances in offline RL [39, 36] hint at exciting possibilities learning gigh-performing policies, entirely from offline datasets, without requiring dangerous [19] expensive [25] active interaction. Analogously to the importance of data diversity in supervised learning [9], the practical benefits of offline RL depend hcavily the coverage of behavior in the offlinc datasets [35]. Intuitively, the dataset must illustrate the consequences of a diverse range of bchaviors, so that an offline RL method can determine what behaviors lead high returns, ideally returns that significantly higher than the best single behavior in the dataset.
One easy option to attain this kind of coverage is to combine many realistic sources of data, but doing so can lead to the variety of demonstrated behaviors varying in highly non-uniform ways across the state space, i.e. the dataset is heteroskedastic For example, driving dataset might show very high variability in driving habits, with some drivers being timid and some more aggressive, but remain remarkably consistent in "critical" states (e.g., human drivers extremely unlikely to swerve in an empty road or drive off a bridge). A good offline RL algorithm should combine the best parts cach behavior in the dataset c.g., in the above example, the algorithm should produce a policy that is as good as the best human in each situation, which would be better than any human driver overall. At the same time, the learned policy should not attempt to extrapolate novel actions in subset of the state space where the distribution of demonstrated behaviors is narrow (e.g., the algorithm should not attempt to drive off bridge). How effectively current offline RL methods selectively choose per-state basis how closely to stick to the behavior policy?
37th Conference on Neural Information Processing Systems (NeurIPS 2023),

---

Most existing methods [32, 33, 28, 27, 53, 17, 23] constrain the learned policy to stay close to the behavior policy with so-called "distribution constraints". Using a combination of empirical and theoretical evidence, we first show that distribution constraints are insufficient when the heteroskedas- ticity of the demonstrated behaviors varies non-uniformly across states, because the strength of the constraint is aagnosa,,, and may be overly conservative at some states even when it is not conservative cnough at other states. Wc also devise a measure of heteroskedasticity that enables us to determine if certain offline datascts would be challenging for distribution constraints.
Our second contribution is simple observation: distribution constraints against a reweighted version of the behavior policy give rise to support constraints. That is, the Sttun-maximization optimization process can freely choose per-state how much the learned policy should stay close to the behavior policy, so long as the learned policy remains within the data support. We show that it is convenient to instantiate this insight on top of conservative Q-learning (CQL) [33], a recent offline RL method. The new method, CQL (ReDS), changes minimally the form of regularization, design decisions employed by CQL and inherits existing syper-parameter values. CQL (ReDS) attains better performance than recent distribution constraints methods on a variety of tasks with more heteroskedastic distributions.
2 Preliminaries
The goal in offline RL is find the optimal policy in Markov decision process (MDP) specified by the tuple M = (S, Po, 7). S,A denote the state and action spaces. T(s'ls, a) and r(s, a) represent the dynamics and reward function. (o(s) denotes the initial state distribution. E (0, denotes the discount factor. We wish to learn a policy that maximizes return, denoted by (Tx) := T- ((s. 7 az)]. We must find this policy while only having access to an offline dataset of transitions collected using a behavior policy Tg, D = {(s, a, r')))
Offiine RL via distributional constraints. Most offline RL algorithms regularize the learned policy æ from querying the target Q-function on unseen actions [17, 30], either implicitly or explicitly. For our theoretical analysis, we will abstract the behavior of distributional constraint offline RL algorithms into a generic formulation following Kumar et al. [33]. As shown in Equation 1, we consider the problem where we must maximize the return of the learned policy a (in the empirical MDP) [T)) while also penalizing the divergence from TTa
max E Eg-ă- (J() aD(T, n3 (s)
where D denotes a divergence between the learned policy and the behavior policy Ts at state Conservative Q-learning. [33] enforces the distributional constraint on the policy implicitly. To see why this is the case, consider the CQL objective, which consists of two terms:
E. E....'' (Qo(s, - Ba Q(s, a))
min a (BanD. (Qo a)] - Es.an (Qo a)))
(2)
R())
where BT (s0((s,) a) is the Bellman backup operator applied a delayed target Q-network, Q: 8 ((, a) ;= r(s,a) + > [Q(s,, a')]. The second term (in blue) is the standard TD error [40, 18, 22]. The first term R()) (in red) attempts to prevent overestimation in the Q-values for out-of-distribution (OOD) actions by minimizing the Q-values under distribution ((als), which is automatically chosen to pick actions with high Q-values Qo a). and counterbalances by maximiz- ing the Q-values of the actions in the dataset. Kumar al. [33] show that Equation 2 gives rise to pessimistic Q-function that modifies the optimal Q function by the ratios of densities, π T(als) / (als) given state-action pair (s, a). Formally, the Q-function obtained after one iteration is given by:
π (a)s)
Qs(s.a) = (ss
(a)s
The Q function is unchanged only if the density of the learned policy i matches that of the behavior policy T. Otherwise, for state-action pairs where T(als) < TT(aass, Eq. 3 increases their Q values and encourages the policy и to assign more mass to the action. Vice versa, x (a)s) > T (a]s)) Eq. 3 encourages the policy и to assign smaller density to the action a. In Eq. 3, a is constant for every state, and hence the value function learned by CQL is altered by the ratio of action probabilities to the same extent at all possible state-action pairs. As we will discuss in the next section, this can be sub-optimal when the learnt policy should stay close to the behavior policy in some states, but not others. We claborate on this intuition in the next section.

---

3 Why Distribution Constraints Fail with Heteroskedastic Data
In statistics, heteroskedasticity is typically used to refer to the condition when the standard deviation in a given random variable varies non-uniformly over time (see for example, Cao ct al. [6]). We call offiin dataset heteroskedastic when the variability of the behavior differs in different regions of the state space: for instance, for ccrtain regions of the state space, the observed behaviors in the dataset assign the most probability mass to few actions, but in other regions, the observed behaviors are more diverse. Realistic offline datasets are often heteroskedastic as they are typically generated by multiple policies, each with its own characteristics, under different conditions. E.g., driving datasets come from multiple humans [11], and many robotic datasets are collected by multiple teleoperators [10], resulting in systematic variability in different regions of the state space.
3.1 A Didactic Example
To understand why distribution constraints are insufficient with heteroskedastic data, we present a didactic example. Motivated by the driving scenario, navigation task shown in Fig. 1. The task to navigate from the position labeled "Start" to the position labeled as "Goal" us- ing five actions every possible state (L: t,, R: -, U: f. D: L. No: No Op), while making sure that the executed actions do not hit the walls of the grid.
consider a maze
hetercokedasticc data
Ei AME RSACK
In DaL sack LEGEND
Geal
Varneblity Depence
D.L CUICy
chose
10 COL PPES) succeeds
Figure 1: Failure mode of distribution constraints. In this navigation task, an offline RL algorithm must find a path from the start state to the goal state as indicated in (a), The offline dataset provided exhibits non-uniform coverage at different states, e.g., in the state marked as "B" located in wide room has more uniform action distribution, whereas the states in the narrow hallways exhibit a more narrow action distribution. This is akin to how the behavior of human drivers varies in certain locations ("B"), but is very similar in other situations ("A"). To perform well, an algorithm must stay close to the data in the hallways ('*A"), but deviate significantly from the data in the rooms ("B"), where the data supports many dif- ferent behaviors (most are not good). AWR and CQL get stuck because they stay too close to the bad behavior policy in the rooms, e.g. the left and right arrows (b) and (c). Our method, CQL (ReDS), learns ignore the bad behavior action in state B and prioritizes the good action, indicated by the downward arrow near B in (d).
Dataset construction. To collect het- eroskedastic dataset, we consider a mixture of several behavior policies that attain a uniform occupancy over different states in the maze. However, the dataset action dis- tributions differ significantly in different states. The induced action distribution is heavily biased to move towards the goal in the narrow hallways (e.g., the behavior pol- icy moves upwards at state A)). In contrast, the action distribution is quite diverse in the wider rooms. In these rooms, the be- havior policy often selects actions that do not immediately move the agent towards the goal ge.g., the behavior policy at state B), because doing so does not generally hit the walls as the rooms are wider, and hence the agent is not penalized. Whereas, the agent must take utmost precaution to not hit the walls in the narrow hallways. More details are in Appendix B.
State B in Fig
Representative distribution constraint algorithms such as AWR [44, 43] and CQL [33] fail to perform the task, as shown in Figure 1. To ensure fair comparison, we tune each method to its best evaluation performance using online rollouts. The visualization in Figure 1 demonstrates that these two algorithms fail to learn reasonable policies because the learned policies match the random behavior of the dataset actions too closely in the wider rooms, and therefore are unable to make progress towards the Goal position. This is direct consequence of enforeing too strong of constraint on the learned policy to stay close to the behaviors in the dataset. Therefore, we also evaluated the performance of CQL and AWR in this example, with lower amounts of conservatism (Appendix B) and found that utilizing lower amount of conservatism suffers from the opposite failure mode: it is unable to prevent the policies from hitting the walls in the narrow hallways. This means that conservatism prevents the algorithm from making progress in the regions where the behavior in the dataset is more diverse, whercas not being conservative enough hurts performance in regions where the behaviors in the dataset agree with each other. The method propose in this paper tackle this challenge, indicated as "CQL (ReDS)", effectively traverses the maze, 80% of the time.
3.2 Challenges with Distribution Constraints
Having seen that distribution constraints can fail in certain scenarios, we now formally characterize when offline RL datasets is heteroskedastic, and why distribution constraints may be ineffeective in
3

---

5
Uniform behavior distribution 1its spioo is mughly unioor betwaon states, Tg 1.82
Sucoess 97.5
behavior diatribution
DCs 4000 la uniform between aterio, Gy
Sunnnos Rater 2
distribution D26 4,000 vares quite heartly states, Ciy
Success Rate:: %
Figure Empirically computing Cäi with three datascts: uniform (top), mixed (middle) and skewed (bottom) on a gridworld. We also visualize D(Tr, (s) across states in the maze as the colors on different cells, a histogram of D(r, The (s) (s) to visualize variation in this quantity and the performance of running standard CQL. The uniform distribution leads to low Can- uniform (1T,, va (*). and highest success. Middle: The mixed distribution leads to medium Cdir. less uniformly distributed ( me (s). and drop task success. The skewed distribution leads to a high Calt. non non-uniform D(n, x and poor performance.
Top:
Bottom:
such scenarios. Similar to how standard analyses utilize concentrability coefficient [46], which upper bounds the ratio of state-action visitation under a policy d" (s. a) and the dataset distribution μr, i.c. max,, d' (s, a)/m(ss a) < C we introduce a new metric called differential concentrability, which measures dataset heteroskedasticity (i.c., the variability in the dataset behavior across different states).
Definition 3.1 (Differential concentrabilityy). Given a divergence D over the action space, the differential concentrability of given policy with respect to the behavioral policy Tg is given by:
(V
D(T, ((T,,)))(s)) ((:))
V D(T,, (52
27
Cifff
(4)
((22)
Eq. 4 measures the variation in the divergence between a given policy x xalss and the behavior policy Te(a))) weighted inversely by the density of these states in the offline dataset (i.c., p(s) in the denominator). For simplicity, let us revisit the navigation example from Section 3.1 and first consider a scenario where μ(s) = Un) For any given policy π, if there are states where i chooses actions that lic the fringe of the data distribution (c.g., in the wider rooms), as well as states where the policy π chooses actions at the mode of the data distribution (e.g., as the narrow passages), then Cäi would be large any policy ı that learn. Crucially, C would be small even the learned policy π deviates significantly from the behavior policy Ta, such that ((T,, T3 TT) (s) is large, but (TT, (s1) D(=, mT)(s2) is small, indicating the dataset is not heteroskedastic.
Connection between variability in the action distribution and high Cair- Consider a simpler formula where remove the counts n(s) from the expression of differential concentrability and set π in Ch to the uniform distribution over actions. Then, we can show that the value of C is exactly equal to twice the variance of D(=, (s)(s) across states. Therefore, we will demonstrate in Section that arbitrary policy checkpoints π learned by offline RL algorithms generally attain a low value of the variance in D T (s) offline datasets from non-heteroskedastic sources, such as those covered in the D4RL [13] benchmark. Of course, we cannot always exclude the counts of note that in nigh-dimensiooal state spaces, such those in our experiments,
states ns),, however, each state in the offline data is likely to be unique, thus validating the condition that n(s) = 1. That said, do compute the exact value of C (with ns)) in didactic gridworld maze shown in Figure 2. In this case, we find that our definition of C dir is actually able to reflect the intuitive notion of heteroskedasticity.
We now use the definition differential concentrability to bound both the improvement and de- provement of w.r.t. ng for distribution constraint algorithms using the framework of safe policy

---

improvement [37, 331. We show that when Cän is large, then constraints (Eq. 1) may not improve significantly over n,, even for the best value for the weight a (proof in Appendix C):
Theorem 3.2 (Informal; Limited policy improvement via distributional constraints.). W.h.p. 2 1 8, for any prescribed level of safety C. the maximum possible policy improvement over choices of ou maxa (J(Ta [TT)]] S 5-, where C+ is given by:
h- (a) (1 (1->)a
C1 Vog
log
C.T Ji
aE
1-? [TTTa TT)
(t :+:= max
Ss
(5)
[DI
where h" is a monotonically decreasing function of a, and h(0) 0(1.
Theorem 3.2 quantifics the fundamental tradcoff with distribution constraints: to satisfy given (-safety constraint in problems with larger Ciffs would need larger a. Since the maximum policy improvement <* is upper bounded by h" (a), the policy may not necessarily improve over the behavior policy if is large. On the flip side, if we choose to fix the value of to be small in hopes to attain more improvement in problems where Cairt is high for all policies, we would end up compromising on the safety guarantee this case, the policy may not improve over the behavior policy reliably.
needs be large for small and large C Thus, in
Note that larger value of Cār need not imply large E g- (D(T,, (T)(s)] because the latter does not involve (s).. Cai also measures the dispersion of D (z). while the latter performs a mean over states. In addition. Theorem 3.2 characterizes the maximun possible improvement with an oracle selection of a, though is not feasible in practice. Thus, when C is large, distribution constraint algorithms could either not safely improve over TT or would attain only a limited improvement with any possible value of a. Finally, we remark that complementing [32, 39] that discuss failure modes of distribution constraints with high-entropy behavior policies, Theorem 3.2 quantifies when this would be the case: this happens when C is large.
4 Support Constraints As Reweighted Distribution Constraints
Thus far, we have seen that distribution constraints can be incffective with heteroskedastic datascts. If we can impose the distribution constraint such that the constraint strength can be modulated per state, then in principle, we can alleviate the issue raised in Theorem 3.2 and Section 3.1.
Our key insight is that by reweighting the action distribution in the data before utilizing distribution constraint, we can obtain a method that enforces a per-state distribution constraint, which corresponds to approximate support constraint. This will push down the values of actions that are outside the behavior policy support, but otherwise not impose a severe penalty for in-support actions, thus enabling the policy deviate from the behavior policy by different amounts at different states. Rather than having distribution constraint between π and na (Eq. 1), if we can impose a constraint between and reweighted version of T3. where the reweighting is state-dependent, then we can obtaín an approximate support constraint. Let the reweighted distribution be π² Intuitively, if . (T|(s) is within the support of the (-)s) then one can find reweighting T=e (-|s) such that D(x, """ (s) whereas if m (-[ss) is not within the support of (-ss). then D x*))(s) still penalizes when и chooses out-of out-of-support actions, since no reweighting nh can put non-zero probability on out-of- support actions. This allows us to handle the failure mode from Section 3: at states with wide behavior policy, even with large a, π is not anymore constrained to the behavior distribution, whereas at other "critical" states, where T. is narrow, a large enough a will constrain I rT(|s) to stay close to (-)ss. We call this Reweighting Distribution constraints to Support (ReDS).
0,
4.1 Instantiating the Principle Behind ReDS
One option is to reweight n to T= and enforce distribution constraint D(T,, TT*) between π and Te However, this is problematic because the nH would typically be estimated by using importance weighting or by fitting parametric model, and prior work has shown that errors in estimating the behavior policy [43, 20] using only one action sample often get propagated and lead to poor downstream performance. For CQL, this issue might be especially severe if we push up the Q-values under "e, because then these errors might lead to severe Q-value over-estimation.
A bbsrract idea of CQL (ReDS). Instead, we devise an alternative formulation for ReDS that modifies the learned policy to π²5 such that applying a distribution constraint on this modified policy imposes support constraint. Thus, with CQL, now instead push down the Q-values under ""* We define -H as a mixture distribution of the learned policy ा and a reweighted version of the behavior policy follows:
5

---

CaL, Distributional Constraint
COL (ReDS), Support Constraint
How an acdiore madifee?
Palay Carely
N Desah
Wyy
Actinn Space
Bed Actions
Foor ud pny apportane
Resuling O G-Aanana
Dencf
OWE
Acton
Aden
food
Put
Ealarioo Pd Scppat
Figure Comparison between support and distributional constraints: Left: CQL pushes down the
O-function under the policy ग, while pushing up the function under the behavior policy æ5. This means that the Q-values for bad actions can go up. Right: In contrast, ReDS re-weights the data distribution to push down the values of bad actions, alleviating this shortcoming.
x" (-1ss 2 --
9 (*(+ss))
(6)
where g() is monotonically decreasing function. We will demonstrate how pushing down the Q-values under n= modifies CQL to enable support constraint while reusing existing components of CQL that impose a distribution constraint. As shown in Figure 3, the second term in Equation increases the probability of actions that are likely under the behavior policy, but are less likely under the learned policy (due to g being decreasing function). We will show Lemma 4.1 that utilizing Tee in CQL enforces a support constraint on æ. Thus, the learned policy can be further away from Tg, allowing ' assign more probability to good actions that are within the behavior policy support, even if they have lower probabilities under T.. Section 4.2 illustrates theoretically why pushing down the Q-values under Eq. approximates a support constraint terms of how it modifies the resulting Q-values. For an illustration, please see Figure 3.
How should we pick g in practice? Since we wish to Te as replacement for in the minimization term in the CQL regularizer (Equation 2), we aim to understand how to design the re- re-weighting g in practice. Since specifically CQL enforees distribution constraint by maximizing the Q-value on all actions sampled from the behavior policy тa, our choice of g should aim to counter this effect by instead minimizing the Q-value on "bad" actions within the support of the behavior policy. Equation 6 quantifies the notion of these "bad" actions using a monontonically decreasing function g (als)) of the policy probability. to be function of the advantage estimate: (s a) == Qo(s, a) Ea-z Ea-r|Qu(s,, a)), that the policy is seeking to maximize. In fact, if entropy regularization
practice, we find it convenient to define g
utilized for training the policy (akin to most offline RL algorithms), the density of an action under policy is directly proportional to ex exponentiated advantages, i.e., I (a)s)) (s) x exp(Ao(s, a)). Hence, we choose g(x) = 1/x, such that g ((exp(A(s ((s, a))) = exp(-A(s, a)) (a decreasing function).
For the rest, we approximate the product distribution x <[ass) g(T(als)) by fitting parametric function approximator Pe (als). Since (als) is being trained to approximate a re-weighted version of the behavior policy, we fit Pv by minimizing using a weighted maximum log -igleeildoood objective, as shown in prior work [44, 43]. The concrete form for our objective for training Ps is shown below (r is a temperature hyperparameter typically introduced in prior work [44, 43]))
( arg max (.s-Daa=pppp1) [log Pv (als) exp ( Ao (s..a)) Pe
(7)
The crucial difference between this objective and standard advantage-weightee updates is the dif- ference of the sign. While algorithms such AWR [43] aim find action that attains a high advantage while being close to the behavior policy, and hence, uses a positive advantage, we utilize the negative advantage to mine for poor actions that are still quite likely under the behavior policy.
The final objective for the Q-function combines the regularizer in Eq. 8 with a standard TD objective:

---

p) 2 (s~D,ams
1 E [Qo(s,
E s~Daar
(Q0(s,2))
E s,ad
(Q, a)]
(8)
1 min JQ(9) R(0: By -(oas
(9)
4.2 Theoretical Analysis of CQL (ReDS)
Next, we analyze CQL (ReDS), showing how learning using the regularizer in Eq. modifies the Q-values and justifies our choice of the distribution p in the previous section.
Lemma 4.1 starestate change of Q-values... Let g(als) be a shorthand for g(als) - g (T =(ass)). In the tabular setting, the Q-function obtained after one iteration of objective in Eq. 9 is given by:
π ( s) + अ 1) () 2 (a
Qo a) := : 1 Q(s,a)
(10)
2 2
where Ba Q(s, a) the Bellman backup operator applied a delayed target Q-network.
Eq. 10 illustrates why the modified regularizer in Eq. 8 leads "soft" support constraint whose strength is modulated per-state. Since g is monotonically decreasing function of и, for state-action pairs where x (a|s) has high values, g (a|s) is low and therefore the Q-value Qss, a) for such state- action pairs are underestimated less. Vice versa, for state-action pairs where πT m[lss) attains low values, g(ass) high to counter-acts the low (a 8) values. Also, since T ( i) appears in the denominator, for out-of-support actions, where a (a for the Q values to well defined. An illustration of this idea is shown in Figure 9. We can use this insight to further derive the closed-form objective optimized by ReDS.
0, π (aas) must also assign 0 probability to the actions
Lemma 4.2 (CQL (ReDS) objective.). ).. Assume that for all policies a E II, V(s, a), T(als) > O. Then, CQL (ReDS) solves the following optimization problem:
max [(z) #Ell
a
E g-
D(T,TT))s)
lø ((aas)
> on] (11)
2(1
a~
((T)) corresponds to the empirical return of the learned policy, i.e., the return of the policy under the learned Q-function. The objective in Lemma 4.3 can be intuitively interpreted as follows: The first term, DTT, 71 (s)( is a standard distribution constraint, also present in naïve CQL, and it aims to penalize the learned policy æ if it deviates too far away from T3. ReDS adds an additional second term that effectively encourages
be "sharp" within the support of the behavior policy (as is monotonically decreasing), enabling ı to potentially put mass on actions that lead to a high J(T).
Specifically, this second term allows us control the strength of the distribution constraint per state: at states where the support of the policy is narrow, i.e., the volume of actions such that (a)s) 0 is small (say, only a single action), the penalty in Equation 51 reverts to a standard distributional constraint by penalizing divergence from the behavioral policy via D(T,, क (s) the second term cannot be minimized. At states where the policy T3 is broad, the second term counteracts the effect of the distributional constraint within the support the behavior policy, by enabling π to concentrate its density on only good actions within the support of Ta with the same multiplier a. Thus even when we need to set a to be large to stay close to (-|s) certain states (c.g.. in narrow hallways in the example in Sec. 3.1), D(T, TT) (s) is not heavily constrained at other states.
In fact, we formalize this intuition below show that for the best possible value of the hyperparame- ters appearing in the training objective for CQL (ReDS) (Equation 51), CQL (ReDS) guaranteed to outperform the best-tuned version of CQL for any offline RI problem. A proof is in Appendix C.
Lemma 4.3 (CQL (RcDS) formal guarantec). We will add the following guarantee to show that the policy learned by ReDS for the best possible value of T (Equation 51) and in CQL (Equation 3) outperforms the best CQL policy. That is, formally we show:
max 2 max J
(12)
0,7
5 Experimental Evaluation
The goal of our experiments is to understand how CQL (RcDS) compares to distributional constraint methods when learning from heteroskedastic offline datasets. In order to perform our experiments,
7

---

we construct new heteroskedastic datasets that pose challenges representative of what we would expect to see in real-world problems. We first introduce tasks and heteroskedastic datasets that we evaluate on, and then present our results compared to prior state-of-the-an methods. We also evaluate ReDS on some of the standard D4RL [13] datasets which are not heteroskedastic in and find that the addition of ReDS, as expected, does not help, or hurt on those tasks.
5.1
Comparison on the D4RL Benchmark
Dataset
BC
10%BC
AWAC
Onestop
TD3+BC
COMBO
CQL
IQr.
Ours
halfcheetah- medium- replay
36.6
40.6
36.6
40.5
38.1
44.6
55.1
45.5
44.2
52.3
hopper- sedee--mddii---pllaaa
18.1
75.9
82.7
37.2
97.5
60.9
89.5
95.0
94.7
101.5
26.0
62.5
66.6
27.0
49.5
8188
56.0
77.2
73.9
85.0
halfchoctah- -xpetm--xxprtt
55.2
92.9
86.8
42.8
93.4
90.7
90.0
91.6
86.7
89.5
hopper-
52.5
110.9
107.6
55.8
103.3
98.0
111.
105.4
91.5
110.0
107.5
109.0
108.1
74.5
113.0
110.1
103.3
108.8
109.6
112.0
locomotion total
295.9
491.8
488.4
277.8
494.8
486.1
505
523.5
500.6
550.3
Table I: Performance comparison the D4RL benchmark. (Top 2 bolded)
Heteroskedastic data is likely to exist in real-world problems such as driving and manipulation, where datasets are collected by multiple policies that agree and disagree at different states. While standard benchmarks (D4RL [13] and RLUnplugged [21]) include offline datasets generated by mixture policies (e.g. the "medium-expert"" generated by two policies with different performance), these policies are trained via RL methods (SAC) that constrain the entropy of the action distribution at each state to be uniform. To measure heteroskedasticity, we utilize an approximation to Catt' the standard deviation in the value of D(T,, as) across states the dataset, using fixed policy obtained by running CQL. We didn't 't use Cän directly, as it is challenging to compute in continuous spaces, In Table 3, the standard deviation is lower for the D4RL antmaze datasets, corroborating our intuition that these datasets are significantly less heteroskedastic.
5.2 Comparisons on Heteroskedastic datasets
Datasct
std
max
Heteroskedastic datasets. To stress-test our method and prior distribution constraint approaches, we collected new datasets for the medium and large mazes used in the antmaze navigation tasks from D4RL: noisy datasets,
noisy (Ours)
18
253
biased (Ours)
31
diverse (D4RL)
11
play (D4RL)
13
where the behavior policy action variance differs in differ- ent regions of the maze, representative of user variability in navigation, and biased datasets, where the behavior policy admits a systematic bias towards certain behaviors in different regions of the maze, representative of bias to- wards certain routes in navigation problems, Table 3 shows that these datasets are significantly more heteroskedastic to the D4RL datasets.
Table 3:
The new antmaze datasets (Ours) are significantly more het- croskedastic than the standard D4RL datascts. We measure hetcroskedastic- ity using the std and max of D(T,, (s) across states in the offlinc datasct.
Using these more heteroskedastic datasets, compare CQL (ReDS) with CQL and IQL [27], recent popular methods, and two prior methods, BEAR [30] and EDAC [3), that also enforce support constraints. For each algorithm, including ours, we utilize hyperparameters directly from the counterpart tasks in D4RL. Due to the lack of an effective method for offline policy selection (see Fu et al. [14]), we utilize oracle checkpoint selection for every method. We compute the mean and standard deviation across 3 seeds. Table 2 shows that the largest gap between CQL (ReDS) and prior methods is on noisy datasets, which are particularly more heteroskedastic (Table 3).
We also compare CQL (ReDS) with recent offline RL algorithms on D4RL, including DT [8]. AWAC [42]. onestep RL [5]. TD3+BC [16] and COMBO [57]. Table shows that CQL (ReDS) obtains similar performance as existing distributional constraint methods and outperforms BC-based baselines. This is expected given that the D4RL datasets exhibit significantly smaller heteroscedasticc ity. as previously explained. Also, a large fraction of the datasets is trajectories with high returns. BC using the top 10% trajectories with the highest episode returns already has strong performance. The previous results compares CQL (ReDS) to baselines in tasks where the MDP states are low- dimensional vectors. Next, we study vision-based robotic manipulation lasks.
Visual robotic manipulation. We consider two types of manipulation tasks, In the "Pick & Place" task, the algorithm controls WidowX robot grasp object and place it into a tray located at test
8

---

Task & Dataset
EDAC
BEAR
CQL
IQL
INAC
RW AW
EQL
SQL
XQL-C
Ours
medium-noisy
0
0
55
44
0
0.0
0.7
4.3
73
medium bediumm-baseed
0
0
73
48
6.5
8.0
11.7
74
page--oisy
0
42
39
10
7.1
2.9
11.3
53
large-biased
0
50
41
8.5
0.5
7.3
45
Table 2: CQL (ReDS) outperforms prior offline RL methods including methods (IQL, XQL-C), and prior support constraint methods (BEAR, EDAC, SQL,EQL RW & AW) three out of four scenarios when learning from heteroskedastic data in the antmaze task. The improvement over prior methods is larger when learning from the noisy datasets, which Table 3, compared to biased datasets.
heteroskedactic, in
Start
Goal
Less Variabiltty
Vadabilltx
Figure 4: Examples rollouts in the heteroskedastic bin-sort data. In this task, offiine RL method must objects in front of into bins with dataset that non-uniform eoverage different states, using visual input. the first half the trajuctory, states exhibit more narrow action distribution but the second hall admits a more uniform action distribution.
location, directly from raw 128x 128 x3 images and sparse 0/1 reward signal. The dataset consists of behavior from suboptimal grasping and placing policies, and the positions of the tray in the offline dataset very rarely match the target test location. The placing policies exhibit significant variability, implying these datasets are heteroskedastic under our definition. We also consider "Bin Sort" task (see Figure 4), where WidowX robot is controlled to sort two objects into two separate bins. Here, heteroskedacity is introduced when sorting objects into the desirable bins. Similar to the Pick Place task, the placing policy exhibits significant variability, showing object placed in the incorrect bin (e.g., recyclable trash thrown into the non-recyclable bin). However, grasping policy is more expert-like grasping the object with low variability. More details in Appendix E.
Table presents the results on these tasks. We utilize oracle pol- icy selection analogous to the antmaze experiments from Table 2. Table 4 shows that CQL (ReDS) outperforms CQL attaining success rate of about 15.1% for the visual pick and place task, whereas CQL only attains 6.5% success. While performance might appear low in an absolute sense, note that both CQL and ReDS do improve over the behavior policy, which only attains success rate of 4%. Thus offline RI. does work on this task, and utilizing ReDS in conjunction with the standard distributional constraint in CQL does result in boost in performance with this heteroskedastic dataset. For the "Bin Sorting", our method out- performs CQL by 3.5x when lcarning from more heteroskedastic datascts. This indicates the effectiveness of our method in scttings with higher heteroskedasticity.
IQM
2 Pollcies
0.75 0.90
1.05
1.20
5 Pollcies
0.45
0.60
0.75
Normalized Score
(ReDS)
Figure 5: CQL vs ReDS: IQM nor- malized score for 10 Atari games. We consider two datasct compositions.
Task
CQL
CQL (ReDS)
std D(T, (s)
max D(n, (s)
Pick & Place
6,5 - 0.4
15.1 0.4
48.7
307.4 4
Bin Sort (Easy)
31.2 +
31.4 0.3
7.9
81.6
Bin Sort (Hard)
6.1 + 0.2
23.1 1 0.7
59.6
988.3
Table 4: CQL (ReDS) vs CQL on robotic manipulation tasks. CQL (RcDS) outperforms CQL significantly when learning from more heteroskedasttic datasets, as measured by Chai the standard deviation and the maximum of D(TT, (s) across states.

---

Atari games. We collect data on 10 Atari games from multiple policies that behave differently at certain states while having similar actions otherwise. We consider case two such policies, and a harder scenario of five. We evaluate the performance of CQL (ReDS) on the Atari games using the evaluation metries from prior works [2, 34], Figure 5 shows that in both testing scenarios: with the mixture of two policies (top figure) and the mixture of five policies (bottom figure), CQL (ReDS) outperforms CQL in aggregate.
To summarize. our results indicate that incorporating CQL (RcDS) outperforms distribution con- straints with heteroskedastic datasets in a variety of domains.
Related Work
Offline Q-learning methods utilize mechanisms to prevent backing up unseen actions [39], by applying an explicit behavior constraint that forces the learned policy to be "close" to the behavior policy [23, 53, 44, 49, 53, 30, 28, 27, 52, 15], or by learning a conservative value function [33, 54, 41, 57, 56, 47, 24, 531. Most of these offline RL methods utilize a distribution constraint, explicit (e.g.. TD3+BC [151) or implicit (e.g., CQL (331), and our empirical analysis of representative algorithms from either family indicates that these methods struggle with heteroskedastic data, especially those methods that use an explicit constraint, Model-based methods (26, 56, 4, 51, 45, 38, 57] train value functions using dynamics models, which is orthogonal to our method.
Some prior works have also made a case for utilizing support constraints instead of distribution constraints, often via didactic examples [30, 29, 39], and devised algorithms that impose support constraints in theory, by utiliziin the maximum mean discrepancy metric [30] or an asymmetric
f-divergences [53] for the policy constraint [53]. Empirical results on D4RL [13] and the analysis
by Wu al. [53] suggest that support constraints are not needed, as strong distribution constraint algorithms often have strong performance. As we discussed in Sections 3.2 (Theorem 3.2 indicates that this distribution constraints may not fail when Cär is small, provided these algorithms are well-tuned.) and 4, these benchmark datasets are not heteroskedastic. as they are collected from policies that are equally wide at all states and centered on good actions (e.g., Antmaze domains in [13], control suite tasks in Gulcehre et al. [2I]) and hence, do not need to modulate the distribution constraint strength. To benchmark with hetcroskedastic data, we developed some novel tasks which may be of independent interest beyond this work, and find that method ReDS can work well here.
7 Discussion, Future Directions, and Limitations
We studicd the behavior of distribution constraint offline RL algorithms when Icarning from het- croskedastic datasets, property constraint algorithms can be highly ineffective in such settings both in theory and practice, as they fail to modulate the constraint strength per-state. We propose ReDS, method to convert distributional constraints into support-bases constraints via reweighting, and validate it in CQL. A limitation of ReDS is that it requires estimating the distribution Pe to enforee support constraint, which brings about its some additional compute overhead. Additionally, the instantiation of ReDS we develop in Section 4.1 is specific methods that utilize conservative regularizer such CQL (or related approaches like COMBO). We clarify that our main contribution in this work is an analysis of when distributional constraints fail (which we study for AWR and CQL), and developing a principle for reformulating distributional constraints to approximate support constraints via rewcighting. Devising approaches for enforcing support constraints that do not require extra machinery is a direction for future work. Understanding if support constraints are less sensitive to hyperparameters or are more amenable to model election is also direction for future work.
likely encounter in the real world. Naïve distribution
10

---

References
[1] Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pp. 22-31. JMLR.. org, 2017.
[2] Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforce- ment learning. In International Conference Machine Learning (ICML), 2020.
[3] An, G., Moon, S., Kim, J.-H. and Song, H. O. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. arXiv e-prints, art. arXiv:2110.01555. October 2021.
[4] Argenson, A. and Dulac-Arnold, G. Model-based offline planning. arXiv preprint arXiv:2008.055550 05556, 2020.
[5] Brandfonbrener, D., Whitney, F. Ranganath, R., and Bruna, J. Offline RL without off-policy evaluation. CoRR, abs/2106.08900) 2021. URL https: / arriv.iiivvi./...///aa///aaaa.a. wttpaaawwiaa.aaaaaaaeee/ www.oog//////..eoooooooooooo 2206. 08909.
[6] Cao, K., Chen, Y., Lu, J., Arechiga, N., Gaidon, A., and Ma, T. Heteroskedastic and imbalanced deep learning with adaptive regularization. arXiv preprint wrXiv:2006.15766 2020.
[7] Castro, P. S., Moitra, S., Gelada, C.,, Kumar, S., and Bellemare, M. G. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http: /tattp:///riivv www.i.trg./a....aee.......oooott hatg/abb/iee /1822.
06110.
[8] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint ar #rX.:22/06.01355, 2021.
[9] Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248-255, Icce, 2009.
[10] Ebert, F., Yang, Y., Schmeckpeper, K., Bucher, B., Georgakis, G., Daniilidis, K., Finn, C., and Levine, S. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13396, 2021.
[11] Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp. B., Qi, C. R., Zhou, Y., et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9710-9711 2021.
[12] Fu, J., Kumar, A., Soh, M., and Levine, S. Diagnosing bottlenecks in deep Q-learning algorithms. arXiv preprint arXiv: 1902. 10250, 2019.
[13] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/1 2020.
[14] Fu, J., Norouzi, M., Nachum, O., Tucker, G., ziyu wang, Novikov, A., Yang, M., Zhang, M. R., Chen, Y., Kumar, A., Paduraru, C., Levine, S., and Paine, T. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021. URL attps://openrett
[15] Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2/06.. 066606 2021.
[16] Fujimoto, S. and Gu, S. S. A minimalist approach offline reinforcement learning. CoRR, abs/2106.006860, 2021. URL https::/ //wpvv///ivvv///////// Ferggabss//11 0122100066600...111111
[17] Fujimoto, S., Meger, D., and Precup. D. Off-policy deep reinforcement learning without 2018.
exploration. arXiv preprint arXiv:
[18] Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning (ICML), pp. 1587-1596, 2018.
[19] Garcia, J. and Fernández, F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):
2015.
[20] Ghasemipour, S. K. S., Schuurmans, D., and Gu, S. S. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.

---

[21] Gulcehre, C., Wang, Z., Novikov, A., Paine, T. L., Colmenarejo, S. G., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D., Paduraru, C., et al. RI unplugged: Benchmarks for offline reinforcement learning. 2020.
[22] Haarnoja, T.. Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. Soft actor-critic algorithms and applications. Technical report, 2018.
[23] Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way off aff-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: I 997.00456, 2019.
[24] Jin, Y., Yang. Z., and Wang, Z. Is pessimism provably efficient for offline rI? arXiv preprint arXiv:20/2. 15085, 2020.
[25] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M.. Vanhoucke, V., al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673, 2018.
[26] Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05957, 2020.
[27] Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 10.06169, 2021.
[28] Kostrikov, I,, Tompson, J., Fergus, R., and Nachum, O. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050, 2021.
[29] Kumar, A. Data-driven deep reinforcement learning.
haterrkeeeeeedd/bboelee edu/blog/
2019//2205/beaae 2019. BAIR Blog.
[30] Kumar, A., Fu, J., Soh, M.. Tucker, G., and Levine, S. Stabilizing off-policy q-lcarning via bootstrapping error reduction. Advances in Neural Information Processing Systems, pp. 11761-11771, 2019.
[31] Kumar, A., Fu, J., Tucker, G., and Levine, S. Stabilizing Off-Policy Q-Learning via Bootstrap- ping Error Reduction. arXiv e-prints, art. arXiv:1906.00990 June 2019.
[32] Kumar, A., Fu, J., Tucker, G., and Levine, S. Stabiliizng off off-policy q-learning via bootstrapping error reduction. 2019. URL httt::/aaa/:ii http:::///axii.oo//aabbaaae /906. 00949.
[33] Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforce- ment learning. arXiv preprint arXiv:2006.0444 (4779, 2020.
[34] Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under- paader--rrmmttiizationn inhibits data-
efficient deep reinforcement learning. In International Conference on Learning Representations, 2021. URL httpp:://a/t http:/::////////eeeeeeeee net http:/oommmmdd00nbiiaa
[35] Kumar, A., Hong, J., Singh, A., and Levine, Should run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022. URL https://openrevier: net/forum? Pid-APdPPP AP 1..
[36] Lange, S., Gabel, T., and Riedmiller, M. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
[37] Laroche, R., Trichelair, P., and Combes, R. d. Safe policy improvement with baseline bootstrapping. arXiv preprint arXiv:1712.06924, 2017.
[38] Lee, B.-J., Lec, J., and Kim, K.-E. Representation balancing offline model- reinforcement learning. In International Conference on Learning Representations, 2021. URL https: / //openreview, meet/ooummidd-pllee 2000pi228r_Ri1....._1
[39] Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 00660000.0........ 2020.
[40] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint
2015. [41] Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., and Schuurmans, D. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.00072 2019.
[42] Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.0933 2020. URL htap:::///rxiit org/abs/ /2g///00/000....///////// 09359.
12

---

[43] Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets, arXiv preprint urXiv:2006.09:355 2020.
[44] Peng, X. B., Kumar. A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off ooflpo reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[45]
Rafailov, R., Yu, T., Rajeswaran, A., and Finn, C. Offline reinforcement learning from images with latent space models. Learning for Decision Making and Control (L4DC), 2021.
[46] Rashidinejad, P. Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv:2 100.2221,,,, 2021.
[47] Rezacifar, S., Dadashi, R., Vieillard, N., Hussenot, L., Bachem, O., Pietquin, O., and Geist, M. Offline reinforcement learning as anti explorixoor arXiv preprint 112xix:2106064// 2021.
[48] Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, Trust region policy optimization. In International conference machine learning, pp. 1889-1897, 2015.
[49] Siegel, N. Y., Springenberg, J. T., Berkenkamp, F., Abdolmaleki, A., Neunert, M., Lampe, T., Hafner, R., and Riedmiller, M. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint FrXxv:2002.08396 2020.
[50] Singh, A., Yu, A., Yang, J., Zhang, J., Kumar, A., and Levine, S. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
[51] Swazinna, P., Udluft, S., and Runkler, T. Overcoming model bias for robust offline deep reinforcement learning. arXiv preprint trXiv:2008.05533 2020.
[52] Wang, Z., Novikov, A., Zolna, K., Springenberg, T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., et al. Critic regularized regression. arXiv preprint arXiv:2006. 15134, 2020.
[53] Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 2019.
[54] Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[55] Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L. Don't t change the algorithm, change the data: Exploratory data for oflline reinforcement learning. arXiv preprint arXiv:2201.13422 2022.
[56] Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
[57] Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2 2021.
13


Reviews:
[{"review_id": "kcERFkZDfW", "reviewer": "Reviewer_XMxS", "summary": "The paper focuses on heteroskedastic datasets, where the distribution of actions may not be uniform in certain states but close to uniform in other states. The authors propose the ReDS method, which re-weights the actions to penalize \"bad\" actions that perform poorly on the dataset but have a higher probability, while encouraging \"good\" actions with lower probabilities in the dataset's support. ReDS outperforms previous baseline methods in multiple experimental scenarios.", "questions": "1. In Fig.1, I wonder if the datasets cover the whole room B or only a point shown in the figure. If the datasets cover the whole room, AWR and CQL should be able to learn to exit room B, since other actions lead to lower returns.\n2. As Line 326-327 stated, the authors have evaluated ReDS on some of the standard D4RL datasets which are not heteroskedastic and find that the addition of ReDS, as expected, does not help, or hurt on those tasks. Why does ReDS hurt the performance on some of the standard D4RL datasets? By Theorem 1, ReDS can at least have better performance.\n3. Why do the experimental domains contain Atari?", "limitations": "The authors generated new datasets and did not conduct a universal analysis on whether heteroskedastic properties exist in some datasets. It would be beneficial to first employ a suitable method to effectively verify the presence of heteroskedastic properties in commonly used datasets before applying ReDS.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The majority of the paper is written in a clear and easy-to-follow manner. The research problem addressed in this paper is indeed relevant to real-world scenarios, and the authors provide didactic examples to illustrate the potential issues with previous methods. The paper offers a detailed analysis and proposes a concise and intuitive plug-in solution called ReDS, while also demonstrating its effectiveness. The experimental settings are diverse, covering multiple domains.", "weaknesses": "ReDS should apply in broader scenarios beyond the specific setting of heteroskedastic datasets. Another point is the choice of experimental settings, as most of them involve manually constructed datasets to satisfy heteroskedastic conditions, rather than utilizing more intuitive scenarios such as autonomous driving or validating whether some existing datasets are really heteroskedastic. Thus the significance is somewhat limited.\n\n\n\n\nMinor issues:\n\nThe figures in the main text, such as Figures 1 and 2, rely heavily on the captions. Adding more conclusive statements in the main text could enhance consistency and clarity.\n\nTable 4 --> Table 2?\n\nLine 326 \"some of the standard D4RL datasets which are not heteroskedastic in and find\" --> extra \"in\"\n\nLine 419 \"WSome prior works ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and positive assessment of our work.  We address your concerns below and will update the paper to clarify each of the questions. **Please let us know if your questions are resolved, and if so, we would appreciate it if you are willing to upgrade your score.  We are happy to discuss further if you have any remaining questions.**\n\n***ReDS should apply in broader scenarios beyond the specific setting of heteroskedastic datasets*** \n\nWhen learning from datasets that are non-heteroskedastic, ReDS perform similarly to distribution constraint algorithms, which indicates that ReDS can be used in other scenarios that distribution constraint algorithms can be used in. We don’t believe that this is a weakness of the method – the aim is not to beat all prior methods in all settings but to address a specific challenge in offline RL. We believe that the paper is scoped carefully around this claim, but we would be happy to revise it if you believe this is unclear.\n\n**The choice of experimental settings, … the significance is somewhat limited.** \n\nWhile we agree that the choice of experimental settings that we study involves specifically constructed heteroskedastic datasets, we believe that this is an important first step in verifying if developing support constraint methods can help in scenarios with heteroskedasticity. The nature of this contribution is akin to prior papers such as Hong et al. ICLR 2023, which proposes methods for re-weighting offline data to deal with imbalanced datasets, but evaluates only on manually constructed datasets that add noise or mix two existing D4RL datasets. We believe that this sort of evaluation is still of value to the community as it provides a stress test for existing algorithms and provides a proof of concept for new ideas. Many of the challenges present in real-world RL problems are not well reflected in existing benchmarks such as D4RL. That being said, we agree with you that the next step in this line of work is to validate our methods on real-world datasets in autonomous driving or robotics, which are heteroskedastic. We discuss some of these scenarios intuitively in Section 3 and in the introduction.  Finally, we also remark that our contribution in this paper is not just a novel algorithm, but also an analysis of why heteroskedasticity is hard for offline RL methods and understanding when support constraint methods can help. We believe that this in and of itself is valuable to the community developing offline RL algorithms.\n\n***I wonder if the datasets cover the whole room B or only a point shown in the figure … AWR and CQL should be able to learn to exit room B since other actions lead to lower returns.***\n\nWe clarify that the dataset consists of several sampled state-action transitions inside the wide room. Note that not every state-and-action pair from within this room is observed in the dataset, but rather only a subset of these samples are observed. We only highlight one single point B in Figure 1(a) for the compactness of visualization. \n\nMore concretely, the dataset consists of sampled state-action-reward-next state transitions that are a subset of the cross-product of the state and action spaces. The samples are not uniformly distributed and are instead drawn from **a skewed action distribution which crucially attempts to keep the agent within the room**, which hinders the ability of algorithms such as AWR and CQL from learning how to exit wide rooms because a strong distributional constraint will make the policy closely match this skewed behavior distribution. This can be seen in the left part of Figure one, where in the narrow regions, the data consists of state and action pairs that uniformly move the agent towards the goal but in the wide rooms, the data consists of state and action pairs that skews the agent towards the edges of the room. Therefore, even if more than one transition is sampled from inside the wide room, the learned policy obtained from distributional constraint algorithms with a strong constraint will have the affinity to stay within the room and fail at the task. \n\n***Authors have evaluated ReDS on some of the standard D4RL datasets. Why does ReDS hurt the performance of some of the standard D4RL datasets?*** \n\nObserve in Table 1 that ReDS performs similarly or better than CQL for all tasks of the D4RL datasets presented except for the half cheetah medium expert task. We believe that this difference is small (only 2.1 performance points) and hence not statistically significant on this task. \n\n***Why do the experimental domains contain Atari?***\n\nWe chose Atari as a representative domain with discrete action spaces to study some of the heteroskedasticity challenges. Many offline RL papers (Badia et al. 2020, Kumar et al 2022) that we are aware of study the Atari domain using the offline datasets generated by Agarwal et al. ICML 2020. Atari also requires learnings from raw high dimensional pixel space, which adds to the breadth of the experiments in the paper since the D4RL  benchmarks only require learning from low dimensional state space.\n"}, {"review_id": "uMBGummMOL", "reviewer": "Reviewer_uYnF", "summary": "The paper introduces ReDS, a novel offline RL method designed to handle heteroskedastic datasets. ReDS incorporates support constraints by reweighting the data distribution based on conservative Q-learning (CQL). This allows the learned policy to deviate from the behavior policy within its support while maximizing long-term return.", "questions": "The main concern relates to the points mentioned in the weaknesses section. Additionally, there is another question that requires clarification:\n\nQ: Is the proposed method plug-and-play? If so, why not integrate it with an updated or foundational algorithm (e.g., SAC+vanilla distribution constraint)?", "limitations": "The paper discusses the limitations. And, these limitations seems not appear to require immediate attention.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The addressed problem of a more fine-grained support constraint in offline RL research is crucial.\n2. The paper includes thorough discussions about the proposed method.\n3. The introduction of the method, which utilizes reweighting distribution constraints to achieve a fine-grained support constraint, is clear and reasonable. The explanation in Figure 3 is particularly intuitive.\n4. The method demonstrates significant performance advantages, especially when applied to heterogeneous datasets.", "weaknesses": "1. The didactic example does not convincingly support the claims. For example, it does not convincingly demonstrate the characteristics of heterogeneous datasets in key states (such as the bottom-left corner of the map) and their impact on the CQL and REDS algorithms. Additionally, the results presented in Figure 1 show that the baseline algorithms AWR and CQL fail, while REDS does not. However, has this demonstration accounted for the potential impact of statistical errors?\n2. The baseline algorithm (CQL) seems somewhat outdated. It would be necessary to discuss and compare with more advanced methods that satisfy the support constraint, such as InAC[1].\n3. There are several spelling errors throughout the paper. A careful proofreading is necessary. For instance, the capitalized \"w\" in \"With\" in the title should be lowercase, and the occurrence of \"WSome\" in line 419 is incorrect.\n\n[1] Xiao, Chenjun, et al. \"The in-sample softmax for offline reinforcement learning.\" arXiv preprint arXiv:2302.14372 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and positive assessment of our work.  To address your concerns, we add 2 comparisons to the support-constraint methods InAC and Hong et al 2023. We also explain how we took statistical errors into account when we compute the statistics for the didactic example.\n\n**Please let us know if our answers resolve your concerns, and if so, we would truly appreciate it if you are willing to upgrade your assessment. We are happy to discuss any further questions.**\n___\n\n***Additional baselines; CQL is outdated***\n\nWe add 2 additional baselines and compare our method ReDS to InAC and Hong et al. 2023 on the heteroskedastic AntMaze datasets.\n\n**InAC [Xiao et al. 2023]** We find this approach fails to attain a non-zero performance on the heteroskedastic AntMaze tasks.  We tuned InAC by sweeping the learning rate and temperature $\\tau$ over the following values:\n\n|   Hyperparameters   |       Values       |\n|:-------------------:|:-----------------:|\n|        Tau          | 0.5, 0.33, 0.1, 0.01 |\n|   Learning Rate     | 1e-3, 3e-4, 1e-4, 3e-5 |\n\nFor each sweep value, we are unable to get non-zero performance. We also ran this method on the D4RL AntMaze tasks, unable to attain non-zero performance even on these standard tasks. Note, our findings don’t contradict the results in InAC because the paper doesn’t study them. This indicates that ReDS outperforms InAC on the heteroskedastic AntMaze tasks.\n\n**Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting [Hong et al. 2023]**: We see that performance for the AntMaze heteroskedastic tasks using the hyperparameters the authors used for the AntMaze results in their paper, is non-zero but lower than the performance of ReDS. We tried two variations that the authors recommended for weighting: Reward Weighting (RW) and Advantage Weighting (AW). We compared against the best-performing variation for each dataset. For these same hyperparameters, we verify performance on the D4RL AntMaze tasks, which match the author’s results.\n\nFor both algorithms, we used the author’s implementation, averaging over 4 seeds and using the max performance after 1M training steps.\n\nAdditionally, in the paper, we compare against two prior methods, BEAR and EDAC, which enforce support constraints, and find that ReDS outperforms all of these prior methods. We are happy to add more comparisons if the reviewer has any suggestions for the final version of the paper.\n\nA full comparison can be found below:\n| Task & Dataset |     EDAC     |    BEAR     |    CQL    |    IQL    |    INAC    | RW and AW | ReDS [Ours] |\n|:--------------:|:------------:|:-----------:|:---------:|:---------:|:----------:|:---------:|:---------------:|\n|  medium-noisy  |      0       |      0      |    55     |    44     |     0      |     5     |      **73**     |\n| medium-biased  |      0       |      0      |  **73**   |    48     |     0      |     0     |      **74**     |\n|  large-noisy   |      0       |      0      |    42     |    39     |     0      |    10     |      **53**     |\n| large-biased   |      0       |      0      |  **50**   |    41     |     0      |     8     |        45       |\n\n***Didactic example doesn’t convincingly demonstrate the characteristics of heterogeneous datasets in key states.***\nThe didactic gridworld consists of narrow hallways and wide rooms that periodically repeat throughout the path the policy needs to take to successfully solve the task. While in Figure 1(a), we visualize the action distributions at locations A and B only, a similar variability arises in behaviors in the bottom-left corner of the map. In regards to why AWR and CQL get stuck at the bottom left corner, this is because while these policies are still able to escape the first two wide rooms along the path, they are unable to do so the third time at the bottom of the maze. This is not due to the particular nature of that location, but the general periodic pattern of the behavior distribution. We are happy to edit this visualization for further clarity if you have any suggestions. \n\n**However, has this demonstration accounted for the potential impact of statistical errors?**\nYes, we do account for statistical errors: **(1)** the offline dataset provided to all algorithms is of finite size, indicating that these algorithms including ReDS must still learn in the face of statistical error, and  **(2)** For visualizing how the policies behave upon evaluation, we directly plot the discounted, long-term state-action visitation following the visualizations in Fu et al. 2019, rather than plotting the empirical rollout of each algorithm, so our visualizations do take into account the impact of statistical errors. Note that the (discounted) visitation distribution of a policy is computed using T steps of forward propagation. \n\n***Is the proposed method plug-and-play?*** \nAs stated in Lines 210-224,  the principle of combining re-weighting with distributional constraints is plug-and-play, applicable to any distributional constraint algorithm including SAC + distributional constraint methods. However, the instantiation of ReDS we develop in Section 4.1 is specific to methods that utilize a conservative regularizer such as CQL (or related approaches like COMBO). This is because our current instantiation of ReDS modifies the distribution for push-up and push-down in the CQL regularizer (Figure 3), but these loss terms are not present in policy constraints.  We clarify that our main contribution in this work is an analysis of when distributional constraints fail (which we study for AWR and CQL), and developing a principle for reformulating distributional constraints to approximate support constraints via reweighting. We validate this through CQL and leave as future work the application of this framework to other offline RL algorithms. \n\n***Careful proofreading is necessary*** Thank you for the suggestion! We will thoroughly proofread to ensure the final paper is polished.\n"}, {"review_id": "5etyKhoh8J", "reviewer": "Reviewer_5pSd", "summary": "The paper has identified the presence of heteroskedasticity in realistic offline RL datasets, which negatively impacts the performance of existing offline RL methods that rely on distributional constraints. To tackle this issue, the authors introduce a new approach called ReDS, which transforms distributional constraints into support-based constraints.  Several novel heteroskedastic datasets were introduced in order to showcase the superior performance of ReDS compared to existing offline RL methods.\n\nTo practically achieve the support constraint, the fundamental change based on CQL is to add a new penalty term $\\pi_{re}$, which can be obtained via \n$\\arg \\max_{\\rho_\\psi} \\mathbb{E}_{s \\sim \\mathcal{D}, a \\sim \\pi_\\beta(\\cdot \\mid s)}[\\log \\rho_\\psi(a \\mid s) \\cdot \\exp (-A_\\theta(s, a) / \\tau)]$.", "questions": "1. Could the policy $\\pi$ in Deﬁnition 3.1 be any policy? What if it is an arbitrary worst policy, i.e., random policy? I do not see clear evidence that $C_{diff}$ is big enough when $D(\\pi, \\pi_{\\beta})$ is arbitrarily large in both $s_1$ and $s_2$. On the other hand, should $s_1, s_2 \\sim d^{\\pi}$ without any other condition? What if they are all sampled from the same place, i.e., the narrow hallways in the didactic example? \n2. Any detail about how to compute the std in Chapter 4.2?\n3. How well does ReDS perform over non-heteroskedastic datasets (random, medium, expert)? Is there any limitation to on non-heteroskedastic datasets? ", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Well-written, with high-quality examples, and technically sound. \n2. Extensive evaluations over many different tasks. ", "weaknesses": "1. The design of the reweighted version of the behavior policy in Eq. 6 is heuristic. I can not see a clear clue why the coefficient is designed as 1/2 and 1/2. \n2. The experiments in the noisy and biased version of Antmaze tasks seem contrived. \n3. Inconsistent of the differential concentrability between theory and real-world practical datasets (std). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback and positive assessment of our work.  We address your concerns below and will update the paper to clarify each of the questions. Please let us know if your questions are resolved, we are happy to discuss further if any questions are remaining.\n\n***Inconsistent of the differential concentrability between theory and real-world practical datasets (std).***\n\nThe intuitive explanation for what makes a dataset heteroskedastic -- namely, that the variability in the policy is different in different states -- is provided primarily to build intuition. Unfortunately, this notion by itself does not make a great formal definition, because it does not capture the count of states $n(s)$ that shows up in a safe-policy improvement bound (the constraint in Equation 5). Therefore, in our analysis, we use the somewhat more complex notion of differential concentrability, which captures a similar intuition (we make the mathematical connection between the intuition of heteroskedasticity and our definition precise below) but provides us with the foundation on which to build our theoretical analysis. \n\nTo better understand the relationship between the variance of the action distribution and our Definition 3.1, consider a simpler scenario where we remove the counts $n(s)$ from the expression of differential concentrability and set $\\pi$ in $C^\\pi_\\text{diff}$ to be the uniform distribution over actions. Then, we can show that the value of $C^\\pi_\\text{diff}$ is **exactly** equal to twice the variance of entropy of the dataset action distribution across states, which matches with the metric (variance = square of the standard deviation) we measure in our experiments. \n\nOf course, we cannot exclude the counts of states $n(s)$ for technical accuracy, however,  it should be noted that in high-dimensional state space tasks, such as those examined in our experiments, each state in the offline dataset is likely to be unique, thus validating the assumption that $n(s) = 1$. This means that measuring the variance in the entropy of the action distribution is an accurate estimate of differential concentrability in our experiments. \n\n***Could the policy $\\pi$  in Deﬁnition 3.1 be any policy?***\n\nWhile the choice of the policy $\\pi$ in the _expression for_  differential concentrability can be any policy, we would like to clarify that we utilize the value of $C^\\pi_\\text{diff}$ only for the policy learned via distributional constraint methods to define how heteroskedastic a dataset is. This is akin to the definition of standard concentrability which, in principle, can be computed for any policy but only its value at the optimal policy is a useful measure of the hardness of the offline RL problem.\n\n**I do not see clear evidence that $C^\\pi_\\text{diff}$ is big enough when $D(\\pi, \\pi_\\beta)$ is arbitrarily large in both $s_1$  and $s_2$. **\n\nWe would like to clarify that to the best of our knowledge, we did not claim this in the paper. In fact, in lines 163-168, we claim otherwise: we say that $C^\\pi_\\text{diff}$ can be _small_ even when $D(\\pi, \\pi_\\beta)$ is large enough because it might take similar values at different states. Does this clarify your question?\n\n**Should $s_1, s_2 \\sim d^\\pi$  without any other condition? What if they are all sampled from the same place?** \n\nPlease note that $d^\\pi$ refers to the state occupancy distribution of the policy that is attempting to maximize the conservative offline RL objective (Equation 1). Intuitively, this means that sampling $s_1$ and $s_2$ from $d^\\pi$ should be enough because $d^\\pi$ is, in effect, close to $d^{\\pi_\\beta}$. Thus, when the dataset behavior distribution has a lot of variability across states, this variability would also be reflected on states sampled from $d^\\pi$. \n\n***Any detail about how to compute the std in Chapter 4.2?*** \n\nWe apologize, but we are not sure which table this question is referring to. Could you please provide a clarification? If you are referring to Table 2, the standard deviation is computed by first recording the value of KL divergence between a model of the behavior policy (trained by maximizing the log-likelihood of the dataset) and the policy learned by CQL for the best hyperparameter $\\alpha$ for all states in the dataset, and then, computing the standard deviation in this divergence value across all states in the dataset. \n\n***Why the coefficient is designed as $\\frac{1}{2}$ and $\\frac{1}{2}$.***\n\nSetting this coefficient to a value of $\\frac{1}{2}$ was the initial choice for our experiments. We found this choice was convenient: it worked well in practice and circumvented the need for introducing a new hyperparameter. That said, future work could aim to study the sensitivity of ReDS to other values of this coefficient, which we believe will only improve the performance of this approach.\n\n***How well does ReDS perform over non-heteroskedastic datasets (random, medium, expert)? Is there any limitation to non-heteroskedastic datasets?*** \n\nWe are running these experiments and will get back to you before the end of the author-reviewer discussion period. However, note that we do already have experiments on some standard and commonly used non-heteroskedastic benchmark tasks in Table 1."}, {"review_id": "XHdZokUvsU", "reviewer": "Reviewer_huak", "summary": "This paper introduces ReDS for offline reinforcement learning (RL) where the variability of demonstrated behaviours changes non-uniformly across the state space. Unlike prior offline RL methods, e.g., CQL and AWR, that directly constrain the learning policy by distribution matching with the behaviour policy, ReDS presents the task as support matching and implements it by reweighting the data distribution in CQL. Both theoretical analysis and experimental results are provided to demonstrate the effectiveness of the resulting algorithm, CQL (ReDS).", "questions": "- Some of the important domains / tasks are missing from the paper. As a standard benchmark for offline RL, the medium-level locomotion tasks, the adroit tasks, and the original antmaze tasks are missing from the paper. Specifically, the original antmaze itself is already heteroskedastic in my opinion, as for most of the states, the ant only needs to go straight and only occasionally it will make turns. How does CQL (ReDS) compare with the well-benchmarked results on these tasks?\n- How to apply ReDS to other distribution-constraint-based methods, including AWR, TD3+BC, etc. If it can be applied, how does it improve them and how does it compare with the aforementioned related works? If ReDS cannot be applied to other algorithms, what is its limitation?", "limitations": "The main limitation of the paper is with its novelty. The idea of reweighting samples and casting the distribution-constrained offline RL, i.e., distribution matching problem, as a support matching problem, has been discussed by prior methods. Also, many important and well-benchmarked domains and tasks are missing in the experiments, which makes it less clear about the performance of the method. Overall, I do think this is an interesting paper, but given the current limitations and issues, it might not be ready for publication yet.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The paper is well-motivated, theoretically sound, and supported by extensive experiments.", "weaknesses": "There are several main weaknesses of the paper.\n\n- The main issue of the paper is with its novelty. The idea of reweighting samples by state-dependent weights is not novel. Specifically, [1] has proposed to reweight the samples by trajectory returns / advantages and demonstrated that such a scheme significantly improves the performance of multiple offline RL algorithms, including CQL, IQL, TD3+BC, and BC. Similarly, [2] has proposed to cast the offline RL constraints as a support matching problem, rather than distribution matching, by reweighting the data samples by normalised trajectory returns. The proposed method in [2] has also been demonstrated to improve a wide range of offline RL methods, ranging from CQL, IQL, TD3+BC to CRR. Nevertheless, despite their importance,  highly similar motivation, problem formulation, and solutions, both [1] and [2] are missing from the related works. \n- Although the paper has conducted extensive experiments across different domains, it lacks several important studies. Similar methods [1, 2] have already demonstrated that support matching methods for offline RL can be algorithm-agnostic, i.e., it can be used to boost the performance of different algorithms, ranging from CQL to IQL, TD3+BC, and etc. However, this paper only discusses CQL (ReDS). From the perspective of algorithmic design, CQL, TD3+BC, and AWR (discussed by the authors in line 127) / CRR / IQL (based on the AWR for policy improvement) are all distribution-constraint algorithms. It is important to discuss how ReDS could improve the aforementioned algorithms and how ReDS differs from [1, 2] in its performance.\n\nReferences: \n\n[1] Hong, Z. W., Agrawal, P., des Combes, R. T., & Laroche, R. (2023, February). Harnessing mixed offline reinforcement learning datasets via trajectory weighting. In The Eleventh International Conference on Learning Representations.\n\n[2] Yue, Y., Kang, B., Ma, X., Xu, Z., Huang, G., & Yan, S. (2022). Boosting Offline Reinforcement Learning via Data Rebalancing. arXiv preprint arXiv:2210.09241.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful response. Regarding novelty, we would like to clarify that while prior works exist that utilize re-weighting with offline RL algorithms, in this work, we not only develop a method for re-weighting distributional constraint methods but also analyze challenges with distribution constraints on heteroskedastic datasets. That is, our contribution isn’t just in developing a re-weighting method for offline RL, but in understanding challenges with existing methods and showing how re-weighting can alleviate these challenges. To further address your concerns, we compare ReDS to the method Hong et al. 2023 and find ReDS outperforms this method, indicating its efficacy. We discuss this and other questions in detail below. **Please let us know if your questions are resolved, and if so, we would appreciate it if you could update your score.  We would be happy to discuss any further questions.**\n\n***Comparisons with Hong et al. 2023***\n\nThank you for pointing out these comparisons. We do believe that these methods are quite relevant and we will discuss them in the related works section of the paper. We also ran experiments on the heteroskedastic AntMaze domains to evaluate these methods and present our results below. In brief, we found that ReDS still outperforms this prior method.\n\n**RW and AW (Hong et al. 2023):** We tried two variations that the authors use for weighting: Reward Weighting (RW) and Advantage Weighting (AW), using the author’s hyperparameters for the D4RL AntMaze tasks. We compared against the best-performing variation for each dataset. For these same variations and hyperparameter settings, we are also able to replicate the author’s performance on the D4RL AntMaze tasks.\n\nNote that the performance is significantly worse compared to the base algorithm (IQL) used by these methods and our approach, CQL (ReDS). Note this isn’t contradictory:  the author’s reported numbers are outperformed by IQL (baseline; with uniform sampling) on the D4RL AntMaze tasks. \nA full comparison is found below:\n| Task & Dataset |     EDAC     |    BEAR     |    CQL    |    IQL    |    INAC    | RW and AW | ReDS [Ours] |\n|:--------------:|:------------:|:-----------:|:---------:|:---------:|:----------:|:---------:|:---------------:|\n|  medium-noisy  |      0       |      0      |    55     |    44     |     0      |     5     |      **73**     |\n| medium-biased  |      0       |      0      |  **73**   |    48     |     0      |     0     |      **74**     |\n|  large-noisy   |      0       |      0      |    42     |    39     |     0      |    10     |      **53**     |\n| large-biased   |      0       |      0      |  **50**   |    41     |     0      |     8     |        45       |\n\nWe aren’t able to compare with Yue et al in time for the rebuttal but will in the final.\n\n**Why does ReDS outperform AW and RW?:** We believe this limitation stems from the re-weighting of entire trajectories. We wouldn’t expect such a strategy to help with heteroskedasticity especially when states with both narrow and wide behavior action distributions appear in the same trajectory such as in the heteroskedastic AntMaze or even in our didactic example in Figure 1. Reweighting a trajectory wouldn’t provide precise-enough control to change the behavior distribution to impose the correct state-specific re-weighting.\n\n***As a standard benchmark for offline RL, the medium-level locomotion tasks, the adroit tasks, and the original AntMaze tasks are missing from the paper***\n\nWhile the standard D4RL tasks provide an effective way to benchmark offline RL methods, our focus in this work was to validate if ReDS can be effective with heteroskedastic datasets. Since D4RL tasks consist of data from policies that typically exhibit uniform action entropy across states, they tend to not be heteroskedastic as observed in Table 2a. Due to this reason, we thought it was more important to study the performance of ReDS on heteroskedastic datasets while making sure it does not degrade performance on a subset of standard D4RL tasks (as in Table 1). That said, we are running additional experiments on medium locomotion tasks and Antmaze tasks, and will report back on the status of our results before the end of the discussion period. \n\n***Support matching methods for offline RL can be algorithm-agnostic***\n\nWhile we agree that support-matching methods for offline RL can be domain agnostic, in this paper, we attempted to instantiate ReDS for CQL: an instantiation specific to methods that use a conservative regularizer like CQL (or related approaches like COMBO). Here, we modify the distribution for push-up and push-down in the CQL regularizer (Figure 3), but these loss terms aren’t present in other algorithms such as policy constraints. We would like to clarify that we don’t claim that ReDS is algorithm agnostic. Other works in offline RL also propose algorithms that aren’t general methodologies (e.g., TD3+BC, IQL, etc). As such, we believe that the lack of a general methodology should not be used for rejecting the paper, provided the method attains good results. \n"}]

Author rebuttal:

Meta-review: The paper introduces ReDS, a new offline RL method for handling heteroskedastic datasets. ReDS incorporates support constraints by reweighting the data distribution based on conservative Q-learning (CQL). This allows the learned policy to deviate from the behavior policy within its support while maximizing long-term return. Various novel heteroskedastic datasets were introduced to showcase the superior performance of ReDS compared to existing offline RL methods. Both theoretical analyses and experimental results are provided to demonstrate the effectiveness of the resulting algorithm.

The reviewers acknowledged that most of this paper is well-written and easy to follow. They argued that the research problem addressed in this paper is relevant to real-world scenarios and that the authors provided interesting didactic examples to illustrate the potential issues with previous methods. Another positive aspect of this work is how the paper proposed a concise and intuitive plug-in solution (ReDS) while demonstrating its effectiveness. Finally, the experimental settings are diverse and cover multiple domains.

During the discussion between reviewers, a few concerns were raised. First, a reviewer argued that the Atari benchmark should be replaced with other domains that potentially have heteroskedasticity, such as autonomous driving datasets. Another reviewer brought up three concerns in particular: **(1)** There appears to be a gap between the theoretical analysis and the practical implementation of the algorithm, particularly concerning the heuristic design of the coefficient; **(2)** The paper lacks a thorough analysis explaining why ReDS outperforms in-sample learning methods; the reviewer believes that providing a more detailed analysis could strengthen this paper; and **(3)** The paper seems somewhat outdated. Finally, two other reviewers argued that parts of the paper only introduce seemingly incremental contributions, and that the experiments in the noisy and biased version of Antmaze tasks seem contrived.

Having said that, all reviewers acknowledged this work's contributions while emphasizing points that need improvement before publication. They encouraged the authors to explore these points and address the limitations brought up in the reviews when preparing an updated version of this paper.
Decision: Accept (poster)