SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our key insight is that by reweighting the action distribution in the data before utilizing a distribution constraint, we can obtain a method that enforces a per-state distribution constraint, which corresponds to an approximate support constraint. This will push down the values of actions that are outside the behavior policy support, but otherwise not impose a severe penalty for in-support actions, thus enabling the policy to deviate from the behavior policy by different amounts at different states. Rather than having a distribution constraint between  $\pi$  and  $\pi_{\beta}$  (Eq. 1), if we can impose a constraint between  $\pi$  and a reweighted version of  $\pi_{\beta}$ , where the reweighting is state-dependent, then we can obtain an approximate support constraint. Let the reweighted distribution be  $\pi^{re}$ . Intuitively, if  $\pi(\cdot|\mathbf{s})$  is within the support of the  $\pi_{\beta}(\cdot|\mathbf{s})$ , then one can find a reweighting  $\pi^{re}(\cdot|\mathbf{s})$  such that  $D(\pi, \pi^{re})(\mathbf{s}) = 0$ , whereas if  $\pi(\cdot|\mathbf{s})$  is not within the support of  $\pi^{re}(\cdot|\mathbf{s})$ , then  $D(\pi, \pi^{re})(\mathbf{s})$  still penalizes  $\pi$  when  $\pi$  chooses out-of-support actions, since no reweighting  $\pi^{re}$  can put non-zero probability on out-of-support actions. This allows us to handle the failure mode from Section 3: at states with wide behavior policy, even with a large  $\alpha$ ,  $\pi$  is not anymore constrained to the behavior distribution, whereas at other "critical" states, where  $\pi_{\beta}$  is narrow, a large enough  $\alpha$  will constrain  $\pi(\cdot|\mathbf{s})$  to stay close to  $\pi_{\beta}(\cdot|\mathbf{s})$ . We call this Reweighting Distribution constraints to Support (ReDS).
4.1 Instantiating the Principle Behind ReDS
One option is to reweight  $\pi_{\beta}$  to  $\pi^{re}$ , and enforce a distribution constraint  $D(\pi, \pi^{re})$  between  $\pi$  and  $\pi^{re}$ . However, this is problematic because the  $\pi^{re}$  would typically be estimated by using importance weighting or by fitting a parametric model, and prior work has shown that errors in estimating the behavior policy [43, 20] using only one action sample often get propagated and lead to poor downstream performance. For CQL, this issue might be especially severe if we push up the Q-values under  $\pi^{re}$ , because then these errors might lead to severe Q-value over-estimation.
Abstract idea of CQL (ReDS). Instead, we devise an alternative formulation for ReDS that modifies the learned policy  $\pi$  to  $\pi^{re}$ , such that applying a distribution constraint on this modified policy imposes a support constraint. Thus, with CQL, now we instead push down the Q-values under  $\pi^{re}$ . We define  $\pi^{re}$  as a mixture distribution of the learned policy  $\pi$  and a reweighted version of the behavior policy as follows:
Figure 3: Comparison between support and distributional constraints: Left: CQL pushes down the Q-function under the policy  $\pi$ , while pushing up the function under the behavior policy  $\pi_{\beta}$ . This means that the Q-values for bad actions can go up. Right: In contrast, ReDS re-weights the data distribution to push down the values of bad actions, alleviating this shortcoming.
$$
\pi^ {r e} (\cdot | \mathbf {s}) := \frac {1}{2} \pi (\cdot | \mathbf {s}) + \frac {1}{2} \left[ \pi_ {\beta} (\cdot | \mathbf {s}) \cdot g (\pi (\cdot | \mathbf {s})) \right], \tag {6}
$$
where  $g(\cdot)$  is a monotonically decreasing function. We will demonstrate how pushing down the Q-values under  $\pi^{re}$  modifies CQL to enable a support constraint while reusing existing components of CQL that impose a distribution constraint. As shown in Figure 3, the second term in Equation 6 increases the probability of actions that are likely under the behavior policy, but are less likely under the learned policy (due to  $g$  being a decreasing function). We will show in Lemma 4.1 that utilizing  $\pi^{re}$  in CQL enforces a support constraint on  $\pi$ . Thus, the learned policy  $\pi$  can be further away from  $\pi_{\beta}$ , allowing  $\pi$  to assign more probability to good actions that are within the behavior policy support, even if they have lower probabilities under  $\pi_{\beta}$ . Section 4.2 illustrates theoretically why pushing down the Q-values under Eq. 6 approximates a support constraint in terms of how it modifies the resulting Q-values. For an illustration, please see Figure 3.
How should we pick  $g$  in practice? Since we wish to use  $\pi^{re}$  as a replacement for  $\pi$  in the minimization term in the CQL regularizer (Equation 2), we aim to understand how to design the re-weighting  $g$  in practice. Since specifically CQL enforces a distribution constraint by maximizing the Q-value on all actions sampled from the behavior policy  $\pi_{\beta}$ , our choice of  $g$  should aim to counter this effect by instead minimizing the Q-value on "bad" actions within the support of the behavior policy. Equation 6 quantifies the notion of these "bad" actions using a monotonically decreasing function  $g(\pi (\mathbf{a}|\mathbf{s}))$  of the policy probability. In practice, we find it convenient to define  $g$  to be a function of the advantage estimate:  $A_{\theta}(\mathbf{s},\mathbf{a})\coloneqq Q_{\theta}(\mathbf{s},\mathbf{a}) - E_{\mathbf{a}\sim \pi}[Q_{\theta}(\mathbf{s},\mathbf{a})]$ , that the policy  $\pi$  is seeking to maximize. In fact, if entropy regularization is utilized for training the policy (akin to most offline RL algorithms), the density of an action under a policy is directly proportional to exponentiated advantages, i.e.,  $\pi (\mathbf{a}|\mathbf{s})\propto \exp (A_{\theta}(\mathbf{s},\mathbf{a}))$ . Hence, we choose  $g(x) = 1 / x$ , such that  $g(\exp (A(\mathbf{s},\mathbf{a}))) = \exp (-A(\mathbf{s},\mathbf{a}))$  (a decreasing function).
For the rest, we approximate the product distribution  $\pi (\mathbf{a}|\mathbf{s})\cdot g(\pi (\mathbf{a}|\mathbf{s}))$  by fitting a parametric function approximator  $\rho_{\psi}(\mathbf{a}|\mathbf{s})$ . Since  $\rho_{\psi}(\mathbf{a}|\mathbf{s})$  is being trained to approximate a re-weighted version of the behavior policy, we fit  $\rho_{\psi}$  by minimizing using a weighted maximum log-likelihood objective, as shown in prior work [44, 43]. The concrete form for our objective for training  $\rho_{\psi}$  is shown below ( $\tau$  is a temperature hyperparameter typically introduced in prior work [44, 43]):
$$
\rho_ {\psi} (\cdot | \mathbf {s}) = \arg \max  _ {\rho_ {\psi}} \mathbb {E} _ {\mathbf {s} \sim \mathcal {D}, \mathbf {a} \sim \pi_ {\beta} (\cdot | \mathbf {s})} [ \log \rho_ {\psi} (\mathbf {a} | \mathbf {s}) \cdot \exp (- A _ {\theta} (\mathbf {s}, \mathbf {a}) / \tau) ]. \tag {7}
$$
The crucial difference between this objective and standard advantage-weighted updates is the difference of the sign. While algorithms such as AWR [43] aim to find an action that attains a high advantage while being close to the behavior policy, and hence, uses a positive advantage, we utilize the negative advantage to mine for poor actions that are still quite likely under the behavior policy.
The final objective for the Q-function combines the regularizer in Eq. 8 with a standard TD objective:
$$
\mathcal {R} (\theta ; \rho) = \frac {1}{2} \left(\underset {\mathbf {s} \sim \mathcal {D}, \mathbf {a} \sim \pi} {\mathbb {E}} \left[ Q _ {\theta} (\mathbf {s}, \mathbf {a}) \right] + \underset {\mathbf {s} \sim \mathcal {D}, \mathbf {a} \sim \rho} {\mathbb {E}} \left[ Q _ {\theta} (\mathbf {s}, \mathbf {a}) \right]\right) - \underset {\mathbf {s}, \mathbf {a} \sim \mathcal {D}} {\mathbb {E}} \left[ Q _ {\theta} (\mathbf {s}, \mathbf {a}) \right] \tag {8}
$$
$$
\min  _ {\theta} J _ {Q} (\theta) = \mathcal {R} (\theta ; \rho) + \frac {1}{2} \mathbb {E} _ {\mathbf {s}, \mathbf {a}, \mathbf {s} ^ {\prime} \sim \mathcal {D}} [ (Q _ {\theta} (\mathbf {s}, \mathbf {a}) - \mathcal {B} ^ {\pi} \bar {Q} (\mathbf {s}, \mathbf {a})) ^ {2} ] \tag {9}
$$
4.2 Theoretical Analysis of CQL (ReDS)
Next, we analyze CQL (ReDS), showing how learning using the regularizer in Eq. 8 modifies the Q-values and justifies our choice of the distribution  $\rho$  in the previous section.
Lemma 4.1 (Per-state change of Q-values.). Let  $g(\mathbf{a}|\mathbf{s})$  be a shorthand for  $g(\mathbf{a}|\mathbf{s}) = g(\tau \cdot \pi (\mathbf{a}|\mathbf{s}))$ . In the tabular setting, the  $Q$ -function obtained after one iteration of objective in Eq. 9 is given by:
$$
Q _ {\theta} (\mathbf {s}, \mathbf {a}) := \mathcal {B} ^ {\pi} \bar {Q} (\mathbf {s}, \mathbf {a}) - \alpha \frac {\pi (\mathbf {a} | \mathbf {s}) + \pi_ {\beta} (\mathbf {a} | \mathbf {s}) g (\mathbf {a} | \mathbf {s}) - 2 \pi_ {\beta} (\mathbf {a} | \mathbf {s})}{2 \pi_ {\beta} (\mathbf {a} | \mathbf {s})} \tag {10}
$$
where  $\mathcal{B}^{\pi}\bar{Q} (\mathbf{s},\mathbf{a})$  is the Bellman backup operator applied to a delayed target  $Q$  network.
Eq. 10 illustrates why the modified regularizer in Eq. 8 leads to a "soft" support constraint whose strength is modulated per-state. Since  $g$  is a monotonically decreasing function of  $\pi$ , for state-action pairs where  $\pi(\mathbf{a}|\mathbf{s})$  has high values,  $g(\mathbf{a}|\mathbf{s})$  is low and therefore the Q-value  $Q(\mathbf{s},\mathbf{a})$  for such state-action pairs are underestimated less. Vice versa, for state-action pairs where  $\pi(\mathbf{a}|\mathbf{s})$  attains low values,  $g(\mathbf{a}|\mathbf{s})$  is high to counter-acts the low  $\pi(\mathbf{a}|\mathbf{s})$  values. Also, since  $\pi_{\beta}(\mathbf{a}|\mathbf{s})$  appears in the denominator, for out-of-support actions, where  $\pi_{\beta}(\mathbf{a}|\mathbf{s}) = 0$ ,  $\pi(\mathbf{a}|\mathbf{s})$  must also assign 0 probability to the actions for the Q values to be well defined. An illustration of this idea is shown in Figure 9. We can use this insight to further derive the closed-form objective optimized by ReDS.
Lemma 4.2 (CQL (ReDS) objective.). Assume that for all policies  $\pi \in \Pi, \forall (\mathbf{s}, \mathbf{a}), \pi (\mathbf{a}|\mathbf{s}) > 0$ . Then, CQL (ReDS) solves the following optimization problem:
$$
\left. \max  _ {\pi \in \Pi} \widehat {J} (\pi) - \frac {\alpha}{2 (1 - \gamma)} \mathbb {E} _ {\mathbf {s} \sim \widehat {d} ^ {\pi}} \left[ D (\pi , \pi_ {\beta}) (\mathbf {s}) + \underset {\mathbf {a} \sim \pi (\cdot | \mathbf {s})} {\mathbb {E}} [ g (\tau \cdot \pi (\mathbf {a} | \mathbf {s})) \mathbb {I} \{\pi_ {\beta} (\mathbf {a} | \mathbf {s}) > 0 \} ] \right]. \right. \tag {11}
$$
$\widehat{J}(\pi)$  corresponds to the empirical return of the learned policy, i.e., the return of the policy under the learned Q-function. The objective in Lemma 4.3 can be intuitively interpreted as follows: The first term,  $D(\pi, \pi_{\beta})(s)$ , is a standard distribution constraint, also present in naïve CQL, and it aims to penalize the learned policy  $\pi$  if it deviates too far away from  $\pi_{\beta}$ . ReDS adds an additional second term that effectively encourages  $\pi$  to be "sharp" within the support of the behavior policy (as  $g$  is monotonically decreasing), enabling  $\pi$  to potentially put mass on actions that lead to a high  $\widehat{J}(\pi)$ .
Specifically, this second term allows us to control the strength of the distribution constraint per state: at states where the support of the policy is narrow, i.e., the volume of actions such that  $\pi_{\beta}(\mathbf{a}|\mathbf{s}) > 0$  is small (say, only a single action), the penalty in Equation 51 reverts to a standard distributional constraint by penalizing divergence from the behavioral policy via  $D(\pi ,\pi_{\beta})(\mathbf{s})$  as the second term cannot be minimized. At states where the policy  $\pi_{\beta}$  is broad, the second term counteracts the effect of the distributional constraint within the support of the behavior policy, by enabling  $\pi$  to concentrate its density on only good actions within the support of  $\pi_{\beta}$  with the same multiplier  $\alpha$ . Thus even when we need to set  $\alpha$  to be large to stay close to  $\pi_{\beta}(\cdot |\mathbf{s})$  at certain states (e.g., in narrow hallways in the example in Sec. 3.1),  $D(\pi ,\pi_{\beta})(\mathbf{s})$  is not heavily constrained at other states.
In fact, we formalize this intuition below to show that for the best possible value of the hyperparameters appearing in the training objective for CQL (ReDS) (Equation 51), CQL (ReDS) is guaranteed to outperform the best-tuned version of CQL for any offline RL problem. A proof is in Appendix C.
Lemma 4.3 (CQL (ReDS) formal guarantee). We will add the following guarantee to show that the policy learned by ReDS for the best possible value of  $\tau$  (Equation 51) and  $\alpha$  in CQL (Equation 3) outperforms the best CQL policy. That is, formally we show:
$$
\max  _ {\alpha , \tau} J \left(\pi_ {\mathrm {R e D S}; \alpha , \tau}\right) \geq \max  _ {\alpha} J \left(\pi_ {\mathrm {C Q L}; \alpha}\right). \tag {12}
$$
The goal of our experiments is to understand how CQL (ReDS) compares to distributional constraint methods when learning from heteroskedastic offline datasets. In order to perform our experiments,
we construct new heteroskedastic datasets that pose challenges representative of what we would expect to see in real-world problems. We first introduce tasks and heteroskedastic datasets that we evaluate on, and then present our results compared to prior state-of-the-art methods. We also evaluate ReDS on some of the standard D4RL [13] datasets which are not heteroskedastic in and find that the addition of ReDS, as expected, does not help, or hurt on those tasks.
5.1 Comparison on the D4RL Benchmark
Dataset	BC	10%BC	DT	AWAC	Onestep RL	TD3+BC	COMBO	CQL	IQL	Ours
halfcheetah-medium-replay	36.6	40.6	36.6	40.5	38.1	44.6	55.1	45.5	44.2	52.3
hopper-medium-replay	18.1	75.9	82.7	37.2	97.5	60.9	89.5	95.0	94.7	101.5
walker2d-medium-replay	26.0	62.5	66.6	27.0	49.5	81.8	56.0	77.2	73.9	85.0
halfcheetah-medium-expert	55.2	92.9	86.8	42.8	93.4	90.7	90.0	91.6	86.7	89.5
hopper-medium-expert	52.5	110.9	107.6	55.8	103.3	98.0	111.1	105.4	91.5	110.0
walker2d-medium-expert	107.5	109.0	108.1	74.5	113.0	110.1	103.3	108.8	109.6	112.0
locomotion total	295.9	491.8	488.4	277.8	494.8	486.1	505	523.5	500.6	550.3
Heteroskedastic data is likely to exist in real-world problems such as driving and manipulation, where datasets are collected by multiple policies that agree and disagree at different states. While standard benchmarks (D4RL [13] and RLUnplugged [21]) include offline datasets generated by mixture policies (e.g. the "medium-expert" generated by two policies with different performance), these policies are trained via RL methods (SAC) that constrain the entropy of the action distribution at each state to be uniform. To measure heteroskedasticity, we utilize an approximation to  $C_{\mathrm{diff}}^{\pi}$ : the standard deviation in the value of  $D(\pi, \pi_{\beta})(\mathbf{s})$  across states in the dataset, using a fixed policy  $\pi$  obtained by running CQL. We didn't use  $C_{\mathrm{diff}}^{\pi}$  directly, as it is challenging to compute in continuous spaces. In Table 3, the standard deviation is lower for the D4RL antmaze datasets, corroborating our intuition that these datasets are significantly less heteroskedastic.
5.2 Comparisons on Heteroskedastic datasets
Heteroskedastic datasets. To stress-test our method and prior distribution constraint approaches, we collected new datasets for the medium and large mazes used in the antmaze navigation tasks from D4RL: noisy datasets, where the behavior policy action variance differs in different regions of the maze, representative of user variability in navigation, and biased datasets, where the behavior policy admits a systematic bias towards certain behaviors in different regions of the maze, representative of bias towards certain routes in navigation problems. Table 3 shows that these datasets are significantly more heteroskedastic to the D4RL datasets.
Table 1: Performance comparison on the D4RL benchmark. (Top 2 bolded)
Dataset	std	max
noisy (Ours)	18	253
biased (Ours)	9	31
diverse (D4RL)	2	11
play (D4RL)	2	13
Table 3: The new antmaze datasets (Ours) are significantly more heteroskedastic than the standard D4RL datasets. We measure heteroskedasticity using the std and max of  $D(\pi, \pi_{\beta})(\mathbf{s})$  across states in the offline dataset.
Using these more heteroskedastic datasets, we compare CQL (ReDS) with CQL and IQL [27], recent popular methods, and two prior methods, BEAR [30] and EDAC [3], that also enforce support constraints. For each algorithm, including ours, we utilize hyperparameters directly from the counterpart tasks in D4RL. Due to the lack of an effective method for offline policy selection (see Fu et al. [14]), we utilize oracle checkpoint selection for every method. We compute the mean and standard deviation across 3 seeds. Table 2 shows that the largest gap between CQL (ReDS) and prior methods is on noisy datasets, which are particularly more heteroskedastic (Table 3).
We also compare CQL (ReDS) with recent offline RL algorithms on D4RL, including DT [8], AWAC [42], onestep RL [5], TD3+BC [16] and COMBO [57]. Table 1 shows that CQL (ReDS) obtains similar performance as existing distributional constraint methods and outperforms BC-based baselines. This is expected given that the D4RL datasets exhibit significantly smaller heteroscedasticity, as previously explained. Also, a large fraction of the datasets is trajectories with high returns. BC using the top  $10\%$  trajectories with the highest episode returns already has strong performance. The previous results compares CQL (ReDS) to baselines in tasks where the MDP states are low-dimensional vectors. Next, we study vision-based robotic manipulation tasks.
Visual robotic manipulation. We consider two types of manipulation tasks. In the "Pick & Place" task, the algorithm controls a WidowX robot to grasp an object and place it into a tray located at a test
Task & Dataset	EDAC	BEAR	CQL	IQL	INAC	RW & AW	EQL	SQL	XQL-C	Ours
medium-noisy	0	0	55	44	0	5	0.0	0.7	4.3	73
medium-biased	0	0	73	48	0	0	6.5	8.0	11.7	74
large-noisy	0	0	42	39	0	10	7.1	2.9	11.3	53
large-biased	0	0	50	41	0	8	8.5	0.5	7.3	45
Figure 4: Examples rollouts in the heteroskedastic bin-sort data. In this task, an offline RL method must sort objects in front of it into two bins with a dataset that has non-uniform coverage at different states, using visual input. In the first half of the trajectory, the states exhibit a more narrow action distribution but the second half admits a more uniform action distribution.
location, directly from raw  $128 \times 128 \times 3$  images and sparse 0/1 reward signal. The dataset consists of behavior from suboptimal grasping and placing policies, and the positions of the tray in the offline dataset very rarely match the target test location. The placing policies exhibit significant variability, implying these datasets are heteroskedastic under our definition. We also consider "Bin Sort" task (see Figure 4), where a WidowX robot is controlled to sort two objects into two separate bins. Here, heteroskedacity is introduced when sorting objects into the desirable bins. Similar to the Pick & Place task, the placing policy exhibits significant variability, showing an object placed in the incorrect bin (e.g., recyclable trash thrown into the non-recyclable bin). However, the grasping policy is more expert-like grasping the object with low variability. More details in Appendix E.
Figure 5: CQL vs ReDS: IQM normalized score for 10 Atari games. We consider two dataset compositions.
Table 4 presents the results on these tasks. We utilize oracle policy selection analogous to the antmaze experiments from Table 2. Table 4 shows that CQL (ReDS) outperforms CQL attaining a success rate of about  $15.1\%$  for the visual pick and place task, whereas CQL only attains  $6.5\%$  success. While performance might appear low in an absolute sense, note that both CQL and ReDS do improve over the behavior policy, which only attains a success rate of  $4\%$ . Thus offline RL does work on this task, and utilizing ReDS in conjunction with the standard distributional constraint in CQL does result in a boost in performance with this heteroskedastic dataset. For the "Bin Sorting", our method outperforms CQL by  $3.5\mathbf{x}$  when learning from more heteroskedastic datasets. This indicates the effectiveness of our method in settings with higher heteroskedasticity.
Table 2: CQL (ReDS) outperforms prior offline RL methods including methods (IQL, XQL-C), and prior support constraint methods (BEAR, EDAC, SQL, EQL, RW & AW) on three out of four scenarios when learning from heteroskedastic data in the antmaze task. The improvement over prior methods is larger when learning from the noisy datasets, which are more heteroskedactic, as in Table 3, compared to biased datasets.
Task	CQL	CQL (ReDS)	std D(π, πβ)(s)	max D(π, πβ)(s)
Pick & Place	6.5 ± 0.4	15.1 ± 0.4	48.7	307.4
Bin Sort (Easy)	31.2 ± 0.3	31.4 ± 0.3	7.9	81.6
Bin Sort (Hard)	6.1 ± 0.2	23.1 ± 0.7	59.6	988.3
Table 4: CQL (ReDS) vs CQL on robotic manipulation tasks. CQL (ReDS) outperforms CQL significantly when learning from more heteroskedastic datasets, as measured by  $C_{\mathrm{diff}}^{\pi}$ : the standard deviation and the maximum of  $D(\pi, \pi_{\beta})(\mathbf{s})$  across states.
Atari games. We collect data on 10 Atari games from multiple policies that behave differently at certain states while having similar actions otherwise. We consider a case of two such policies, and a harder scenario of five. We evaluate the performance of CQL (ReDS) on the Atari games using the evaluation metrics from prior works [2, 34]. Figure 5 shows that in both testing scenarios: with the mixture of two policies (top figure) and the mixture of five policies (bottom figure), CQL (ReDS) outperforms CQL in aggregate.
To summarize, our results indicate that incorporating CQL (ReDS) outperforms distribution constraints with heteroskedastic datasets in a variety of domains.
Offline Q-learning methods utilize mechanisms to prevent backing up unseen actions [39], by applying an explicit behavior constraint that forces the learned policy to be "close" to the behavior policy [23, 53, 44, 49, 53, 30, 28, 27, 52, 15], or by learning a conservative value function [33, 54, 41, 57, 56, 47, 24, 53]. Most of these offline RL methods utilize a distribution constraint, explicit (e.g., TD3+BC [15]) or implicit (e.g., CQL [33]), and our empirical analysis of representative algorithms from either family indicates that these methods struggle with heteroskedastic data, especially those methods that use an explicit constraint. Model-based methods [26, 56, 4, 51, 45, 38, 57] train value functions using dynamics models, which is orthogonal to our method.
Some prior works have also made a case for utilizing support constraints instead of distribution constraints, often via didactic examples [30, 29, 39], and devised algorithms that impose support constraints in theory, by utilizing the maximum mean discrepancy metric [30] or an asymmetric f-divergences [53] for the policy constraint [53]. Empirical results on D4RL [13] and the analysis by Wu et al. [53] suggest that support constraints are not needed, as strong distribution constraint algorithms often have strong performance. As we discussed in Sections 3.2 (Theorem 3.2 indicates that this distribution constraints may not fail when  $C_{\mathrm{diff}}^{\pi}$  is small, provided these algorithms are well-tuned.) and 4, these benchmark datasets are not heteroskedastic, as they are collected from policies that are equally wide at all states and centered on good actions (e.g., Antmaze domains in [13], control suite tasks in Gulcehre et al. [21]) and hence, do not need to modulate the distribution constraint strength. To benchmark with heteroskedastic data, we developed some novel tasks which may be of independent interest beyond this work, and find that our method ReDS can work well here.
7 Discussion, Future Directions, and Limitations