Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
Authors: Zhang-Wei Hong
Aviral Kumar
Sathwik Karnik
Abhishek Bhandwaldar
Akash Srivastava
Joni Pajarinen
Romain Laroche
Abhishek Gupta
Pulkit Agrawal
Venue: NeurIPS 2023 poster (2023)
Keywords: offline reinforcement learning
reinforcement learning
sampling
experience replay
TL;DR: We improve offline RL performance on imbalanced datasets

Abstract: Offline reinforcement learning (RL) enables learning a decision-making policy without interaction with the environment. This makes it particularly beneficial in situations where such interactions are costly. 
However, a known challenge for offline RL algorithms is the distributional mismatch between the state-action distributions of the learned policy and the dataset, which can significantly impact performance. State-of-the-art algorithms address it by constraining the policy to align with the state-action pairs in the dataset. However, this strategy struggles on datasets that predominantly consist of trajectories collected by low-performing policies and only a few trajectories from high-performing ones. Indeed, the constraint to align with the data leads the policy to imitate low-performing behaviors predominating the dataset. Our key insight to address this issue is to constrain the policy to the policy that collected the good parts of the dataset rather than all data. To this end, we optimize the importance sampling weights to emulate sampling data from a data distribution generated by a nearly optimal policy. Our method exhibits considerable performance gains (up to five times better) over the existing approaches in state-of-the-art offline RL algorithms over 72 imbalanced datasets with varying types of imbalance.

Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
Zhang-War Wei Hong'; Aviral Kumar?, Sathwik Karnik', Abhishek Bhandwaldar?, Akash Srivastava", " Joni Pajarinen', Romain Laroche Abhishek Gupta", Pulkit Agrawal*
Abstract
Offline policy learning is aimed at learning decision-making policies using existing datasets of trajectories without collecting additional data. The primary motivation for using reinforcement learning (RL) instead of supervised learning techniques such as behavior cloning is to find a policy that achieves a higher average return than the trajectories constituting the dataset. However, we empirically find that when dataset dominated by suboptimal trajectories, state-of-the-arr offline RL algorithms do not substantially improve over the average return of trajectories in the dataset. We argue this is due to an assumption made by current offline RL algorithms of staying close to the trajectories in the dataset. If the dataset primarily consists of sub-optimal trajectories, this assumption forces the policy to mimic the suboptimal actions. We overcome this issue by proposing sampling strategy that enables the policy to only be constrained to "good data" rather than all actions in the dataset (i.e., uniform sampling). We present realization of the sampling strategy and an algorithm that can be used as plug-and-play module in standard offline RL algorithms. Our evaluation demonstrates significant performance gains in 72 imbalanced datasets, D4RL dataset, and across three different offline RL algorithms. Code is available at https: //g/ thtt....bc. wtw.ttt..oooooo...eoooooottttttt cowmImprobablleeeeeeeooeeeeoeeee AI/dw-of off-eefffe--rrr
Introduction
Offline reinforcement learning (RL) [23, 27] aims leam decision- policy that maximizes the expected return (i.e., the sum of rewards over time) using a pre-collected dataset of trajectories, making it appealing for applications where data collection is infeasible or expensive (e-g., recom- mendation systems [28]). Without loss of generality, it can be assumed that the dataset is generated from an unknown policy (a)s))) also known as the behavior policy [24]. The goal in offline RI. to learn policy, (a)a))) with parameters 0, that exceeds the performance of the behavior policy, In offline RL, a widely recognized issue the overestimation of Q-values for out-of-distribution state-action pairs, leading to suboptimal policies [8, 20, 22]. This stems from incomplete coverage of the state- state-action action space in the dataset, causing the learning algorithm to consider absent states and actions during optimization.
Most fa-te-o-thee art offline RL algorithms [7. 20. 22, 25] mitigate the issue of OOD Q-values by constraining the distribution of actions of the learned policy ma(as)), to be close to the distribution of actions the dataset. This results in a generic objective with the following form:
max J (T²) aL(saa)D [C(s, a)),
%
where J(mo) denotes the expected return of the policy me. D denotes the dataset, C is regularization term that penalizes the policy for deviating from the state-action pairs in the dataset, and is the
Correspondence: zwhong@mit. edu, Improbable Lab, Massachusetts Institute of Technology', RAIL Lab, UC Berkeley?. MIT-IBM Lab3, Aalto University", University of Washington's and independent researcher
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Action likelihood
Action likellhood
Action
Action
(a) Regularized offline RI. uniform sampling
(b) Regularized
re-weighted
Behavior
Learned policy %
Re-weighted behavior policy Sig,
High-return action
Low-rcturn
policy #g
action
The dots represent actions in the dataset, where imbalanced datasets have more low-return actions. (a) Regularized offline RL algorithms [22, 7, 18] equally regularize the policy Tg on each action, leading to imitation of low-return actions and a low- www-performing Ta. The color under the curves shows the policy's performance JTT)) with red indicating higher performance and blue indicating lower performance. (b) Re-weighting the dataset based on actions' returns allows the algorithm only regularize on actions with high returns, enabling the policy we to imitate high-return actions while ignoring low-return actions.
Figure 1:
hyper-parameter balancing the conflicting objectives of maximizing returns while also staying close to the data distribution. This prevents offline RL algorithms from learning behaviors that produce action distributions that diverge significantly from the behavior policy.
An masy-to-understande example of choice for C the squared distance between the policy and the data [7], C(s, == || (s) all2. where (s(s) denotes the mean of the action distribution (-(sss) and a is an action sampled from the dataset. When the collected dataset is good, i.e., mostly comprising high-return trajectories, staying close to the data distribution is aligned with the objective of maximizing return, and existing offline RL algorithms work well. However, in scenarios where the dataset is skewed imbalanced, i.e., contains only few high-return trajectories and many low-return trajectories, staying close to the data distribution amounts to primarily imitating low-performing actions and is, therefore, detrimental. Offline RL algorithms struggle learn high-return policies in such scenarios [12]. We present a method that overcomes this fundamental limitation of offline RL. the sense that is agnostic to the choice of the offline RL algorithm.
Our method
Our key insight stems from the observation that current methods are unnecessarily conservative by forcing the policy Te to stay close to all the data. Instead, we would ideally want the policy Ty to be close to the best parts of the offline dataset. This suggests that we should constrain Ty to only be close state-action pairs that would be generated from policy that achieves high returns, for instance, the (nearly) optimal policy ... In offline scenarios, where collecting additional data re-weight existing
is prohibited, to mirror the data distribution of xt much possible, data (i.c., importance sampling [17]). We instantiate this insight in the following way: Represent the distribution induced by better policy TD.. (initially unknown) by re-weighting data points in the dataset with importance weights w(s, a) and denote this distribution as D (s, a). Under this weighting, the offline RL algorithm's training objective can be written as:
max J (TT) (..(())) [w(s, a)C(s, a)],
Solving for Ty using the re-weighted objective constrains the policy to be close to the better policy TD. and, therefore, allows learning of performant policies. The key challenge determining the weights w since the state-action distribution of the better policy TD... is initially unknown. To address this, we employ off-policy evaluation techniques [32, 26, 44] to connect the data distribution D., with the expected return of the policy TD. that would generate it. This allows one to optimize the importance weights with respect its expected return as follows:
max J(TD..) E(e,a)>D.,, [r(s, a)] E(E.a)>D (w((s,)r(ss))
Here r(s, a) denotes reward of state-action pair (s, a). By exploiting this connection, we optimize the importance weights maximize the expected return J(TD..) of its corresponding policy TD. subject to necessary constraints (i.e., Bellman flow conservation constraint [35, 32]). This enables us obtain better importance weights w.
We evaluate our method with state-of-the-art offline RI. algorithms [22, 18] and demonstrate perfor- mance gain on 72 imbalanced datasets [12, 6]. Our method significantly outperforms prior work [12]

---

in challenging datasets with more diverse initial states and fewer trajectories (20x smaller than exist- ing datasets). These datasets pose greater challenges, yet they are crucial for practical applications since real-world datasets are often small and exhibit diverse initial states (e.g., robots with different starting positions, where data comes from human teleoperation).
2
Preliminaries
Typical (online) RL. Reinforcement learning a formalism that enables us to optimize an agent's policy in a Markov decision process (MDP [35]). The agent (i.e., decision- state so sampled from an initial state distribution (o At each timestep f, the agent perceives the state St, takes an action a, ~ xT (-Iss) with its policy receives reward re = r(st, az) from the envi- ronment, and transitions to next state 844-1 sampled from the environment dynamics T [. at) until reaching terminal states. The goal in RL to learn policy to maximize the 7-discounted expected,
starts from an initial
Ei- 7'r(s;,.:) Typical (on- line) RL algorithms estimate the policy It expected return (policy evaluation) from trajectories (So, 4o, ro, $1, 41, ...) generated by rolling out and update the policy π toward increasing (T) (policy improvement), and repeat the processing by performing rollouts with the updated policy.
infinite-horizon return J (T) Eso (a
=
Offline RL. With interaction with the enviromment allowed during the course of learning, offline RL algorithms aim to learn a policy π that maximizes return, entirely using a fixed dataset 'D that was collected by an arbitrary and unknown "behavior policy" TD (e.g., humans controllers). These methods typically aim to estimate the return of a policy a via techniques such as Q-learning or actor-critic, only using batches of state-action pairs (st, at) uniformly drawn from D. We will denote this estimate value of the return as (T)). The dataset consists of N trajectories rolled out by TD:
pre-programmed
D= [T+ = (sg. Go, ru, $1, 01, ... sT.))
(1)
where T3 denotes the length of Tj. In practice, a limit on trajectory length is required since we can only collect finite-length trajectories [34]. When the states that the policy т would encounter and the actions that x would take are not representative in the dataset D, the estimated return (T) is typically inaccurate [8, 20, 22]. Thus, most offline RI. algorithms learn the policy æ with pessimistic or conservative regularization that penalizes shift of ı from the behavior policy uD that collected the dataset. Typically, implicitly or explicitly, the policy т learned by most offline RL algorithms can be thought of as optimizing the following regularized objective:
max 5. al al [C(ss, az)l
(2)
where C measures some kind of divergence (e.g.. Kullback-Leibler divergence (7]) between π and TD. and a IR+ denotes the strength of regularization.
Problem Statement: Unnecessary Conservativeness in Imbalanced Datasets
In this section, we describe the issue of offline RL in imbalanced datasets. While algorithms derived from the regularized offline RL objective (Equation 2) attain good performance on several standard benchmarks [6], recent work [12] showed that it leads to "unnecessary conservativeness" on imbalanced datasets [12] due to the use of constant regularization weight on each state-action pairs (s, a) in Equation 2. To illustrate why, we start by defining imbalance of a dataset D using the positive-sided variance of the returns of the dataset (RPSV [12] defined in Definition 3. In essence, RPSV measures the dispersion of trajectory returns in the dataset. It indicates the room for improvement of the dataset. Figure 2 illustrates the distribution of trajectory returns in imbalanced datasets with high and low RPSV. Datasets with low RPSV exhibit pronounced concentration of returns around the mean value, whereas datasets with a high RPSV display return distribution that extends away from the mean, towards higher returns. Intuitively, a dataset with high RPSV has trajectories with far higher returns than the average return of the dataset, indicating high chances of finding better data distribution through reweighting. Throughout this paper, we will use the term imbalanced datasets to denote datasets with high RPSV.
3

---

Definition 3.1 (Dataset imbalance). RPSV of a dataset, V+ V+(G(T))),, corresponds to the second-
order moment of the positive component of the difference between trajectory return: G(T,) ==
-x-71 y (s) a)) and its expectation, where 7; denote trajectory the dataset:
G(T)) EnLD (G(T)) - -
with x = max 0}
(3)
Imbalanced datasets common in real-world scenarios, as collecting high-return trajectories often costly than collecting low-return ones. An example of an imbalanced offline dataset autonomous driving, where most tra-
RPSV
Density
High
Low
jectories are from average drivers, with limited
data from very good drivers. Due the dom- inance of low-return trajectories, state-action over-
-0.5
0.0
0.5
1.0
1.5
pairs (s, from these trajectories sampled in Equation 2. Consequently, optimiz- ing the regularized objective (Equation 2) would result in a policy that closely imitates the actions from low- www-prrforminggg trajectories that constitute the majority of dataset DD but ideally, we want the policy to imitate actions only on state-action pairs from terhhperforming trajectories. How-
Return G(T,)
Figure 2: Return distribution of datasets with high and low RPSV. Low RPSV datasets have returns centered at the mean, while high RPSV datasets have wider distribution extending towards higher returns. See Appendix A.4 for details.
ever, current offline RL algorithms [21, 18, 7] use constant regularization weight a (Equation 2). As result, each state-action pairs is weighted equally, which leads the algorithm to be unnecessarily conservative on data (i.e., imitating all actions state-action pairs in the dataset). Further analysis of imbalanced datasets can be found in Appendix A.1.
4 Mitigating Unnecessary Conservativeness By Weighting Samples
In this section, we seek to develop an approach to address the unnecessary conservativeness issue (Section 3) of regularized offline RL algorithms in imbalanced datasets. Adding more experiences from high-performing policies to the dataset would regularize the policy to keep close to high- performing policies and hence easily mitigate the unnecessary conservativeness issue. Though collecting additional experiences (i.e., state-action pairs (x, a)) from the environment is prohibited in offline RL, importance sampling [32] can emulate sampling from another dataset D,, since the weighting can be regarded as the density ratio shown below:
as,
w(s, a)
a(s,a)
where (s, a) and D(s,a) a) denote the probability density of state-action pairs (s, in dataset Dw, and D. Note that D., is unknown but implicitly defined through given weighting function w. This allows adjust the sampling distribution that train the policy to, as suggested in the following equivalence:
max (T) (e [w(s, a))(s ax)] < max Jo(T)) (E(s,,)) [C(se, uz)].
(5)
The remaining question is: how we determine the weighting w(s, a) so that emulate sampling from a better dataset Du, collected by a policy that achieves higher return than the behavior policy TD that collected the original dataset D
4.1 Optimizing the Weightings: Emulating Sampling from High-Performing Policies
Our goal is to discover weighting function that can emulate drawing state vate-action samples from a better dataset D., that is collected by an alternative behavior policy TD.. with higher return than the behavior policy тo that collected the original dataset (i.e., J (TTD.) 2 (TD)) We make use of density-ratio-based off-policy evaluation methods [32, 29, 44] to determine if weighting function corresponds to a high-return policy. Note that we do not propose a new off-policy evaluation approach but rather apply the existing off- policy evaluation technique in our problem. By using these

---

techniques, we can relate the weighting w to the expected return of the alternative behavior policy ((DD)) via importance sampling formulation as follows:
(TD..) R ((s.a)DD [r(*, a)] = F (e.a) [io(s, a)r(s,
In Equation 6, (TDD.. ) evaluates the quality of given weighting function w. It also provides feasible objective to optimize w, since it only requires obtaining samples from the original dataset D. However, it is important to note that Equation 6 measures 7-discounted return only when the the dataset D., represents stationary state-action distribution that satisfies Bellman flow conservation constraint [35, 32, 44] in the MDP, as shown in the following equation:
Dw = - p)'('') +
Tss'ss, (u, E Dw(s') == 8,0
Du (7)
a'EA
where S and A denote the state and action spaces, respectively, and the discount factor y determines what discount factor corresponds to in (TD.) in Equation 6. We slightly abuse the notation, denoting state marginal as (s').
To estimate J (TD..) from the weighting function w, it is required to impose Bellman flow con- servation constraint (Equaton 7) on w. However, it is difficult impose this constraint due to the dependence of initial state distribution po in Equaton 7. Estimating Po from the first state of each trajectory in the dataset is an option, but is infeasible when the trajectories do not consistently start from initial states sampled from the distribution Po. While we could make the assumption that all trajectories begin from initial states sampled from Pu. it would limit the applicability of our method to datasets where trajectories start from arbitrary states. We thus choose not to make this assumption since current offline RI. algorithms do not require
Instead, since the Bellman flow conservation constraint (Equation 7) only depends on the initial state distribution Po when 7 * 1, it is possible to bypass this dependence, if we maximize the undiscounted return _(TD_.)
(TD_) (i.e.. setting = 1 in Equation 6) of the alternative behavior policy TD... While it deviates from the RL objective presented in Equation 2, undiscounted return is often more aligned with the true objective in various RL applications, as suggested in [13]. Many RL algorithms resort to employing discounted return as an approximation of undiscounted return instead due to the risk of divergence when estimating the undiscounted return using Q-learning [39]. Thus, we constrain the weighting function w to satisfy the Bellman flow conservation constraint with y = 1 as shown below:
Dur T((ss E.
8
To connect the constraint in Equation 8 to the objective in Equation 6, we rewrite Equation 8 in terms of weightings ws according to [29, 321², as shown below:
(s((s
D(s')w(s')= = Σ T(s' s. a.))) Vs' Es,
w(s) = Σ
(9)
D(s,a)
8,a
dEAA
where w(s) denotes state marginal weighting. Putting the objective (Equation 6) and the constraint (Equation together, we optimize to maximize the undiscounted expected return of the corre- sponding alternative behavior policy TD... as shown in the following:
max J (TT.... E(s,))-D [w(s, a))(s, a)]
(10)
subject to [(,,a,>'>~ (w(s') w(s a) = E..
As the weightings w can be viewed as the density ratio (Equation 4), we call our method as Density- ratio Weighting (DW). We then re- re-weight offline RL algorithm, shown in Equation 5. Note that while these weights correspond to (s, a) in the dataset, this is sufficient to reweight policy optimization for offline RL.
4.2 Practical Implementation
Optimizing weightings. We begin by addressing the parameterization of the weighting function w(s, a) and its state marginal w(s') in Equation 10. Though state marginal w(s') can derived from
"Se Equation 24 in [29]
5

---

summing w(s, a) over action space A, as defined in Equation 9, it can difficult to take summation over a continuous or infinite action space. Thus we opt to parameterize the weightings w(ss a) and its state marginal w(s') separately. By using the identities [32] D (8, a) = Dir (a|s) and
D(s, a) = (()) D (als): we can represent w(8,
a) as the product of two ratios:
vo(s; a) = (s. Du (s) Dio(s) D(s)
TD_ (als)
(11)
a))
D(s)
TD(aas)
Michel et al. [31] showed that ratios can be parameterized by neural networks with exponential oulput. Thus, we represent state-action weighting w(s, a) as wo a) and its state marginal as ws(s), as shown below:
W ( a) = exp ((s) exp (s,
m exp (ss)
(12)
where & and sb are neural networks. Next, present how to train both neural network models. As the dataset often has limited coverage on state-action space, it is preferable to add KL-divergence regularization D&L (D) to the objective in Equation 10, as proposed in Zhan et al. [43]. This regularization keeps the state-action distribution De induced by the learned weighting w close to the original dataset D. preventing Wa (8, a) from overfitting a few rare state-action pairs in D. Note that this does not prevent the learned weightings to provide a better data distribution for regularized offline RL algorithms. See Appendix A.2 for the detailed discussion. Another technical difficulty on training w is that it is difficult to impose Bellman flow conservation constraint in Equation 10 every state in the state space since only limited coverage of states are available in the dataset. Thus, we instead use penalty method [3] to penalize the solution of W on violating this constraint in expectation. As result, we optimize wass for Equation 10 using stochastic gradient ascent to optimize the following objective (details can be found in Appendix A.3):
max E(s,0,")>> Wo (s, ())(())()))))))))))))) AF (w)()d) - we (s, (s, a))² 0.v
AK (Pul)D KL regularization
(13)
Return
Bellman flow conservation penalty
where s' denotes the next state observed after taking action a at state 8, and Ar, Ak R-+ denote the strength of both penalty terms. Note that the goal of our work is not to propose new off-policy evaluation method, but to motivate ours in the specific objective to optimize the importance weighting for training offline RL algorithms. Importantly, our approach differs from previous off-policy evaluation methods [32, 26, 44], as further discussed in the related works (Section 6).
Applying the weighting to offline RL. The weighing function Wass could be pre-trained before training the policy, but this would introduce another hyperparameter: the number of pretraining iterations. As a consequence, we opt to train Wa.s in parallel with the offline RL. algorithm (i.e., value functions and policy). In our experiments, we perform one iteration of offline RL update pairs with one iteration of weighting function update. We also found that weighting both Jz and C(s, a) at each state-action pairs sampled from the dataset D with (s a) performs better than solely weighting the regularization term C(s, a). For example, when weighting the training objective of implicit Q-learning (IQL.) [18) (an offline RL method), the weighted objective JD., (*) is: E (s [ws.e (8, (s, a) log log (a|s)), where A(s, a) denotes advantage values. Please, see Appendix A.3 for implementation details. We hypothesize that weighting both the policy optimization objective (T)) and regularization C(s, a) in the same distribution (i.e., same importance weights) is needed to prevent policy x increasing (T) by exploiting out-of-distribution actions on states with lower weights Wy ( a). which could lead to poor performance [8]. Appendix A.5.5 compares weighting both and only one objective. The training procedure is outlined in Algorithm 1.
5 Experimental Evaluation
Our experiments aim to answer whether our rensity-ratio weighting (DW) (Section 4) approach can improve the performance of offline RL algorithms with different types of imbalanced datasets (Section 3). Prior work on imbalanced datasets [12] focused exclusively on imbalanced datasets with trajectories originating from a similar initial state. However, in real-world scenarios, trajectories can be collected from diverse initial states. For instance, when collecting datasets for self-driving cars, is likely that drivers initiate the recording of trajectories from drastically different initial locations. We

---

Algorithm Density-ratio weighting with generic offline RL algorithms (details in Appendix A.3)
1: Input: Dataset D
2: Initialize policy क and weighting function was
3: while not converged do
4: Sample a batch B of tuples of states, actions, rewards, and next states (s,a, (,,r,ss' from
5: Update Wass with batch B using Equation 13
6: Update policy æ and value function with an offline RL training objective with weights
Wo.s (s. and B (e.g., [22, 7, 18]
7: end while
found that imbalanced datasets with diverse initial states exhibit a long-tailed distribution of trajectory returns, while those with similar initial states show a bimodal distribution (see Appendix A.4 for details). As diversity of initial states affects the type of imbalance, we focus experimentation the two types of datasets: (i) Trajectories with similar initial states and (ii) Trajectories with diverse initial states.
Following the protocol in prior offline RL benchmarking [6], we develop representative datasets of each type using the locomotion tasks from the D4RL Gym suite. Our datasets are generated by combining - %% of trajectories from the random-v2 dataset (low-performing) and a% of trajectories from the medium-v2 or expert-v2 dataset thigh-performiin)) for each locomotion environment the D4RL benchmark, For instance, dataset that combines a% of random and 5% of medium trajectories is denoted as baselines on these imbalanced datasets across four a E {1, 5, 10, 50), four environments. Both types of datasets are briefly illustrated below and detailed in Appendix A.4. Additionally, present the results on the rest of original D4RL datasets in Appendix A.5.
%. We evaluate our method and the
(i) Trajectories with similar initial states. This type of datasets was proposed in [12], mixing trajectories gathered by high- and low-performing policies, as described Section 3. Each trajectory is collected by rolling out policy starting from similar initial states until reaching timelimit or terminal states. We consider a variant of smaller versions of these datasets that have small number of trajectories, where each dataset contains 50, 000 state-action pairs, which is 20 times smaller. These smaller datasets can test if a method overfits to small amounts of data from high-performing policies.
(ii) Trajectories with diverse initial states. Trajectories in this type of dataset start from wider range of initial states and have varying lengths. One real-world example of this type of dataset is a collection of driving behaviors obtained from fleet of self self-driving cars. The dataset might encompass partial trajectories capturing diverse driving behaviors, although not every trajectory accomplishes the desired driving task of going from one specific location to the other. As not all kinds of driving behaviors occur with equal frequency, such dataset is likely to imbalanced, with certain driving behaviors being underrepresented
5.1 Evaluation Setup
Baselines and prior methods. We consider uniform sampling (denoted as Uniform) the primary baseline for comparison. In addition, we compare our method with two existing approaches for improving offline RL performance on imbalanced datasets: advantage-weighting (AW), proposed in the recent work by [12] and percentage-filering (PF) [5]. Both AW and PF sample state-action pairs with probabilities determined by the trajectory's return to which they belong. The sampling probabilities for AW and PF are given as follows:
PAw (st, at) x exp((G(T)) - Vo(st))\n)
(Advantage-weighting
(14)
Ppr(st (stt at) x 1 [G(+)) 2 Gxs)
(Percentage-fiterinng),
(15)
where (sf, at) denotes the state-action pair at timestep f of trajectory т₁ Gxy represents a threshold
for selecting the top-K% of trajectories, with K chosen from {10, 20,50) 50) as practiced in [12]. V(s)) denotes the value of the initial state st in trajectory Ti, and the coefficient represents the temperature coefficient in a Boltzmann distribution. We consider three levels of ry: low (L), medium (M), and high (H) in our experiments. Further details of the hyperparameter setup can be found in Appendix A.4. For the following experiments, we implement our DW and the above baselines on the top of ofte-off-heearr offline RL algorithms: Conservative Q-Learning (CQL) [22], Implicit
7

---

Q-Learning (IQL) [19], and TD3BC [7]. Note that as AW can provide better initial sampling distribution to train the weighting function in DW, we initialize training DW with AW sampling (denoted as DW-AW) and initialize training DW with uniform sampling (denoted as DW-Uniformy in the following experiments. We refer the readers to Appendix A.3 for the implementation details.
Evaluation metrics. Following the settings of [6], we train all algorithm for one million gradient steps with three random seeds each dataset. We evaluate the performance of policies acquired through each method in the environment corresponding to the dataset by conducting 20 episodes every 1000 gradient step. To determine the policy's 's performance at given random seed, we compute the average returns over 20 episodes during the final 10 evaluation rounds, each round separated by 1000 gradient steps. We chose to average over the performance at the last 10 rounds rather than solely at the last evaluation round because we observed that the performance of offline RL algorithms oscillates during gradient steps. The main performance metric reported is the interquartile mean (IQM) [1] of the normalized performance across multiple datasets, along with its 95% confidence interval calculated using the bootstrapping method. As suggested in [1], IQM is a robust measure of central tendency by discarding the top and bottom 25% of samples, making it less sensitive to outliers.
5.2 Scenario (i): Trajectories with Similar Initial States
Figure 3a shows IQM of the normalized return for thirty-two different datasets where trajectories start from similar initial states (Section 5). For all the datasets, we use the best hyperparameterrs for AW and PF found in [12] and the hyperparameters for DW-AW and DW-Uniform are presented in Appendix A.4. The results demonstrate that both DW-AW and DW-Uniform outperform the uniform sampling approach confirming the effectiveness of our method. Moreover, combining DW with AW enhances the performance of DW, indicating that DW can benefit from the advantages of AW. This is likely because AW can provide a good initial sampling distribution to start training the weighting function in DW. While DW-Uniform did not exceed the performance of AW in this experiment, it should be noted that our method can be applied when datasets are not curated with trajectories such frsee-frre play style datasets where data is not collected in episodic manner. This is useful for continuing tasks, where an agent (data curator) performs a task infinitely without termination (i.e., locomotion).
Limited size datasets. Figure 3b presents the results on smaller versions of 8 of these datasets used in Figure 3a. Note that as we observe the higher temperature 1) enables AW with CQL to perform better in this type of dataset, we additionally consider AW-XH (extra high temperature) for comparison to provide AW with as fair a comparison point as possible. Further details on the hyperparameter settings can found in Appendix A.4. Our methods consistently achieve significantly higher returns compared to AW and PF when combined with CQL. This suggests that our methods effectively utilize scarce data in smaller datasets better than weighted sampling approaches (AW and PF) that returns rather than purely transition level optimization like DW. In
rely on episodic trajectory- the case of IQL and TD3BC, we see a clear performance improvement of our methods over uniform sampling and PF while our methods perform is because IQL is less prone to overfitting on small amounts of data due to its weighted behavior cloning objective [18), which always uses the in-distribution actions. However, it is worth noting that IQL falls short in performance compared to CQL with DW- Uniform.. This suggests that IQL may primarily focus on replicating behaviors from high-return trajectories instead of surpassing them, as it lacks the explicit dynamic programming used in CQL.
par with AW. For IQL., we hypothesize that this
Takeaway. Since CQL with DW-AW outperforms the other two offline RL algorithms in both dataset types, our suggestion is to opt for CQL with DW-AW, especially when dealing with datasets that might exhibit an imbalance and include trajectories originating from comparable initial states.
5.3 Scenario (ii): Trajectories with Diverse Initial States
Figure 4 presents the results on thirty-two datasets of trajectories with diverse initial states (Section 5). We observe that uniform sampling's performance drops significantly these datasets compared to trajectories with similar initial states, indicating that the presence of diverse initial states exacerbates the impact of imbalance. Both of our methods consistently outperform all other approaches considered in Section 5.2, including AW and PF methods. Notably, even the best-performing variant of AW
8

---

Uniform
PF-20%
AW-L AW-M
AW-H AW-XH
DW-AW (ours)
PF-10% 10%
PF-50%
DW-Uniform (ours)
normalized return
80
60
IQM
40
CQL
IQL
TD3BC
(a) Results on imbalanced datasets of trajectories with similar initial states (Section 5.1).
normalized return
40
30
IQM
20
10
CQL
IQL
TD3BC
(h) Results on smaller version datasets used Figure 3a.
Figure 3: (a) Our methods, DW-AW and DW-Uniform, achieve higher return than Uniform, indicating that DW can enhance the performance of offline RL algorithms on imbalanced datasets. Note that methods in IQL,, although not surpassing AW and PF-10% in performance,
can be applied to (b) Our methods outperform Uniform in CQL, IQL,, and TD3BC, indicating no significant overfitting in smaller datasets. DW-AW demonstrates superior returns compared to AW and PF, particularly in CQL, indicating our method effectively leverages limited data. IQL shows limited gains likely due its difficulties in utilizing data from the rest of low-return trajectories in the dataset (see Section 5.2).
offline RI. dataset that are not curated with trajectories.
(AW-M) falls short of matching the performance of our DW-AW, demonstrating the effectiveness of DW in leveraging the initial sampling distribution provided by AW and furthering its performance. The performance degradation AW can be attributed to the presence diverse initial states and varying trajectory lengths in these datasets. In such cases, state-action pairs in trajectories with high returns are not necessarily generated by high-performing policies. For instance, a sub-optimal policy also easily reach the goal and achieve high return if starts close the goal (i.e., lucky initial states). Consequently, over-sampling state-action pairs from high-return trajectories can introduce bias towards data in trajectories starting from lucky initial states. Although AW attempts address this issue by subtracting the expected return of initial states (see Section 5.1), our results show that AW has limited success addressing this issue. This is because the estimated expected returns of initial states can be inaccurate since AW uses the trajectories' returns in the dataset to estimate initial states' returns (i.e., Monte Carlo estimates). The trajectories in the dataset are finite in length, which makes the Monte Carlo estimates of expected returns inaccurate. To conclude, when an imbalanced dataset consists of trajectories starting from diverse initial states, we recommend using DW-AW to re-weight the training objectives in offline RL algorithms.
Related Work
Our approach builds upon recent advances off-policy evaluation techniques, specifically density- ratio importance correction estimation (DiCE) [32]. DiCE has been primarily used for policy evaluation [29, 32, 10], while our method make use DiCE (i.e., the learned importance weights) to re-weight samples for offline RL algorithms. Recent works [43, 37, 33, 26] optimize the policy using DiCE via re-weighting behavior cloning with DICE, while we found it fails to match offline RL algorithms' performance even in datasets with plenty of expert demonstration (Appendix A.5).
Offline imitation learning approaches [15, 30, 41] also consider imbalanced datasets similar ours. However, these methods assume prior knowledge of which data points generated by experts,

---

Uniform
PF-20%
AW-L
AW-H
DW-Uniform (ours)
PF-10%
PF-50%
AW-M
DW-AW (ours)
normalized return
60
IQM
40
20
0
CQL
IQL
TD3BC
Figure 4: Results on imbalanced datasets with trajectories starting from diverse initial states (Sec- tion 5.3). Compared to Figure 3a, the performance of uniform sampling and AW decrease, showing that diverse initial states exacerbate the issue of imbalance. Our methods, DW-AW and DW-Uniform, achieve higher return than all the baselines, which suggests DW is advantageous in broader types of imbalanced datasets.
while our approach does not rely on such information. Furthermore, our method can effectively handle datasets that include mixture of medium-levs level policies and now-performing policies, whereas existing approaches often rely on expert-labeled data.
Multi-task offline RL algorithms [42, 14] filter data relevant the current task of interest from datasets collected from multiple task. For example, Yu et al. [42] employ task relevance estimation based Q-value differences between tasks. While motivation aligns with data filtering, problem setting differs as we do not assume knowledge of task identifiers associated with the data points. Additionally, our dataset comprises varying levels of performance within the same task, while existing works mix data from different tasks.
Support constraints [20, 38, 2, 40] have been proposed as alternative approach prevent offline RL algorithms from exploiting out-of-distribution actions, distinet from distributional constraints used in state-of-the-art methods [22, 18]. While support constraints theoretically suit imbalanced data, the prior work [38] found that support constraints have not shown significant improvements beyond distributional constraint manserint--bsed algorithms. Note that our method is independent of the constraint used in offline RL algorithms. Thus support constraints is orthogonal to our approach.
7 Conclusion, Future Directions, and Limitations
Our method, density-ratio weighting (DW) improves the performance of state-of-the arr offline RL algorithms [22, 18] over 72 imbalanced datasets with varying difficulties. particular, our method exhibits substantial improvements in more challenging and practical datasets where the trajectories in the dataset start from diverse initial states and only limited amount of data are available. Future works can explore other optimization techniques to better address the Bellman flow conservation constraint importance weights optimization (e.g., Augmented Lagrangian method [37].. Additionally, would be valuable to study the impact of violating this constraint the effectiveness of importance- weighted offline RL algorithms.
Limitations. Although our method improves performance by optimizing sample weights, we lack theoretical guarantees due to the absence of unified theoretical analysis on the dependence of state- of-the-art offline RI. algorithms on imbalanced data distribution. While some theoretical works [4, 36] have analyzed the interplay between data distribution and offline RL algorithm performance, they primarily focus specific algorithms that differ significantly from the practical state-of-the-art offline RL algorithms.
10

---

Author Contributions
Zhang-We Wei Hong:
Led the project and the writing of the paper, implemented the method, and conducted the experiments.
Aviral Kumar: Advised the project terms of theory, algorithm development, and experiment
design. Revised the paper and positioned the paper in the field.
Sathwik Karnik: Prepared the datasets and proofread the paper.
Abhishek Bhandwaldar: Helpd scaling up experiments in the cluster.
Akash Srivastava: Advised the project in the details of the practical and theoretical algorithm design and coordinated the compute.
Joni Pajarinen: Advised the project in the details of the practical and theoretical algorithm design and experiment designs.
Romain Laroche: Advised the project in the theory of the algorithms and dataset designs.
Advised the project in terms of theory, algorithm development, and experiment design. Revised the paper and positioned the paper in the field.
Abhishek Gupta:
Pulkit Agrawal: Coordinated the project, revised the paper, and positioned the paper in the field.
A Acknowledgements
We thank members of the Improbable AI Lab for helpful discussions and feedback. We are grateful MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. This research was supported in part by the MIT-IBM Watson Al Lab, an AWS MLRA research grant, Google cloud credits provided as part of Google-MrT support, DARPA Machine Common Sense Program, ARO MURI under Grant Number W91 WNF-III-------
ONR MURI under Grant Number N00014-22-1-27400 and by the United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number 148750--1-2-- 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-2953
2021.
[2] G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
[3] S. P. Boyd and Vandenberghe. Convex optimization. Cambridge university press, 2004.
[4] Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799, 2020.
[5] I. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv: 2rXiv:2 106.01345, 2021.
[6] J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.0722 2020.
[7] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 24:20132-20144 2021.
[8] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint arXiv: 8 2,09000, 2018.
[9] Fujimoto, H. van Hoof, and Meger. Addressing function approximation error actor-critic methods. arXiv preprint arXiv: 100.11880..947,.2111

---

[10] S. Fujjmotoo, D. Meger, and D. Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. In International Conference on Machine Learning, pages 3518 3518-3529. PMLR, 2021.
[11] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arviv 1812.05905, 2018.
[12] Z.-W. Hong, R. Tachet des Combes, P. Agrawal, and R. Laroche. Harnessing mixed offline rein- forcement learning datasets via trajectory reweighting. International Conference on Learning Representations (ICLR), 2023.
[13] S. Kakade. Optimizing average reward using discounted rewards. In Computational Learning Theory: 14th Annual Conference on Computational Learning Theory, COLT 2001 and 5th European Conference Computational Learning Theory, EuroCOLT 2001 Amsterdam, The Netherlands, July 16-19, 2001 Proceedings, pages 605-615. Springer, 2001.
[14] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and
K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104. 01.222404000222 2021.
[15] G.-H. Kim, S. Sco, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
[16] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.6980, 2014.
[17] T. Kloek and H. K. Van Dijk. Bayesian estimates of equation system parameters: an application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pages 1-19, 1978.
[18] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06100 2021.
[19] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https: //tp////opee/eeee///mmm net/forum? incerrummiiiie 1d=:6n2292n222eeee
[20] A. Kumar, J. Fu, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. URL http://ttt/te //t.x//aivvvv//////////e//aaaava wrw/abs/////oeeooooo/o.ooeeeeeae abs/1906. .00999.
[21] A. Kumar, Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[22] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 178179889944..
[23] S. Lange. T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learning. pages 45-73. Springer, 2012.
[24] R. Laroche and R. des Combes. On the occupancy measure non-Markovian policies in continuous MDPs. In Proceedings of the 40th International Conference on Machine Learning, ICML 2023, Proceedings of Machine Learning Research. PMLR, 2023.
[25] R. Laroche, Trichelair, and R. Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learning (ICML oral EWRL UAI workshop). 2019.
[26] Lee, W. Jeon, B. Lee, Pineau, and K.-E. Kim. Optidice: Offlinn policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning. pages 6120-6130. PMLR, 2021.
[27] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[28] L. Li, W. Chu, J. Langford, and X. Wang. Unbiased offline evaluation of contextual-bandie based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 297-306, 2011.
12

---

[29] Q. Liu, L. Li, Z. Tang, and D. Zhou. Breaking the curse of horizon: aefinit-hhorizon off off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
[30] Y. Ma, A. Shen, D. Jayaraman, and O. Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14639-14663, PMLR, 17-23 Jul 2022. URL https: httpps///proceeingg..o mlr wwt.s/resss/e.
[31] P. Michel, T. Hashimoto, and G. Neubig. Distributionally robust models with parametric likelihood ratios. International Conference Learning Representations, 2022.
[32] O. Nachum, Y. Chow, B. Dai, and I.. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems,
pages 2315-2325, 2019.
[33] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: 2019.
[34] F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev. Time limits in reinforcement learning. In International Conference on Machine Learning, pages 4054. PMLR, 2018.
[35] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[36] P. Rashidinejad. B. Zhu, C. Ma, J. Jiao, and S. Russell. Bridging offline reinforcement learning and imitation learning: tale of pessimism. Advances Neural Information Processing
Systems, 34:11 1702-11717611 16, 2021.
[37] P. Rashidinejad, H. Zhu, K. Yang, S. Russell, and J. Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716. 2022.
[38] A. Singh, A. Kumar, Q. Vuong, Y. Chebotar, and Levine. Offline rl with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052 2022.
[39] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
[40] Q. Vuong, A. Kumar, S. Levine, and Y. Chebotar. Dasco: Dual-generator adversarial support constrained offline reinforcement learning. Advances in Neural Information Processing Systems, 35:38937-38949, 2022.
[41] H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminator-weighhee offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pages 24725- 24742. PMLR, 2022.
[42] Yu, Kumar, Chebotar, Hausman, S. Levine, and C. Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1150: 14:11501-1 2021.
[43] W. Zhan, B. Huang, A. Huang, N. Jiang, and J. Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730-270 2775. PMLR, 2022.
[44] R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072, 2020.
13

---

A Appendix
A.1 Additional Discussion on Imbalanced Datasets
We consider imbalanced datasets that mixes the trajectories logged by high- and low- performing behavior policies, where trajectories from low-performing behavior policies predominate the dataset. When dealing with such imbalanced datasets, the regularized offline RL objective (Equation 2) tends to constrain the learned policy π to stay close the overall mediocre behavior policy rather than the high-performing one, which is an unnecessary form of conservativeness. We illustrate this issue from intuitive and analytical aspects in the following.
Intuitively, for example, when considering a dataset D that mix the trajectories from two behavior policies TH and 75 where Th significantly outperforms T/ (i.e., (TTT) 2 J(xb)), if the dataset is imbalanced such that state-action pairs (st, a,) logged by TD predominate, the regularized offline RL objective will penalize deviation from To more than deviation from v.. This is because uniform sampling on the imbalanced dataset oversamples state-action pairs (s),a)) from The leading to an over-weighting of the regularization term on those state-action pairs (s,,a))
Analytically, the issue of unnecessary conservativeness (Section 3) can be explained by how the performance of offline RL. (Tr) is dependent the performance of the behavior policy J(TD), as suggested in [12]. In the case of an imbalanced dataset where the behavior policy can be regarded mixture of n and nh. the performance of the behavior policy J(TD) can be estimated by the expected return of the distribution of trajectory in the dataset D, as shown below:
N
((TD) se (G
Σ G(T.)
(16)
r(s), at) is the return (i.e., sum of rewards) of trajectory Ti, and T; is its length. Since the trajectories from nh dominate the dataset, the average return is mostly determined by low-return trajectories, leading to low J(TD). This prevents pessimistic and conservative algorithms [22, 7, 18, 20, 25] from achieving high (TT) with low J(TT). Both types of algorithms aim to learn a policy т that outperforms TD (i.e., J(x) 2 (TT)), while constraining the policy ı to stay close to aD [4]. Recent theoretical analysis by Singh et al. [38] on conservative Q-learning (CQL.) [22] (an offline RI. algorithm) shows that the performance improvements of the learned policy æ over the behavior policy is upper-bounded by the deviation of π from TD as shown in: ( ((T)) I ())() [C(s, a)]. where C' is a constant and C denotes the regularization penalty (see Section 2). This implies that regularized objective
where D(T.) denotes the probability mass of trajectory TÜ in D. := G(T+):
would constrain the policy π from deviating from TD. thus corroborating the findings in [12] despite the absence general theoretical guarantees for all offline RL algorithms.
Why our method improves performance on imbalanced datasets? Our importance reweighting method can be seen as a modification of the performance of the hehavior policy, which represents the lower bound in conservative and pessimistic offline RL algorithms. By adjusting the weights of state- action pairs sampled from the dataset, we simulate the sampling of data from an alternative dataset D',, that would be generated by an alternative behavior policy TD_ By maximizing the expected return J (TD.) of this alternative behavior policy TD.., we can obtain a RD_ that outperforms the original behavior policy Tz. i.e., ((TT))= 2 (TT).. As mentioned earlier, most offline RL algorithms aim to achieve a policy that performs at least as well as the behavior policy that collected the dataset. By training an offline RL algorithm using dataset D. indueed by reweighting with w, we obtain higher lower bound on the policy's performance, as J(TD.) 2 (TT).
A.2
Why regularizing the importance weightings with KL divergence does not prevent policy improvement?
In Section 4.2, we employ a KL divergence penalty denoted as DRI(Di| to impose penalty on the extent to which the weighted data distribution D., diverges from the original data distribution D. It is important to note that this penalty does not inhibit the learned weightings from potentially improving the data distribution for regularized offline RL algorithms. Essentially, our approach involves optimizing the data distribution for offline RI. algorithms first. Subsequently, these offline RL algorithms are subjected to regularization based on this optimized distribution, thereby further
14

---

enhancing the policy. This framework enables offline RL algorithms to deviate more significantly from the original data distribution.
To illustrate this concept, consider a scenario where our density-ratio weightings are constrained within KI. divergence margin with respect to the original data distribution. A regularized offline RL algorithm, which is also subject to an KL divergence constraint with respect to this weighted data distribution, can potentially improve its performance even when deviates by more than 2e from the original data distribution. It is worth noting that some readers might think this as being analogous reducing the regularization weight a. However, reducing (Equation 2) lets offline RL algorithms to determine how far they should deviate from the original data distribution on their own. In our experimental investigations, we have observed that reducing a often results in worse performance compared to utilizing our DW method, as evidenced by the results presented in Figure [small kl].
A.3
Implementation details
A.3.1
Offfine RL algorithms
We implement our method and other baselines on top of two offline RL algorithms: Conservative Q-Learning (CQL) [22] and Implicit Q-Learning (IQL) [18]. Our baselines, advantage-filtrinng (AW) and percentage-filtent filleringg (PF), are weighted-sampling methods (see Section 5.1). Therefore, they do
not require changing the implementation of offline RL algorithms. In the following sections, we explain the details of our importance weighting method for each algorithm.
CQL. We reweight both the actor (policy) and the critic (Q-function) in CQL as follows:
F (amm()))))) (s) (Q(s, a) log (a)s)))
(Actor)
(17)
min oF (,,a)~
Wo war(s,,) a) log Σ
e(p - Q(0.0))
(Critic)
(18)
Q
a'EA
wos (s, (r(s, + +nQ Q(s a))
min E EaDDa (w)(s) (- log T (als) H)]
(Entropy coefficient [11]), (19)
where H denotes the target entropy used in soft actor critic (SAC) [11] and we and Was denote
the weights predicted by our method (see Section 4). We follow the notations in CQL paper [22].
The implementation details of computing Equation 17 be found in [22]. Our implementation is adpated from open-sourced implementation: JaxCQL'.
We reweight state-value function (V), state-action value function (i.e.. Q-function, Q), and policy (r)) in IQL as follows:
IQL.
min E ((AA))DD (1)(() v(s))
(20)
min (SSA,NND
+) - v(s)) v(3))
(21)
Q
max E ((,)))D (s. (Q(s, V(s))) log [[ass)
(22)
where L denotes the upper expectile loss [18] and 3 denotes the temperature parameter for IQL.
Our implementation is adapted from the official implementation' for implicit Q-learning (IQL) [18].
We attach our implementation in the supplementary material, where CQL and IQL implementations are JaxCQL and implicit, puppicit_q_leet gelaning respectively.
hhups://githhbbbu hhans:/giihaah.mm/yommgggege hatp:///w...ommyyoooggooggnooCC https://gihuub.cm/ikststtttiiiit learning
15

---

A.3.2 Weighty- function
The following is the training objective of the importance weighting function Waye and we in our method, as described Equation 10 (Section 4.2):
max (s, 0,19
(o Ap - wa
AK (P (D)
Bellman flww conservation penalty
regularaizztion
The KL regularization term can be expressed follows:
D
(xI(PuelP)) = Σ D (s, a) log
(23)
()
9,a
Σ D(s, a) Du(s,a) Dw(s,a)
log
D(s,a) D(s.a)
(s, (s.
= (s.aa>D
log
D)
D(s,a)
=((,,)) (woss, a) log (ss, a)).
Thus, rewrite the training objective (Rquation 10) to the follows:
(s, 0,0
(s)()s)))(ssa)) -) ~p (wa(s') - ws (s, AK (s, a) log (s,
Return
comsevatiin penalty
regulurization
(24)
The objective in Equation 24 can be optimized by minimizing the loss function L(ø, v)) using stochastic gradient descent shown below:
L( vb) In(d, vb) (FIr((0
dx Ax
(25)
B
Ln(ó, v) =
1 (
(26)
B
LF( =
Σ
(
wa (si, az))²
(27)
B
i=1
B
Lx( = Σ
Ue (s, a) log Th (s,a),
(28)
d=1
where B denote batch size and (s;, ax) denotes the 7hh state-action pair in the batch. T (s a:) denotes the batch normalized weights predictions [31] that is defined as follows:
Ws (s,, 0.) = Wo.v (8i,
(29)
EB Wy Wa (s),
where Wa. (si, a;) denotes the importance weights predictions from the network ó and & (see Section 4.2). As [31] suggests, applying batch-level normalization on the importance weights predictions can make the normalization requirement of importance weights, i.e., E [w] = 1, more likely being satisfied during optimization than other approaches (e.g., adding normalization penalty term) (see [31] for details).
A.4 Evaluation detail
A.4.1 Training details
For both the baselines and our method, we train CQL IQL. for total of one million gradient steps. The hyperparameters used for CQL and IQL are set to the optimal values recommended in
16

---

their publicly available implementations. As for the training of our density weighting functions o, v,, we employ the same network architecture as the Q-value function architectures in CQL and IQL. which consists of two-layer Multilayer Perceptron (MIP) with 256 neurons and ReLU activation in each layer. To minimize the objective defined in Equation 25, we train & and v using the Adam optimizer [16] with learning rate of 0.0001 and batch size of 256. In each gradient step of CQL and IQL, we train & and v/ for one gradient step as well.
We conducted hyperparameter search in the datasets with diverse trajectories. For AW with CQL we searched temperatue vy: 0.01 (L), 0.1 (M, the best from the original paper), 1.0 (H). 5.0 (XH). For AW with IQL, we searched temperature 5.0 (XH). For PF in CQL and IQL, we searched K over 0.1, 0.2, and 0.5. The hyperparameter search results are all already presented in Figures 3a. For DW-AW, we use the temperature yj used in AW-M for CQL and IQI,, except for small datasets. small datasets (Figure 3b), we use the temperature used in AW-XH for DW-AW. For DW-AW and DW-Uniform, we searched Ax E {0.2, 1.0) and Ar E {0.1, 1.0, 5.03. We use the best found hyperparameter by the time we started the large scale experiments: (Ax, AF) (0.2, 0.1) for CQL and (AK,Ap) (1.0, 1.0) for IQL. We present the hyperparameter search results of DW-Uniform in Figure 2.
0.01 (L), 0.2 (M) (the best from the original paper), 1.0 (H),
A.4.2 Evaluation details
We follow the evaluation protocol used in most offline RL research [22, 7, 18]. Each offline RL algorithm and sampling method (including our DW approach) combination is trained for one million gradient steps with three random seeds in each dataset. We evaluate the policy learned by each method in the environment corresponding to the dataset for 20 episodes every 1000 gradient step. The main performance metric reported is the interquartile mean (IQM) [1] of the normalized mean return over the last 10 rounds of evaluation across multiple datasets, along with its 95% confidence interval calculated using the bootstrapping method. As suggested in [1], IQM is a robust measure of central tendency by discarding the top and bottom 25% % of samples, making it less sensitive to outliers.
Compute. We ran all the experiments using workstations with two RTX 3090 GPUs, AMD Ryzen Threadripper PRO 3995WX 64-Cores CPU, and 256GB RAM.
A.4.3 Dataset curation.
Following the protocol in prior offline RL benchmarking [6, 12], we develop representative datasets for Scenarios (i) and (ii) using the locomotion tasks from the D4RL. Gym suite.
Scenario (i): Datasets with trajectories starting from similar initial states. This type of datasets was proposed in [12], mixing trajectories gathered by high- and low-performing policies, as described in Section 3. Each trajectory is collected by rolling out a policy starting from similar initial states until reaching timelimit or terminal states. As suggested in [12], these datasets are generated by combining 1 - 0% of trajectories from the random-v2 v2 dataset (low-performing and a% of trajectories from the medium-v2 or expert-v2 dataset thigh-performinn D4RL benchmark. For instance, a dataset that combines 1 - a% of random and o% of medium trajectories denoted as random-med: um %% We evaluate our method and the baselines on these imbalanced datasets across four a E {1, 5, 10, 50), four environments. We construct 32 of this type of datasets in all the combinations of ( ant, halfcheetal hopper, walker2d) random-expoe expert) x fa E {1, 5, 10, 50) Also, we consider a variant of smaller versions of these datasets that have small number of trajectories, where each dataset contains 50, 000 state-action pairs, which is 20 times smaller. These smaller datasets can test if a method overfits to small amounts of data from high-performing policies. As every method fails to suprpass random policy when a = and a = 5, we only evaluate all the methods in = 10. Note that the results in = 50 is not different from the results in larger version of mixed dataset with o 50 since the amount of high-performing trajectories is still sufficient when 50.
for each locomotion environment in the
(random-mend um,
Scenario (ii): Datasets with trajectories starting from diverse initial states. Trajectories in this type of dataset start from a wider range of initial states and have varying lengths. This characteristic designed to simulate scenarios where trajectories from different parts of task are available in the dataset. One real-world example of this type of dataset is collection of driving behaviors
17

---

obtained from a fleet of self-driving cars. The dataset might encompass partial trajectories capturing diverse driving behaviors, such as merging onto highway, changing lanes, navigating through intersections, and parking, although not every trajectory accomplishes the desired driving task of going from one specific location to the other. As not all kinds of driving behaviors oceur with equal frequency, such dataset is likely to be imbalanced, with certain driving behaviors being underrepresented. We curate this type of datasets by adapting the datasets from Scenario (i). These datasets are created by combining 1 - a% of trajectory segments from the random-v2 dataset (representing low-performing policies) and 0% of trajectory segments from either the medium-v2 or expert-v2 dataset (representing high-performing policies) for each locomotion environment in the D4RL benchmark. Each trajectory segment is a subsequence of a trajectory in the dataset of Scenario (i). Specifically, for each trajectory T₁, trajectory segment is selected, ranging in length from 10 to 50 timesteps within т₁. We chose the minimum and maximum lengths to be 10 and 50 timesteps, respectively, based our observation that most locomotion behaviors exhibit intervals lasting between 10 and 50 timesteps. Since locomotion behaviors are periodic in nature, these datasets can simulate locomotion recordings from various initial conditions, such as different poses and velocities.
The datasets in Scenario (ii) capture different form of imbalance that is overlooked in the datasets of Scenario (i). The imbalanced datasets in Scenario (i) combine full trajectories from low- and nigh-performing policies, starting from similar initial states. These datasets capture the imbalance resulting from the policies themselves, while overlooking the imbalance caused by different initial conditions. Even when using the same behavior policy, the initial conditions, i.c., the initial states of trajectories, can lead to an imbalance in the dataset, as certain initial conditions tend to yield higher returns compared others. For example, an agent starting near the goal and another starting far away from the goal may achieve significantly different returns, even when following the same policy. This type of imbalance can exist in real-world datasets if certain initial conditions are oversampled during the dataset collection process.
Figure 5 presents the distribution of normalized returns for both types of datasets in both scenarios. The trajectory returns are normalized to a range of 0 to using max-min normalization. specifically (G(r.) - Gmin) mG minn), where Gmin and G denote the minimum and maximum trajectory returns in the dataset, respectively. From Figure 5, we observe that the two types of datasets exhibit different return distributions. The return distribution in Scenario (i) (i.e., mixed) is closer to bimodal distribution, where a trajectory is either from a high- or low- now-performing policy. On the other hand, the return distribution in Scenario (ii) (i.e., mixed (diverse)) follows a heavy-tailed distribution, where high-return trajectories are located in the long tails. This indicates that in addition to the imbalance resulting from the combination of trajectories from low- and tigh-performing policies, the presence of diverse initial states introduces another type of imbalance in the dataset, where initial states that easily lead to high returns are much less prevalent than others.
A.5 Additional results
We present the full results in total of 113 datasets in Tables 4 and 5, including original D4RL datasets [6], mixed datasets (Figure 3a, denoted as Mixed), mixed datasets with diverse initial states (Figure 4, denoted as Mixed (diverse)), and small datasets (Figure 3b, denoted as Mixed (small)). As [12] suggested, the original D4RL datasets are not imbalanced and thus weighted sampling and importance weighting methods perform on par with uniform sampling.
A.5.1 Four room example for trajectory stitching
To assess DW's trajectory stitching ability, we conducted an experiment in a didactic four-room environment [26].
See Figure 6 for the environment illustration. The agent starts from an orange initial state, traverses non-red cells, gaining +1 reward at the green goal, else zero. To test trajectory stitching, a suboptimal dataset with 1000 trajectories was generated, where none of each is optimal trajectory. Due to absence of optimal trajectories, up- peighiitting trajectories (i.e., AW and PF) with high-returns won't produce state-action distribution matching the optimal policy. Thus, if method can generate state-action distribution matching the one of the optimal policy, it indicates that the method is able to stitch trajectories because it can identify optimal state-action pairs leading to the goal even though those optimal state-action pairs are not observed in the same trajectory in the dataset.
Experiment setup:
18

---

delccaniom-mmddimmee 01--22
ant--andom--eellee med-um--mmmmmll 05w22
antrandee-mdduum-- 1-22
antrondom-meddme
20
Denetty
20
(divecsu)
mited (diverss)
15
(diverss)
5.0
maxed (diveras)
10
2.5
0.50
0.00
0.25
0.75
0.00
0.50
antorundom-aqpartee -22
ant-rondorm-eeqprr--- 05-x2
ant-rundoo--eqpar--0 1-12
ant-rundom-rye www.ndoo---eetrr.o 5u22
10
Deastty
(diverse)
20
mittdd (diverso)
20
(diverss)
mixed (diverso))
0.50
0.00
0.25
9.70
1.00
0.00
0.50
1.00
0.0
po-joe--rnddom-mmll 0.01.22
Beogper www.s-....... 0.05222
Iospeerdaandooe Seggerrraddommmddisee 1-12
hagger- wndddn-mmdiiiusseooo 5u22
75
60
60
(dtverse)
mitted (diverse)
(diverse)
20
mixed (diverse)
40
40
10
3.00
0.25
0.50
1.00
0.00
0.25
1.50
0.75
1.00
5,00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1,00
hogper-random 01.
hogpen-randon expert.0..05555 0,05-2
hopper-random expert--0 1o22
hopper randocs- expert.ppp......ee
30
100
50
(diverse)
40
50
20
10
0.00
0.50
0.00
1.00
0.50
1.00
0.25
0.75
halfcheetah candoes medium 0,01-22 balfchectah medium 0 v5
balfchectah rundoms medium 0.1
baifehhctahh randoms
mised
mised
Dencity
10
0.5
0.0
halfcbeetah random expert-0 0.00 ghalfsheethh expert- halfcheetah expert- 0.1-2
balfcheets random expert-o
40
mixed
mixed
mixed
20
(divwrse)
10
0.5
walkor2dl-mddome
01-42
wallenr2ddenndem -onerndde--memu------ 05-4
walker 2e-drredddnndm-mee 1-42
Dousky
150
100
mitedd
mixedd
100
50
50
25
2.00
0.25
9.50
0.75
1.00
0.00
6.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
wilk 25-rnnoo---eerrr----
150
miteed
100
mited
60
20
mised
Donsity
1nn
10
50
0.00
6.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Mocmallzed return
Normallsed returs
Normallzed reture
Normalized ceturn
Figure 5: Return distributions of mixed datasets with similar initial states (blue, Scenario (i)) and diverse initial states (orange, Scenario (ii).. Both types of datasets lead to different kinds of imbalance and return distributions. See Section A.4.3 for details.
Figures 6a, 6b, and 6e display the state-action distributions of behavior policy, optimal policy, and DW; the number above each plot is expected return under the state-action distribution. We see that both AW (Figure 66) and PF (Figure 6d fail to match the state-action distribution of the optimal policy in Figure 8(b), hence leading to suboptimal performance. In contrast, DW successfully approximates the optimal policy's s state-action distribution, confirming DW can identify optimal state-action pairs observed in different suboptimal trajectories and stitch them to optimal trajectories.
Results:
A.5.2 Comparison with OPtDICE
OptDiCE [26] and AlgaeDiCE [33], as well as our method, all involve learning importance weights for policy optimization. However, the usage and learning of importance weights vary across these
19

---

J=0.17
=
0.37
0
(a) Behavior policy
Optimal policy
(c) PF
(d) AW
(e) DW (ours)
Figure 6: The stationary state-action distributions of behavior policy (a), the optimal policy (b), percentage filtering (i.e., top K%) (c), advantage weighting (d), and density weighting (c). J denotes the expected return for each. The agent starts from the orange state and will receive the rewards at the green state (goal) and episode termination. Figure 6a is the empirical state-action distribution formed by 1000 suboptimal trajectories without optimal trajectory to the goal. Reweighting trajectories (i.e., AW and PF) cannot produce the optimal state-action distribution (Figure 6b) in the absence of optimal trajectories in the dataset. In contrast, since DW assign weights to transitions rather than trajectories, can up- up-weight good transitions suboptimal trajectories and stitch them to optimal state-action distribution. As we see, only DW (Figure 6е) successfully matches the optimal policy. This indicates that DW has the ability to stitch trajectories from suboptimal data distribution like Figure 6a while the other methods cannot.
methods. We compare our method with OptDiCE as it is the state-of-the-art method on policy optimization using DiCE. In the following, illustrate the difference between method and OpIDiCE.
Learning importance weights. OptDiCE learns the importance weights through solving the op- timization problem same as Equation in an approach different from ours. OptDiCE learns the importance weights through optimizing the following primal-dual objective:
#20 min max [(a.))D| ev a)w(s, a) - af(w(s, a))] + (1- pp
(30)
min E(o (eo (
(31)
(o00)o [M(s))],
where / can be any convex function (e.g., f(x) := x
log x),
(s),))))))..)))=== a) + +w((") v(s).
and x4 == max(0, x). The coefficient a denotes the strength of regularization: the higher the a is, the uniform the importance weights are. The learned importance weights w(s, a) are expressed in terms of y as follows:
w(s, a) = max (o. ( ( a, (0.l((((
(32)
On the other hand, our method learns the importance weights without solving the min-max optimiza- tion.
Applying importance weights. OptDiCE extracts the policy from the learned importance weights using information projection (I-projection) [26]. The implementation of I-projection is non-trivial, while its idea is close to weighted behavior cloning objective shown as follows:
max ((s,a)>D.. [log π [a|s)] = max E(s,a)~D [w]ss, a) log π (a]s)).
(33)
Differing from OpiiCCE, we apply the importance weights actor-critic offline RL algoritmhs like CQL [22] and IQL [18].
We present the performance of our methods in CQL and IQL, and OptDiCE in imbalanced datasets used in Scenario (i) in Figure 7, showing that OptDiCE underpeform all the other methods on imbalanced datasets and underperforms uniform sampling approach. This result indicates that even though OptDiCE learns the importance weights by solving similar objective with our method, OptDiCE not effective imbalanced datasets.
A.5.3 Comparison with weights learned by OptDiCE
While the policy learned using OptDiCE may not be effective on imbalanced datasets, it remains uncertain whether the importance weights Ieamned with OptDiCE can enhance the performance of
20

---

normalized retur
80
Uniform
PF-10%
60
IQM
AW
40
OptDiCE
DW-AW (ours)
20
DW-Uniform (ours)
CQL
IQL
Figure Results on imbalanced datasets of trajectories with similar initial states (Scenario (i) in Section 5.1). OptDiCE [26] fails to achieve high return on imbalanced datasets and even performs worse than CQL and IQL with uniform sampling. Note that OptDiCE a density- importance correction estimation (DiCE) based offine RL algorithm rather than sampling or importance weighting method. While both bars OptDiCE in this figure denote the same result, we plot the performance of OptDiCE alongside CQL and IQL with different sampling (or weighting) methods for comparison.
OptDiCE
OptDiCE
OptDCCEW 5.0)
hopper random expert
(.17.5)
39.1 (+18.3)
(-1688)
hopper-ran hupper-random-- random-exper expert-divereee 10%-vv
1,5 (:56.4)
29.7 (-28.2)
11.9 (-46.0)
saee---nnooo-mmi----iiie hopper- ronper--andom- medium diverse- 10%
(-58.5)
57.3 (-7.6)
2.2 (-52.9)
30.1 (-25.0)
41.0 (-14.))
(* training gets lerminated dus to NaN values)
Table 1:
method and that learned by OptDiCE approach. Each cell in the table denotes the mean return obtained by CQL trained with OptDCCE weights and the number in the parenthesis indicates the improvements (i.e., ScOTeOpDiCT: score over the performance of DW-Uniform reported in Figure 4. It shows that OptDiCE underperforms our
Comparison of the weights learned
method in most datasets (i.e,. negative improvements), and is sensitive to the regularization strength parameter a (see Section A.5.3).
offline RL algorithms. To investigate this hypothesis, reweight the training objective of CQL (Equation 17) using the importance weights learned with OptDiCE (Equation 32). We present a comparison with method in Table where OptDiCEW refers CQL trained with OptDiCE weights, and a represents the regularization strength (Equation 30), We examine different values of ranging from 0.1 to 5.0, but none of them consistently outperforms noting that each configuration results in significant performance loss in certain datasets. While this observation suggests that the importance weights learned using OptDiCE are not effective in improving the performance of offline RL algorithms, it is important to note that our work does not aim propose a new off off-policy evaluation method based DiCE. Therefore, a comprehensive comparison between different methods for learning importance weights is left for future research endeavors.
method. It is worth
A.5.4 Hyperparameter studies
In this section, we investigate the hyperparameters of KL regularization strength (Aş) and Bellman flow conservation strength (Ap). We present the average returns obtained in four specific imbalanced datasets in Table 2. These datasets are chosen because they exhibit significant performance differences between uniform sampling and other methods. Our observations indicate that higher penalties for flow conservation and lower strengths of KI. regularization tend to improve performance in both CQL and IQL.
A.5.5 Comparison of re-weighting both objectives and only regularization objective
In theory, the regularization term E(s.a)~D [C(s, a)] (Equation 2) the only component that can potentially harm performance imbalanced datasets, as it encourages the policy to imitate the suboptimal actions that dominate the dataset. However, our findings suggest that reweighting all the training objectives in offline RL algorithms leads to improved performance. In this section,
21

---

(Ax,
(0.2, 0.1)
(0.2, 1.0)
(0.2,5.0)
(1.0,0.1)
(1.0, 1.0)
(1.0, 5.0)
hopper-random. random expert -
20.4
13.8
22.1
9.6
7.5
hogper-raddon-eppert.. -et--in---eeee--iiieee-------- 10%---
$1.7
27.8
42.1
10.9
9.8
5.3
bopper-random-mooee
66.1
60.3
16.2
5.5
25.4
hopper-rundon tundon-meeiim- medium-diversee diverse- 10S-v2
55.4
58.9
65.3
3.6
7.2
7.3
(a) CQL
(Ax,
(0.2, 0.1)
(0.2, 1.0)
(0.2,5.0)
(1.0, 0.1)
(1.0, 1.0)
(1-0, 5.0)
56.4
54.2
75.1
suppee-randoo-cceerr-e peert-diverse-00-----
16.2
67.8
51.0
78.5
63.0
67.5
hopper-randons peoper-randommmedde mediin------------- diverse-59$-v2 555-v2
53.8
50.8
51.5
50.9
52.6
hopper random medium diverse 10%-v2
43.4
48.8
53.0
42.5
53.2
50.8
(b) IQL
Table Hyperparameter studies of our DW method in (a) CQL and (b) IQL. Xx and Ap denote the strength of re-regulaiization and Bellman flow conservation penatly, respectively (see Section A.3). Lower KL-regularization strength Ap and higher flow conservation penalty strength AF lead to better performance.
compare the performance of CQL trained with reweighted regularization only (referred to as "Reg. only") and reweighting all objectives (referred to as "All"), For the "All" approach, we train CQL using Equation 17, On the other hand, for the "Reg. only" approach, remove the term Wo,v (s, a) from the a training and the term (s) from the π training Equation 17. Table presents the mean return of both approaches, indicating that "All" exhibits slightly better performance compared to "Reg. only".
Reg. only
All
hooperrraadoo--pprrrrddvveee----
57.1
64.9
1.4
8.2
Table Reweighting terms Equation 17 shows better performance than reweighting only regularization term. See Section A.5.5.
22

---

Unifum
AW
1upezt-nunsko-122
9.2
7.9
7.8
6.5
hopper- ceterm-mmmm--mppreeeeee
104.5
104.5
101.8
02.9
100.5
101
102.1
10144
99.7
hopper medium V2
62.1
67.7
65.7
65.2
65.1
hoppersss parter2
107.2
108 2
1086
10K.0
105.
16.3
12.5
expert
71.2
baischetthh medium replay 12
46.7
42.3
44.3
hallche belleceecssillll-ggggg
75.1
76.7
750
76.3
77.1
46.5
46.5
45.4
46.6
46.5
MuJoCoo
hallchhemth expert-22
81.3
57.5
65.
22.0
57.1
meddenn----eeeee uns medium replay .22 ant- 2 mi--mddumm?
118.1
16.0
58.5
$2.7
95.9
U5.3
120
134.
127.5
128.6
120
1000
01.0
97.
expert walker
124.5
127.
6
15.2
walker walker Meneerenooleeeeeeeeee
10966
106
108.9
1094
189 7
walker 2umnodummm-lppo
24.4
78.1
71.
78.4
75.3
walker22 full replay
91.2
88.5
90.5
02.7
90.8
108
7511
697
anmmase umare diverse-00
48.0
16.0
248
40.0
34.0
D
atrmzze meduum-dirme-t
meSium play-u0
10.7
6.0
0.6
diverse
0.4
0.6
0.7
0.0
2788
10.2
9-3
955
16.5
DRRL Kitchen
Sitchen partal vo
25.0
36.0
50.8
54.0
20.0
Slichen
41.5
50.5
$2.0
469
penecloncllrl
-1.3
-2.5
8.6
-3.6
9.1
tammer humez v1 hammer clened-vi
-3.0
-7.0
6.9
-70
-7.0
Addit
docs-hemmn-v]
-1.0
-1.0
-511
7.0
-69
-9.4
-9.4
-9.4
relocsto humaaee
2.1
-2.1
-08
-2.1
-2.1
relecto-elmeezz
-2.3
-2.4
0.2
-2.1
-2.0
ats mando madiam 15--22
37.1
73.5
6.7
71.5
62.1
rundom mzdium
105--1
82.8
att--on--mmmmllmm -o-n:limm-55----
97.4
S4.5
95.3
97.4
97.3
ant nandmm expert 15 v2
10.0
77.7
58
65.1
42.4
ant nandom expert 3%,
25.2
114.8
65.0
115.5
96.3
152.7
115.6
rindom medram-
62.
57.2
68.7
hoppte random medram- 316-42
62.1
426
63.1
630
hojper--nat nan--m-mediam 10%--12
1.6
66.5
663
69.2
60.
suppas-nundenn man--m-mmmdiimmm
22.6
46.7
64.5
86
hoppre-mme mondom--sprtte 5
16.5
59-7
40.
107.5
14.9
100.T
466
100
85
Mixed
hopger random expent 40% V2
medium
100.6
37.1
109.5
108.5
105.7
39.6
84.2
halle beshchhl-raaaereee hall bechhl--annnm---ma molium 10%
43.9
44.6
45.8
45.
45.8
42.2
had hallcheerah random medium 50%6
25.6
46.5
45.1
45.5
45.5
hallcheetthh randems expert 18-12
21.4
26.4
31
18.9
18.8
hall castud--unluusscppr
24
66.2
69.
20.6
1098-22
754
fe Scherth random- expprrr----------- -056-22
56.7
50.7
86.3
52.8
wal watker 26-mnddm medum 155-22
2.9
41.9
33
51.5
0.8
waller 24 sundom medium 5%
0.0
75.0
463
75.2
15.9
walker midoo medium
50x
76.9
$3.0
70.5
walker dd-e-dmm-mmmeee comestpert- 8k-22
4.0
66.
57
99.1
-0.1
wa walker2d random expert dis v2
0.2
100.7
32.7
105.3
21.3
walker22 24 random expert 10%
3.1
108.1
38.3
108.6
0.
walker co pect------------
108.
105.8
96.2
nandmm medium diverse v2
53.5
85.4
39.5
85.1
76.4
ant minde medium diverse 10%-12
76.0
89.9
932
95.2
91.
unt att-mndo--mmle mediam-diiveree 500
101.9
91.3
95.7
97.7
102 4
aut-nundm expert-ivvens
33.5
13.6
expert
ant-nnndm dornt--iveee--i--- (05-v]
23.4
76.5
581
115.7
89 0
andom expert divense 50% v2
112.5
116.3
121.3
125.8
1236 9
hopper midom medium diverse
21-1
325
61.1
158
24.5
58.5
65.7
25.0
1009--11
ranom medium diverse 50%
52.4
62.0
61.5
63.2
hopper random expert divense 19
12.0
20.0
36
21.1
4.3
10.3
27.7
600
37.0
hurpeat-munkneee www.eeent---neeeeee... 10
61.1
60.9
M0.3
(diverse)
hoppoe
heethhhhhhh randum medium-
expert
19--2
39.6
41.2
327
25.0
hestchh candets
44.5
44.2
46.
39.8
45.5
balkhhetthh randucs mednum--ieees 109--22
43.7
45.3
425
45.2
46.2
46.8
46.8
170
haidchertal randoms
29.4
35.4
106
25.7
hal hatfchertth randoms expert divene 10%
24.1
29.5
20.3
1644
18.
hall hecthllraddmmsespee eep.eseeeee------------- 20
51.3
69.1
86
$2.2
21.2
walker 3leemmmmmmmmmimmm
medium
walker22 random
divense 105-42
24.7
523
75.5
21.7
walker 2leendom- aesemmmmmmmmmaaem daluddiis diverse 50%
76.9
800
77.5
78.4
79.0
walker Si-nndan- ex expert-diss
18-22
12.3
6.1
9.1
97
walker&d 2i-neemm-- expert diverse
37
85.3
2i-nnmmmmmmmm expert
33.9
walker
diverse 50%6-12
95.9
munkon muuumm T0
2013
12
191
4335
att-nndem-- 1omt--19- -0s--mill-2
62
14.9
68
11.8
20.7
hupprr-mmdkmmee moneemmeiimmmmmmmm.......... (0l-lelllec)
39.
466
51.5
(small)
horper
haischetthh randocs medium 10% small
expert
11.2
10.1
18.
25.4
3.3
3.5
32
4.5
4.0
walker 2lenndot-teiiin 102-mnal-e2
2.2
0.7
066
41.2
42.7
wa random expert (0% small vi
13.6
0.0
0.2
38.5
56.7
Table 4: Full results of average returns of CQL in total of 113 datasets.
23

---

Unifum
AW
1upezt-nnnsko-122
7.6
6.5
9000
6.4
hopper ceterm-mmmm--mprreeeeee
111:1
11188
1108
81.0
118.1
102.1
35.3
107.5
99.8
hopper medium V2
65.7
58.3
64.5
61.7
62.5
180
111.0
110.3
105.2
108 3
12.7
108
expert
baischetthh medium replay 12
64.0
44.0
29.4
44.1
66.6
hallche belleceecsiilllllggggg
73.5
76.3
72.3
76.4
75.9
hadlche maifebeetbennniiinne
47.5
47.8
45.4
47.9
47.7
MuJoCoo
hallcheerth expert-22
94.9
50.3
738
05.2
meddinn----neee uns medium replay .22 ant- ont-full-rrpop--2
129.3
1301
93.8
$2.9
71.4
95.8
89.8
130.
179.9
178.0
131.4
180
09
expert walker
126.2
125.0
122.3
10.4
walker walker Memollume
110.1
1007
1998
199.8
1893
walker 2umnooumm-plppo
81.3
47.0
42.3
62.6
65.1
walker22 full replay
86.8
84.5
35.6
80.6
95.0
109.5
109
101
anmmase umare diverse-00
87.3
78.3
0.6
22.0
61.0
D
atrmzze meduum-dirme--
meSium play-v0
76.0
22.0
0.0
70.0
78.7
diverse
36.7
20.7
40.0
41.3
9.5
0.6
10.0
42.0
61.8
16.3
101
1938
10000
DRRL Kitchen
sitchen partal vo
47.7
73.2
72.3
06.3
57.0
Slichen
47.5
8014
74.1
penecloncllrl
82.9
80.2
53.8
84.4
91.5
tammer humez v1 hammer clened-vi
3.1
0.5
3.2
0.8
1.2
Additt
docs-hemm--v]
2.3
1.4
relocsto humanee
0.5
-00
0.1
relecto-clmelzz
-0.0
0.1
0.6
0.0
-0.0
ats mando madiam 155-92
17.5
$6.0
311
55.5
55.3
rundom mzdium
$3.3
13.4
19.3
105--1
82.0
att-oon--mmmlllmm- ---mm-la----------
93.7
101
96.8
94.3
98.9
ant nandmm expert 15 v2
13.7
28.5
35
31.5
43.5
ant nandom expert 3%,
26.3
100.9
33
05.2
105 4
73.7
140
1150
rindom medram-
52.2
56.1
424
56.5
S1.7
hoppte random medram- 516-42
5900
57.1
634
65.2
59.8
hojper--nai mond---dddimm 10%5-12
63.2
$7.1
65.3
63.6
62.8
suppas-nundenn nun--m-mmddimm-50
612
619
hoppre-met mondom--sprtte 2
22.7
111
26.9
110.0
23.4
7
111S
33.0
1106
3
Mixed
hopger random expent 40% V2
medium
88.0
111.
92.2
109.
105 L
hallel beshchhl-raaaereee hall bechal-annnnm--maa molium- 10%
30.1
43.2
11
40.3
43.1
45.3
45.0
45.
had hallcheerah random medium 50%6
45.5
47.3
43.
47.1
46.6
hallcheetthh randems expert 18-12
4.2
3.8
23
2.8
3.6
hall castud--unluusscpar
74.0
1098-22
17.0
81.5
K7.1
70.6
fe Scherth random- expprrr----------- 5056-22
83.7
3200
94.4
93.8
wal walker 26-mndom medum 155-22
54.6
45.4
39.4
40.6
8
waller 24 sundom medium 5%
66.2
62.5
473
67.8
67.
walker midoo medium
65.8
62.8
74.3
50x
70.7
walker dd-emdmm-mmmeee com-etpert- 8k-22
20.0
9.6
11.
9.8
35.9
wa walker2d random expert dis v2
25.3
108.5
93.5
104.3
65.0
walker22 24 random expert 10%
64.4
106.1
107.
1091
58.1
walker eo pact------------
109.3
109
naddmm medium diverse 42
29.1
83.7
26.8
88.1
23.3
ant minde medium diverse 10%-12
61.0
91.8
58.
89.3
89.3
unt att-mndo--mmll mediam-diveese 500
91.2
96.0
890
97.0
96.
aut-nundm expert-iivens
3000
18.1
20.7
expert
ant-nnndm donnt--ieee---i--- (05-v]
27.9
101
22.9
1005
93.8
andom expert divense 50% v2
101.9
123.3
1228
127.7
126.9
lopper midom medium diverse
57.0
48.2
47.4
54.9
630
46.7
$1.2
ranom medium diverse 50%
51.2
48.7
536
54.0
$2.1
hopper random expert divense 19
10.5
14.5
83
22.5
17.4
17.1
50.0
314
78.7
410
hurpeat-munkneee www.eeent---neeeeeee. 11
33.7
103.1
63.0
(diverse)
hoppoe
heethhhhhh randum medium-ciwenn 19--2
expert
14.6
2.7
80
16.4
hestchh candets
34.9
20.1
17.1
2.3
15.2
halkhhetthh randucs mednum--ieees 109-v2
39.7
39.7
2410
30.9
24.3
44.6
466
8.8
haidchertal randams
8.5
10.3
hal hatfchertth randoms expert divene 10% v2
9.0
16.4
43
13.2
28.0
hall hecthleaaddmmseepee eep.eseeeee------------- 50
66.4
70.7
-08
72.9
85.0
walker 3leemmmmmmmmmimmm
38.5
61.
89.8
medium
walker22 random
divense 105-42
58.
58.9
592
66.2
60.5
walker 2leendon- aememmm-mmmmaaai dis 50%...2
668
49.4
746
63.7
64.0
walker Ei-endaan- ex expert-diss 18-22
10.5
13.0
16
31.0
26.4
walker&d 2f-neemm- expert diverse
1922
15.1
77
65.4
76.4
2i-nnmmmmmmmm expert
walker 24
diverse 50%6-12
64.1
106.8
153
105.1
munkon muuumm T0
33.4
21.7
att-nndem-- 1omt--19- -0s--mill-2
K.1
17.8
4.
24.0
13.9
hupprr-mmskmmee moneemmeiimmmmmmmm......... (0llleollec)
11.0
51.7
49.5
39.0
(small)
horper nadom expert
haischetthh randocs medium 10% small
15.0
22.6
14.6
2.4
3.1
-19
3,2
2.2
walker 2lennnot--nncinn 102-mnal-e2
1.6
34.1
11.
37.6
R4
walker2d random expert (0% small vi
0.5
2.8
0.2
4.8
1.7
Table 5: Full results of average returns of IQL in total of 113 datasets.
24

---

Unifum
AW
1upezt-nnnnko-222
9.0
7.5
6.4
hopper ceterm-mmmm--mprreeeeee
105 5
10h-3
105.8
97.5
70.4
102.5
103.3
75.5
hopper medium
59.8
63.5
65.1
65.6
57.9
hopperess partey2
110.5
111.
101.5
1100
182.5
11.3
10.3
12.0
expent
baischerthh medium
12
64.7
296
45.0
46.9
74.1
77.7
75.4
75.7
75.2
43.4
48.5
48.3
45.0
48.3
MuJoCo
hallcheemth expert-22
00.0
50.5
78A
97.8
97.8
150.5
medium replay
196.1
10019
30.0
109.
M0.3
ant-fa frpp--p mnn-mdiumm))
136.3
139.7
131
199.0
134
130.
115.0
1190
expert walher
123
1167
255
walker walker Meneerenooleeeeeeeeee
110.0
110.2
110.5
110.3
1100
walker 2uenooumm--pppo
80.2
90.8
72.4
83.1
77.4
walker22 full replay
93.3
96.2
96.0
05.7
91.6
110.2
130.3
110.
17.
32.33
RSD
anmmase umare diverse-o0
64.7
67.3
29.0
06.3
40.3
Antnere
atrmzze medum--dinme-t
meSium play-u0
3.7
10.7
diverse
0.0
0.
0.6
0.0
000
05
000
000
DRRL Kitchen
Sitchen partal vo Slichen
0.0
9.6
17.8
24.5
2010
penecloncllrl
11.9
2.2
17.1
-1.1
33.4
tammer humez v1 hammer clened-vi
1.1
0.8
0.4
1.2
1.3
Addit
docs-hemm--r]
0.4
0.3
-0.3
-1.3
-0.3
relocsto humaael
-0.5
-0.3
-0.3
-0.3
relecto-clmeezz
-0.3
40.2
-0.3
-0.6
-0.4
ats mandom madiam 15-92
41.2
21.5
11.
21.6
47.2
rundom mzdium
105--1
55.6
75.2
att-omn--mmmllimm- -n-mcllam-S----- Stri-d
85.2
114.5
111.5
120.0
108
ant nandom espert 15 v2
18.7
14.0
0.3
15.6
16.7
ant nandom expert
27.8
39.5
28
47.6
21.0
65.9
1366
79.5
31.6
rindom medram-
134
51.5
20.3
hoppte random medram 3t6-mm-----
48.2
58.0
498
S6.
85.8
hojper--nat random-mmee sm-mediam- 1055-12
29.4
56.4
5U5
53N
55.9
suppes-nundenn mww.-mmmmmmiimmmmmmm
64.7
53.7
61.
51.9
hoppre-met mondom--sprtte -
31.0
376
93.5
676
conjere-nnnmmmmmmeer 1055.y2
52.0
100 T
84A
91.3
95.4
Mixed
hopger random expent 40% 92
medium
87.2
100.5
1075
110.3
108.
hallel beccctl-aandmm--mn hall bechhh--annlt---ma moliumm 10%
21.0
36.2
47.8
478
48.3
46.8
hal hallcheerth random medium 50%
43.5
49.3
480
48.7
48.2
hallcheetth randecs expert 18-12
277
3.5
82
14.3
hall castud--usluusscpar
58.8
25.0
TH.4
70.9
fe Scherth random- expprrr----------- -0
85.8
96.0
700
95.2
918
wal walker 26-mnddm medum 155-22
6.0
40.
18
20.0
5.2
waller sundom medium 5%5
14.5
74.7
18
70.5
1.7
walker midoo medium
74.2
10.
walker 2d-nndmm-- comeeteert- 18-02
15A
0.1
4.2
-0.9
wa walker2d random expert dis v2
6.4
73.1
35
109.9
walker 24 rundom expert 10%
110.1
13
110.0
277
walker eo pect-------------
nandmm medium diverse 42
32.6
76.8
1038
97.1
ant mindom medium diverse 10%-12
64.5
108 3
38
95.7
48.0
unt att-mnnk---mll mediam-divesee 500
95.7
118.5
1108
115.8
107.2
aut-nundm bom expert-ivvens
23.7
24.5
expert
ant-nnddm donnt--ivere------ (05-v]
28.9
76.6
34
85.0
81.2
andom expert divense 50% v2
25.2
09.5
63.
87.0
76.0
lopper madom medium diverse
25.9
16.2
24A
37.1
43.7
37.9
1079--2
29.7
ranom medium diverse 50%
54.
50.3
508
48.0
47.
hopper random expert divense 19
15.2
13.7
132
32.6
10.3
11.7
57.6
37.
84N
SR
hurpent-munkneee www.eeent---neeeeee... 10
76.6
75.
86.6
hoppoe
expert
50%
heethhhhhh randum medium-
19-42
39.5
27.7
53.5
hestchh candets medium--deeeeee S
18.3
36.8
460
45.9
44.8
balkhhetthh randecs avves 10
20.4
23.2
466
47.4
45.5
189
484
45.0
haidchestal randams
15.9
12.1
458
62.3
hal hatfchertth randoms expert divene 10%
6.3
13.5
123
64.6
80.2
hall hecthllaaddmmmeepee eep.eseeeee------------- 20
82.1
49.5
14.2
86.3
93.4
walker 3leemmmmmmmmmimmm
medium
walker22 random
divense 105-42
10.8
1500
walker 2leendom-- aesemmmmmmmmaammm diverse 50%...2
12.1
47.4
78.
132
9.9
walker 2i-ennan expert-divs divn 16-22
4.4
2.1
13
555
walker&d 2i-neemm-- expert diverse
11.3
8.3
-0.
expert
walker
diverse 50%5-12
158
munkon muuimm T0
2010
1722
aet-nnddm--n 1omt--19- -0s--mlll-2
03
15.2
04
18.2
17.1
huppar-mmdkmmee moneemmmmimmmmmmmm.......... (0l-seollec)
46.7
20.3
(small)
horper
haischetthh randocs medium 10% small
expert
21.5
11.9
6.
20.4
25.2
65
5.4
-1.1
6.8
7.3
walker 2lenmnot--nncinn 102-mmalle22
1.8
13.9
110
5.4
6.3
2a random expert (0% small vl
0.3
0.2
0.8
1.6
Table 6: Full results of average returns of TD3BC in total of 113 datasets.
25


Reviews:
[{"review_id": "hPvNWmkoJq", "reviewer": "Reviewer_WKvr", "summary": "The authors consider an offline RL problem where the offline dataset has a small number of high-reward episodes and a larger pool of low-reward episodes. They argue this setting is fairly common in reality, since generating high-reward episodes is often higher effort.\n\nConsidering offline RL objective functions that are structured like \"maximize reward while staying close to the behavior policy\", the argument is that staying close to the behavior policy causes optimization to be too conservative / biased too much towards the large pool of low-reward episodes. Borrowing the DiCE techniques from off-policy evaluation, the proposed method is to learn importance weights to bias the \"stay close to data\" objective towards the high-reward (expert) trajectories over the support of the original offline dataset. The learning of these weights is done standard to the DiCE style techniques, although in practice the importance weights are given a KL constraint to stay close to the original dataset to avoid collapsing the distribution to a few rare examples, and the importance weights are also applied to the reward-maximizing terms in offline RL (even though theoretically this should not be required).\n\nThis is then compared with two other approaches that bias the offline distribution towards higher return episodes - sampling based on the advantage of the offline episode, or sampling the top K% of the offline data. The best results are found by combining advantage weighting with density ratio weighting", "questions": "Could the authors comment on how the importance weights $w(s,a)$ evolve over time? To me the most dangerous outcome is that $w(s,a)$ learning becomes unstable, or training overfits too heavily due to upsampling a smaller section of the data. Is there any way to compare how the weighting of $w(s,a)$ compares to methods like top K%, in how much they consider different examples in the data?", "limitations": "Seems fine.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I have some objections about claiming that offline RL methods have not considered the downsides of constraining to poor return trajectories. (More on this later). But in general, I agree that many datasets often have a smaller number of good examples, importance weights to adjust the conservatism penalty make sense, and using methods from the OPE literature is a reasonable way to learn those importance weights. The evaluation is also extensive enough for me to trust the results.", "weaknesses": "There have been methods that attempt to constraint the policy to the support of the offline dataset, rather than the behavior of the offline dataset. BEAR and BRAC come to mind due to using Kernel MMD to measure policy divergence (and they ought to be cited.) In practice these methods have underperformed CQL and TD3+BC so I think it is okay to benchmark against just CQL, but it is still a notable omission.\n\nI also suspect that the KL penalty is one of the more important hyperparams, would need to be tuned separately per dataset (as the acceptable level of deviation will change depending on dataset composition), and there is not much guidance on how to set this parameter. This weakness is common to many offline RL methods though (i.e. the weight of CQL penalty also needs to be tuned).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s valuable comments and appreciations on our extensive evaluation. The following address the reviewer’s questions.\n\n> There have been methods that attempt to constraint the policy to the support of the offline dataset, rather than the behavior of the offline dataset. BEAR and BRAC come to mind due to using Kernel MMD to measure policy divergence (and they ought to be cited.) In practice these methods have underperformed CQL and TD3+BC so I think it is okay to benchmark against just CQL, but it is still a notable omission.\n> \n\n**Answer:**\nThank you for your suggestion! We will cite BEAR and BRAC in the updated manuscript. Meanwhile, we added BEAR as a baseline in Figure 8 in the attached PDF. We ran BEAR based on this public codebase (https://github.com/takuseno/d3rlpy). The results show that both DW-Uniform and DW-AW with CQL and IQL outperform BEAR by a large margin in imbalanced datasets. This indicates that support constraint implemented with kernel MMD may not be able to impose support constraint nicely, hence leading worse performance than CQL with uniform sampling.\n\n> Could the authors comment on how the importance weights $w(s,a)$ evolve over time? To me the most dangerous outcome is that $w(s,a)$ learning becomes unstable, or training overfits too heavily due to upsampling a smaller section of the data. Is there any way to compare how the weighting of $w(s,a)$ compares to methods like top K%, in how much they consider different examples in the data?\n> \n\n\n**Answer:**\n- **Does $w(s,a)$ learning become unstable?** We observed the importance weight $w(s,a)$ converge in the early stage of training (~10,000 gradient steps, which is 1% of training steps). Figure 9 presents the data’s weights at different training epochs, indicating that the weights converge after ten epochs. This means that the weights learned by DW are stable during training rather than oscillating over time.\n- **Does training overfit a small section of the dataset?** Since DW outperforms uniform sampling by a large margin, overfitting may not be a severe issue. Even in the small dataset (Figure 2b) that is prone to have overfitting issues, DW outperforms uniform sampling, which suggests that overfitting is less of a concern.\n    \n    **DW can assign finer-grained weights per transition than AW and top K% (i.e., PF in our paper).** We visualize the weights generated under dataset `walker2d-random-medium-small-v2`, where DW outperforms the baselines. See Figure 9 for the plot of the evolution of weight over time and the weights of AW and PF. The color at each index denote the weight of a transition at a particular dataset index (horizontal axis). The left partition (i.e., left of the red line) store the data from medium policy, and the others are data from random policy.\n    \n    - **For AW**, we see that only one block of transitions receives weights. It means that all the weights concentrate on the same trajectory. As such, AW is prone to overfit a few trajectories in the dataset.\n    - **For top K%**, it weights each trajectory from medium policy uniformly with the same weight. This may not suffer from overfitting since it’s close to uniform sampling, but it may not improve over uniform sampling a lot.\n    - **Our DW**, in contrast, can assign finer-grained weights per transition, avoiding biasing to only one trajectory and improving the data distribution. This may explain why DW outperforms the baselines in this case. Moreover, we see that the weights of DW converge after epoch 10 (each epoch consists of 1000 gradient steps), which means that the weight evolution is stable.\n\n\n> I also suspect that the KL penalty is one of the more important hyperparams, would need to be tuned separately per dataset (as the acceptable level of deviation will change depending on dataset composition), and there is not much guidance on how to set this parameter. This weakness is common to many offline RL methods though (i.e. the weight of CQL penalty also needs to be tuned).\n> \n\n**Answer:**\n- **Per-dataset KL penalty tuning:** We found that per-dataset tuning is not required. DW outperforms the baselines in imbalanced datasets using the same KL penalty weight across all the datasets. This means that in addition to the hyperparameter of the base offline RL algorithm, we did not observe DW needing any more tuning. \n- **Guidance on setting KL penalty weight $\\lambda_K$:**  We suggest using a high KL penalty ($\\lambda_K$) for datasets with limited coverage in the state-action space, like expert or medium datasets in D4RL. This prevents undesired weight assignments on state-action pairs that might lead to out-of-distribution next states. If such data get high weights, training can lead to error accumulation and poor policy performance in offline RL [1]. However, setting high $\\lambda_K$ does not hinder policy improvement by DW, as datasets with limited coverage also offer fewer opportunities for theoretically-possible maximal policy improvement due to limited rooms for stitching trajectories covering different regions in the state space.\n    \n    [1] Liu, Yao, et al. \"Provably good batch off-policy reinforcement learning without great exploration.\" *Advances in neural information processing systems* 33 (2020): 1264-1274."}, {"review_id": "b6iIPrzb3x", "reviewer": "Reviewer_yys4", "summary": "This paper proposes a method to improve offline reinforcement learning (RL) performance on imbalanced datasets, where most of the data comes from low-performing policies and only a few from high-performing ones. The method, called density-ratio weighting (DW), optimizes the importance sampling weights to emulate sampling data from a data distribution generated by a nearly optimal policy. The paper shows that DW can enhance the performance of state-of-the-art offline RL algorithms on 72 imbalanced datasets with varying types of imbalance.", "questions": "- Though obtaining a performance bound is quite difficult as describing the imbalanced dataset is hard, is it possible to prove the optimization with the proposed loss function will converge? For example, I guess, derive an iteration equation of $w$ as the Bellman equation, and prove the operator is a contraction mapping.\n- In Section 6 related work, the paper discusses offline imitation learning approaches. It claims that these approaches \"assume prior knowledge of which data points are generated by experts\". However, there are some offline imitation or RL methods which only use the reward signals in the dataset and filter out low-quality data to improve the final performance, such as BAIL [1] and COIL [2]. I am curious about what the advantages of DW are compared to them, especially for BAIL which does not need trajectory information. Also, it will be better if you can try BAIL on your imbalanced dataset and show the results.\n\n[1] Chen, Xinyue, et al. \"Bail: Best-action imitation learning for batch deep reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 18353-18363.\n\n[2] Liu, Minghuan, et al. \"Curriculum offline imitating learning.\" Advances in Neural Information Processing Systems 34 (2021): 6266-6277.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It proposes a novel method of DW that optimizes the importance sampling weights to emulate sampling data from a data distribution generated by a nearly optimal policy, rather than the original dataset.\n- It demonstrates overall performance gains over SoTA offline RL algorithms and other baselines on imbalanced datasets with varying types and degrees of imbalance.\n- Its written style is clear and easy to follow.", "weaknesses": "- The theoretical analysis is relatively weak.\n- Slightly lack of discussion with a type of offline RL algorithms which filter out low-performance trajectories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novelty of our work, extensive evaluation, and clarity of our writing. We clarify the reviewer’s questions in the following.\n\n> Though obtaining a performance bound is quite difficult as describing the imbalanced dataset is hard, is it possible to prove the optimization with the proposed loss function will converge? For example, I guess, derive an iteration equation of $w$ as the Bellman equation, and prove the operator is a contraction mapping.\n> \n**Answer:**\n\nNote that as we don’t use fixed point iteration (i.e., not using target network of $w$) like value iteration to learn $w$, the update operator doesn’t have to be contraction mapping for convergence. Regarding convergence, the optimization objective of $w$ is a convex optimization problem over $w(s,a)$. We use gradient descent to optimize w. According to the standard results in Boyd et al. [1], gradient descent converges as long as the optimization domain of $w$ is bounded, the gradient w.r.t., the loss $\\nabla_w L(w)$, is bounded, and the learning rates are not too large asymptotically: $\\sum_t \\alpha_t^2 < \\infty$. All three conditions are met in our case. Thus optimization of w will converge too. \n\n[1] Boyd, Stephen P., and Lieven Vandenberghe. *Convex optimization*. Cambridge university press, 2004.\n    \n\n> In Section 6 related work, the paper discusses offline imitation learning approaches. It claims that these approaches \"assume prior knowledge of which data points are generated by experts\". However, there are some offline imitation or RL methods which only use the reward signals in the dataset and filter out low-quality data to improve the final performance, such as BAIL [1] and COIL [2]. I am curious about what the advantages of DW are compared to them, especially for BAIL which does not need trajectory information. Also, it will be better if you can try BAIL on your imbalanced dataset and show the results.\n> \n**Answer:**\n\nThank you for your suggestion! We’ve added BAIL as a baseline and presented the results in Figure 7 in the attached PDF. The results show that DW performs better than BAIL. We are currently running COIL as well and will update the manuscript accordingly when we have the results. Here is the BAIL codebase used in our experiment: https://github.com/lanyavik/BAIL."}, {"review_id": "pQEYcgGrG3", "reviewer": "Reviewer_qcFR", "summary": "Authors propose a new method for weighting the samples from datasets in order to mimic the sampling from dataset which is collected by the better policy the the  behavioral policy. This method can be integrated into any algorithm and combined with previous approaches.", "questions": "What are the avereged scores across domains in tables 4 and 5? As mentioned in \"Weaknesses\" it seems like the proposed method does not benefit a lot (if benefit at all) when applied to the original D4RL datasets. Could you please add those numbers to see what is the situation on average? I mean average scores over locomotion, antmaze, kitchen, adroit, generated datasets and averaged scores over all of the domains.\n\nHow does the method affect algorithms time needed for training?", "limitations": "Beside the uncetainty whether approach helps when dataset is not imbalanced that hard there is another limitation in design. If I understand approach correctly, it can't be applied to the datasets which require trajectories stiching to complete task. The example of such task is AntMaze and authors' approach mostly decrease performance when applied to those datasets.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Method seems to be not that hard to implement and it shouldn't introduce a notable computational overhead while can be applied with the arbitary algorithm.On imballanced locomotion datasets generated by the authors proposed approach helps to achieve much better results than other approaches. Benchmarking over high number of  the datasets is another  strength.", "weaknesses": "While approach helps to improve performance when D4RL locomotion datasets are mixed with the random datasets, it seems like there is no advantage  when applied to the original mixed datasets (medium-replay, medium-expert, full-replay) or the performance even drops on those if we look at Tables 4 and 5. \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our extensive evaluation and simplicity of implementation. In the following, we answer each of the reviewer’s question.\n> Could you please add those numbers to see what is the situation on average? \n> \n\n**Answer:**\n\n**Advantage in the original mixed datasets:** DW performs similarly with AW because the original mixed datasets (i.e., medium-expert, medium-replay, full-replay) in D4RL already have enough proportion of good data. The mixed datasets used in our experiment (Figure 1) only have an average normalized trajectory return of 0.08, but the original mixed datasets in D4RL have 0.5, which is far higher than ours. As the average return of datasets indicates the performance of behavior policies, this means staying close to the behavior policy won’t hurt the performance much, which explains why reweighting doesn’t lead to much performance gain.\n\n**Average scores across domains:** Table 7 in the attached PDF shows the average scores for the original D4RL MuJoCo, antmaze, kitchen, and adroit datasets, as well as our imbalanced MuJoCo dataset. We also included the average scores across all domains. The numbers inside the parenthesis denote the relative score compared with the Uniform.\n\n**DW matches the baselines in the original D4RL datasets (D4RL Adroit, Antmaze, Kitchen, and Gym-MuJoCo).**\n\n- **For CQL**, both DW+AW and DW+Uniform match Uniform’s and AW’s performance, with average relative performance of +3.45 (DW+AW) and +1.47 (DW+Uniform). This shows that in terms of average performance, DW+AW and DW+Uniform even slightly improve the performance over Uniform in the original D4RL datasets.\n- **For IQL**, DW+AW and DW+Uniform also perform closely with the baselines, except for DW+AW in Antmaze and Kitchen. The performance drop of DW+AW is likely because AW is worse than Uniform by 16.9 and 5.9 points in both domains.\n\n> How does the method affect algorithms time needed for training?\n> \n\n**Answer:**\n\nWe provide the running time of each method below. DW only incurs 10mins more in the training time, not causing excessive overhead.\n|  | Uniform | AW | PF | DW-AW | DW-Uniform |\n| --- | --- | --- | --- | --- | --- |\n| CQL | 58mins | 60mins | 60mins | 70mins | 67mins |\n| IQL | 40mins | 43mins | 43mins | 60mins | 57mins |\n\n> If I understand approach correctly, it can't be applied to the datasets which require trajectories stiching to complete task.\n> \n**Answer:**\n\nWe would like to clarify that DW can be applied to tasks requiring trajectory stitching and illustrate why in the following.\n\n**DW doesn’t hurt in AntMaze.** The following table presents the average return of CQL and IQL with varying sampling/reweighting methods in AntMaze datasets. DW+Uniform matches the performance of Uniform, showing that it doesn’t hurt the performance in tasks requiring trajectory stitching. The reason why applying DW doesn’t improve over Uniform can be that offline RL algorithms have already done trajectory stitching well. Thus there is not much room for improving the sampling distribution. It is likely because the state-action distribution in AntMaze is not skewed toward low-performing trajectories. As such, regularized offline RL algorithms won’t suffer in such cases.\n\n|  | Uniform | AW | PF | DW+AW (ours) | DW+Uniform (ours) |\n| --- | --- | --- | --- | --- | --- |\n| CQL | 15.2 | 23.4 | 17.7 | 22.1 | 19.1 |\n| IQL | 63.8 | 47.0 | 0.0 | 45.5 | 63.2 |\n\n**Four-room experiment shows that DW can stitch trajectories.** To assess DW's trajectory stitching ability, we conducted an experiment in a didactic four-room environment [2].\n\n- **Experiment setup:** See Figure 8 in the attached PDF for the environment illustration. The agent starts from an orange initial state, traverses non-red cells, gaining +1 reward at the green goal, else zero. To test trajectory stitching, a suboptimal dataset with 1000 trajectories was generated, where none of each is optimal trajectory. Due to absence of optimal trajectories, up-weighting trajectories (i.e., AW and PF) with high-returns won’t produce a state-action distribution matching the optimal policy. Thus, if a method can generate state-action distribution matching the one of the optimal policy, it indicates that the method is able to stitch trajectories because it can identify optimal state-action pairs leading to the goal even though those optimal state-action pairs are not observed in the same trajectory in the dataset.\n- **Results:** Figures 8(a), 8(b), and 8(e) display the state-action distributions of behavior policy, optimal policy, and DW; the number above each plot is expected return under the state-action distribution. We see that both AW (Figure 8(c)) and PF (Figure (d)) fail to match the state-action distribution of the optimal policy in Figure 8(b), hence leading to suboptimal performance. In contrast, DW successfully approximates the optimal policy's state-action distribution, confirming DW can identify optimal state-action pairs observed in different suboptimal trajectories and stitch them to optimal trajectories.\n\n**DW improves BC in Ant U-maze.** To see if the learned importance weight mirrors a data distribution of a better policy than the behavior policy, we evaluate DW upon BC. This way, we can exclude trajectory stitching from offline RL algorithms like CQL and IQL since BC cannot do trajectory stitching. To improve beyond the behavior policy, one needs to change the data distribution to make it reflect a better policy. Our results show that DW achieves better performance than uniform sampling (65 v.s. 45), indicating that DW is applicable in tasks requiring trajectory stitching.\n\n[1] Hong Z W, Agrawal P, des Combes R T, et al. Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting, ICLR'23\n\n[2] Lee, Jongmin, et al. \"Optidice: Offline policy optimization via stationary distribution correction estimation.\" ICML'21"}, {"review_id": "rmnPYG4Kja", "reviewer": "Reviewer_XiFQ", "summary": "This paper investigates the offline RL with the imbalanced dataset setting. The proposed method, which optimizes a parameterized density-ratio weighting (DW) model for each transition, anchors a policy close to the good part of trajectories in the dataset. The proposed method is shown to outperform state-of-the-art methods on 72 imbalanced datasets both in varying imbalance and initial state distribution.", "questions": "4. It is reasonable to set $\\gamma=1$ to avoid the estimation of $\\rho_0$. Have the authors analyzed the role of the discount factor for offline RL [5]? It will be appreciated if any theoretical analysis or/and experiments are provided.\n5. Lacking the AW-L benchmark in Figure 2(b). The medium temp is extremely beneficial in IQL (see Figure 2(b)), why do not present the performance of AW-L?\n7. In Figure 3, It is counterintuitive that DW is not influenced by diverse initial states and varying trajectories because DW does not consider the initial state distribution. Could the authors explain this? \n8. Thank the authors for providing technical details including their hyperparameter search range with $\\lambda_f$ and $\\lambda_k$. However, I am a bit concerned that the necessity of penalties, i.e., Bellman flow and KL-divergence. Could the authors please introduce more experiences to show their necessities and the guidance for hyperparameter research with new tasks?\n\n[1] Hong Z W, Agrawal P, des Combes R T, et al. Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting[C]//The Eleventh International Conference on Learning Representations. 2023.\n\n[2] Brandfonbrener D, Whitney W F, Ranganath R, et al. Quantile filtered imitation learning[J]. arXiv preprint arXiv:2112.00950, 2021.\n\n[3] Wu Y, Zhai S, Srivastava N, et al. Uncertainty weighted actor-critic for offline reinforcement learning[J]. arXiv preprint arXiv:2105.08140, 2021.\n\n[4] Xu H, Jiang L, Li J, et al. Offline rl with no ood actions: In-sample learning via implicit value regularization[J]. arXiv preprint arXiv:2303.15810, 2023.\n\n[5] Hu H, Yang Y, Zhao Q, et al. On the role of discount factor in offline reinforcement learning[C]//International Conference on Machine Learning. PMLR, 2022: 9072-9098.\n\n", "limitations": "See questions and weaknesses.\n\n---\nraise the score to 6 after reviewing the response from the authors.\n\n---\n\n**Minors**:\n1. Cite twice: The classic results in [34] indicate that a policy’s expected return can be expressed in terms of its stationary state-action distribution [34].\n2. IQL training object for policy improvement (line 234) is wrong. \n3. What is DW-uniform? Only use DW? I suggest changing the name of DW-uniform because *uniform* will cause misunderstanding that DW-uniform is a combination sampling method between DW and Uniform. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Overall good writing quality and clarity.\n2. The proposed approach is well-motivated by adopting existing approaches. \n3. Sufficient technical details for reproducing the experiments. ", "weaknesses": "1. Novelty limited: Fixed conservativeness is defective for imbalanced (heterogeneous) datasets and reweighting or filtering samples/trajectories are not new in offline RL [1, 2, 3]. On the other hand, this paper does not connect with the previous most similar work [1] and explains why DW is better than AW, though it claims that AW is empirically prone to overfitting. \n2. Lack of details for the dataset setup. \n3. Lack of comparison: the imbalanced dataset is also called the noisy dataset [4]. If the authors follow the setup of the dataset from AW [1], the same as the noisy data setup [4], it is reasonable to include more comparisons between SQL/EQL enforcing a Q-function weighting on the transitions and show the edge of DW. \n3. Experiments between AW and DW: In Figure 2, AW-M is better than DW-Uniform in three of four tasks. However, the authors claim that they are better than AW.  More explanation is needed about why DW is better than AW.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the quality and clarity of our paper, and breadth of our experimental evaluation. We respond to the main questions in the following (W=weakness, Q=questions).\n\n> W1: Novelty\n\nWe're the first to use density ratio optimization to address fixed conservativeness in offline RL (like CQL, and IQL), differing from prior work [1, 2, 3].\n\n[1] need trajectories to start from similar initial states; otherwise, excessive weights can be assigned to trajectories starting from lucky initial states with higher returns, and data in other good trajectories with unlucky initial states will be underutilized. DW doesn't require trajectories to start from similar initial states, outperforming AW in datasets with diverse initial states in Fig. 3.\n\n[2] filter low-value data, but we maximize the expected rewards of the reweighted data distribution. Since their code is not released, we compared DW with its closet work [4], showing that our DW outperforms [4] in Fig. 7 in the attached PDF.\n\n[3] weight samples by Q-values' variance, but DW weights data by their rewards (with some constraints). In Fig. 7, UWAC underperforms the other methods, suggesting it may not be an effective approach to address the issues in imbalanced datasets.\n\n> W1: Why DW is better than AW\n> \n\nWe want to clarify that we already cited and compared with AW [1], showing that DW-AW outperforms AW on various imbalanced datasets (Figs 2 and 3). **DW is better than AW as it can up-weight valuable data in both low- and high- return trajectories**, while AW only does this for high-return ones. This is crucial as valuable state-action pairs might be hidden in low-return trajectories. In Fig. 8 (PDF), AW and PF fail to find optimal state-action distribution due to missing optimal trajectories. In contrast, DW learns the optimal distribution, highlighting the benefit of DW over trajectory reweighting.\n\n> W2 & 3 and Q2: Lack of details for the dataset setup / Lack of comparison / Lack of AW-L\n> \n\nThe dataset setup is in Appendix A.3.3. We’ve added AW-L, SQL and EQL in Fig. 7. AW-L is close to AW-M in small datasets. Both DW-AW and DW-Uniform outperform SQL and EQL in imbalanced datasets.\n\n> W4: Why DW is better than AW in Figure 2\n> \n\nDW is better than AW since combing DW and AW (i.e., DW-AW) achieves higher average return or match the baselines in Figures 2(a) and 2(b). Please let us know if this explanation answers the question.\n\n> Q1: Role of the discount factor for offline RL [5]\n\nWe keep the original discount factors for training offline RL algorithms (CQL, IQL), only setting $\\gamma=1$ for DW. The effects of discount factors in offline RL should directly transfer to the combination of DW with offline RL algorithms. \n\n> Q3: Why DW isn’t influenced by diverse initial states\n\nThe reviewer might believe DW works only with a single initial state due to Eq. 10 omitting initial state distribution. However, we'd like to kindly clarify that DW's formulation isn't restricted to one initial state but agnostic to initial states because it optimizes undiscounted returns instead of discounted ones, as detailed in Sec. 4.1.\n\n> Q4: Necessity of Bellman flow and KL penalties.\n\n- **Bellman flow penalty is required** to ensure the learned weights are valid in the MDP (Sec. 4.1). Otherwise, maximizing the objective (Eq. 6) makes all the weights assigned to the data with the highest reward.\n- **KL penalty is required,** according to the theoretical analysis shown by Sec. 5.1 in [6]. Otherwise, high weights can be assigned to the actions that lead to next states that are absent in the dataset.\n\n> Q4: Guidance for hyperparameter search\n\n- We suggest starting with low **flow penalty $\\lambda_F$** since it is lower bounded by zero, and increasing it often leads to performance gain. The following table shows the average return shown in Table 2 in Appendix, where K and F denote $\\lambda_K$ and $\\lambda_F$, respectively. Increasing F improves the performance in IQL. We found that CQL is less sensitive to F.\n\n    | K,F | 0.2,0.1 | 0.2,1.0 | 0.2,5.0 | 1.0,0.1 | 1.0,1.0 | 1.0,5.0 |\n    | --- | --- | --- | --- | --- | --- | --- |\n    | CQL | 48.1 | 41.7 | 47.5 | 10.1 | 7.5 | 11.5 |\n    | IQL | 32.0 | 56.0 | 52.4 | 47.2 | 58.5 | 61.5 |\n- Regarding **KL penalty weight $\\lambda_K$**, we discuss in two datasets:\n    1. **Low-return dominant:** To ease offline RL's conservatism on data from low-performing policies, we suggest starting with low $\\lambda_K$ since high $\\lambda_K$ limits deviation from the dataset. The above table displays average performance in low-return dominant datasets, showing higher $\\lambda_K$ can lead to drops in CQL.\n    2. **High-return dominant:** As data here is nearly optimal, we suggests beginning with high $\\lambda_K$. The following table  shows the performance in **`halfcheetah-expert-v2`** dataset, showing that high $\\lambda_K$ leads to better performance.\n        \n  | K,F | 0.2,5.0 | 1.0,5.0 |\n  | --- | --- | --- |\n  | CQL | 59.2 | 81.9 |\n  | IQL | 95.2 | 95.1 |\n\nWhile optimal coefficients vary by dataset, we've shown that the same hyperparameters can enhance performance in imbalanced datasets while not harming overall performance in high-return dominant datasets from the original D4RL.\n\n> Minors\n\nThank you for your feedback. We'll revise the manuscript accordingly. \"DW-Uniform\" refers to DW trained with uniform sampling, and \"DW-AW\" to DW trained with AW sampling, based on the optimization of Eq. 13.\n\n[1] Hong et al. “Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting”\n\n[2] Brandfonbrener et al. Quantile filtered imitation learning\n\n[3] Wu et al. Uncertainty weighted actor-critic for offline reinforcement learning\n\n[4] Chen et al. Bail: Best-action imitation learning for batch deep reinforcement learning\n\n[5] Hu et al. On the role of discount factor in offline reinforcement learning\n\n[6] Zhan et al. Offline reinforcement learning with realizability and single-policy concentrability"}]

Author rebuttal:
We thank all the reviewers’ suggestions and want to highlight **five new baselines** **(Figure 7)** and **two new analysis (Figures 8 and 9)** in the attached PDF file. The following is the summary of our new experiments. 

1. We added the comparison to five new prior works, BEAR [1] (Reviewer WKvr), BAIL [2] (Reviewer yys4), SQL and EQL [3], and UWAC [4] (Reviewer XiFQ) in Figure 7 in the attached PDF. The results showed that **our DW-Uniform and DW-AW with CQL and IQL outperform all the newly added baselines on 72 imbalanced datasets.**
2. **Figure 8 (Reviewers** `qcFR` and `XiFQ`)**:** We presented a new analysis to answer why our DW performs better than AW and PF (i.e., top-K% filtering) by showing that DW can stitch trajectories but AW and PF struggle to do so in didactic four-room environment. 
3. **Figure 9 (Reviewers `Wkvr`):** we presented the evolution of the distribution of learned importance weight over training epoch, showing that the training of DW is stable.

We list the public baseline implementation used in our experiments in the following:
- BEAR: [github.com/takuseno/d3rlpy](github.com/takuseno/d3rlpy)
- BAIL: [github.com/lanyavik/BAIL](github.com/lanyavik/BAIL)
- SQL/EQL: [github.com/ryanxhr/IVR](github.com/ryanxhr/IVR)
- UWAC: [github.com/apple/ml-uwac](github.com/apple/ml-uwac)

[1] Kumar, Aviral, et al. "Stabilizing off-policy q-learning via bootstrapping error reduction." *Advances in Neural Information Processing Systems* 32 (2019).

[2] Chen, Xinyue, et al. "Bail: Best-action imitation learning for batch deep reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 18353-18363.

[3] Xu H, Jiang L, Li J, et al. Offline rl with no ood actions: In-sample learning via implicit value regularization[J]. arXiv preprint arXiv:2303.15810, 2023.

[4] Wu Y, Zhai S, Srivastava N, et al. Uncertainty weighted actor-critic for offline reinforcement learning[J]. arXiv preprint arXiv:2105.08140, 2021.

Meta-review: This paper addresses the distributional shift problem in offline RL by optimizing the importance sampling weights, so that the data distribution is similar to that generated by a good policy.  This improves upon existing methods that enforce alignment with the given trajectories, including the low-performing ones.  Experimental results on 77 datasets with varying imbalance demonstrate that the method is more effective than SOTA offline RL algorithms.

This paper is well written, proposing a solid solution to an important problem.  All reviewers, and myself, agree that it is a good addition to the conference. We hope that the reviews are helpful for improving the paper.
Decision: Accept (poster)