Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
Authors: Zhang-Wei Hong
Pulkit Agrawal
Remi Tachet des Combes
Romain Laroche
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
reinforcement learning
sampling
experience replay

Abstract: Most offline reinforcement learning (RL) algorithms return a target policy maximizing a trade-off between (1) the expected performance gain over the behavior policy that collected the dataset, and (2) the risk stemming from the out-of-distribution-ness of the induced state-action occupancy. It follows that the performance of the target policy is strongly related to the performance of the behavior policy and, thus, the trajectory return distribution of the dataset. We show that in mixed datasets consisting of mostly low-return trajectories and minor high-return trajectories, state-of-the-art offline RL algorithms are overly restrained by low-return trajectories and fail to exploit high-performing trajectories to the fullest. To overcome this issue, we show that, in deterministic MDPs with stochastic initial states, the dataset sampling can be re-weighted to induce an artificial dataset whose behavior policy has a higher return. This re-weighted sampling strategy may be combined with any offline RL algorithm. We further analyze that the opportunity for performance improvement over the behavior policy correlates with the positive-sided variance of the returns of the trajectories in the dataset. We empirically show that while CQL, IQL, and TD3+BC achieve only a part of this potential policy improvement, these same algorithms combined with our reweighted sampling strategy fully exploit the dataset. Furthermore, we empirically demonstrate that, despite its theoretical limitation, the approach may still be efficient in stochastic environments. 

Published as a conference paper at ICLR 2023
HARNESSING MIXED OFFLINE REINFORCEMENT LEARNING DATASETS VIA TRAJECTORY WEIGHTING
Chang-Wei Hong & Pulkit Agrawal
Rémi Tachet des Combes" & Romain Laroche'
Massachusetts Institute of Technology USA (wwhong, pu pulkitag)emit..ccmmm
remi a chet @gma com
romain. la arocheeggaaa com
ABSTRACT
Most offline reinforcement learning (RL) algorithms return target policy maximiz- ing trade-off between (1) the expected performance gain over the behavior policy that collected the dataset, and (2) the risk stemming from the out-of-distribution- ness of the induced state-action occupancy. It follows that the performance of the target policy strongly related to the performance of the behavior policy and, thus, the trajectory return distribution of the dataset. We show that in mixed datasets consisting of mostly low-return trajectories and minor high-return trajectories, state- of-the-art offline RI. algorithms are overly restrained by low-return trajectories and fail to exploit high-performing trajectories to the fullest. To overcome this issue, we show that, in deterministic MDPs with stochastic initial states, the dataset sampling can be re-weighted to induce an artificial dataset whose behavior policy has higher return. This re- weighted sampling strategy may be combined with any offline RL algorithm. We further analyze that the opportunity for performance improvement over the behavior policy correlates with the positive-sided variance of the returns of the trajectories in the dataset. We empirically show that while CQL. IQL, and TD3+BC achieve only a part of this potential policy improvement, these same gorithms combined with our reweighted sampling strategy fully exploit the dataset. Furthermore, we empirically demonstrate that, despite its thcoretical limitation, the approach may still be efficient in stochastic environments. The code is available at https: / / //gittuuuuuu thub. com/ Impr obable-bbbeeeee.. weweee-Aaaaaaeeeeeeeeee feffiinerre rl.
INTRODUCTION
Offline reinforcement learning (RL) currently receives great attention because it allows one to optimize RL policies from logged data without direct interaction with the environment. This makes the RL training process safer and cheaper since collecting interaction data is high-risk, expensive, and tine-consuming in the real world (c.g., robotics, and health care). Unfortunately, several papers have shown that near optimality of the offline RL task is intractable sample-efficieney--ii (Xiao et al., 2022; Chen & Jiang, 2019; Foster et al., 2022).
In contrast to near optimality, policy improvement over the behavior policy is objective that is approximately realizable since the behavior policy may efficiently be cloned with supervised learning (Urbancie, 1994; Torabi et al., 2018). Thus, most practical offline RL algorithms incorporate a component ensuring, either formally or intuitively, that the returned policy improves over the behavior policy: pessimistic algorithms make sure that lower bound on the target policy (i.e., policy learned by offline RL algorithms) value improves over the value of the behavior policy (Petrik al., 2016; Kumar et al., 2020b; Buckman et al., 2020), conservative algorithms regularize their policy search with respect to the behavior policy (Thomas, 2015; Laroche et al., 2019; Fujimoto et al., 2019), and one-step algorithms prevent the target policy value from propagating through bootstrapping (Brandfonbrener et al., 2021). These algorithms use the behavior policy as a stepping stone. As a consequence, their performance guarantees highly depend on the performance of the behavior policy.
WWrk done while Microsoft Research Montreal.

---

Published as a conference paper at ICLR 2023
Due to the dependency behavior policy performance, these offline RL algorithms susceptible to the return distribution of the trajectories in the dataset collected by a behavior policy. To illustrate this dependency, we will say that these algorithms anchored to the behavior policy. Anchoring in a near-optimal dataset (i.e., expert) favors the performance of an algorithm, while anchoring in low-performing dataset (e.g., novice) may hinder the target policy's s performance. In realistic scenarios, offline RL datasets might consist mostly of low-performing trajectories with few minor high- performing trajectories collected by mixture of behavior policies, since curating -eg--prrorming trajectories is costly. It is thus desirable to avoid anchoring on low-perforning behavior policies and exploit
ones in mixed datasets. However, we show that state-of-the ttttheeart offline RL algorithms fail to exploit high-performing trajectories to their fullest. We analyze that the potential for policy improvement over the behavior policy is correlated with the positive-sided variance (PSV) of the trajectory returns in the dataset and advance that when the return PSV is high, the algorithmic anchoring may be limiting the performance of the returned policy.
In order to provide better algorithmic anchoring, we propose to alter the behavior policy without collecting additional data. We start by proving that re-weighting the dataset during the training of an offline RL algorithm is equivalent performing this training with another behavior policy. Furthermore, under the assumption that the environment is deterministic, by giving larger weights to high-return trajectories, we can control the implicit behavior policy to be high performing and therefore grant a cold start performance boost to the offline RI algorithm. While determinism is strong assumption that we prove to be necessary with a minimal failure example, we show that the guarantees still hold when the initial state is stochastic by re-weighting with, instead of the trajectory return, a trajectory return advantage: G(T,) - V (s..)) where G(T:) is the return obtained for trajectory& i, ('(sp)) is the expected return of following the behavior policy μu from the initial state Si,0. Furthermore, we empirically observe that our strategy allows performance gains over their uniform sampling counterparts even in stochastic environments. We also note that determinism is required by several state-of-the-ae offline RL algorithms (Schmidhuber, 2019; Srivastava et al., 2019; Kumar et al., 2019b; Chen al., 2021; Furuta et al., 2021; Brandfonbrener et al., 2022).
Under the guidance of theoretical analysis, our principal contribution is two simple weighted sampling strategies: Return-weighting (RW) and
(AW). RW and AW re-weight trajectories using the Boltzmann distribution of trajectory returns and advantages, respectively. Our weighted sampling strategies agnostic to the underlying offline RL algorithms and thus can be drop-in replacement in any off-the-shelf offline RL algorithms, essentially at no extra computational cost. We evaluate our sampling strategies on three ofate-of-the-art offline RL algorithms, CQL. IQL, and TD3+BC (Kumar et al., 2020b; Kostrikov et al., 2022; Fujimoto & Gu, 2021), as well as behavior cloning, over 62 datasets in D4RL benchmarks (Fu et al., 2020). The experimental results reported in statistically robust metries (Agarwal et al., 2021) demonstrate that both our sampling strategies significantly boost the performance of all considered offline RL algorithms in challenging mixed datasets with sparse rewarding trajectories, and perform at least on par with them on regular datasets with evenly distributed return distributions.
2 PRELIMINARIES
We consider reinforcement learning (RL) problem in a Markov decision process (MDP) characterized by a tuple (S. A. R, P. Po). where S and A denote state and action spaces, respectively, R : S x A -> R is a reward function, P S x A + As is a state transition dynamics, and Po As is an initial state distribution, where Ax denotes a simplex over set X. An MDP starts from an initial state 8o Po. At each timestep i, an agent perceives the state 8t, takes an action a, ~ (.1stt) where : S -> AA is the agent's policy, receives a reward r R(s,, 0z), and transitions St41 ~ P($t41|s,, at). The performance of a policy is measured by the expected return ((z) starting from initial states 80 ~ Pu shown as follows:
next state
((T) =E
Σ
R(st, at) so ~ pa. ~ T(]ss), ~ P()ls,
2

---

Published as a conference paper at ICLR 2023
Given dataset D collected by behavior policy μ 8 A.. offline RL algorithms aim to learn target policy x such that J(T) 2 J(H) from dataset D shown as follows:
$1,0 ~ Po: D = (0L,0 di,0, Ti, 10,07 8..T.) Ti, = R(si,,, ax,)),
04,1 pl(-1s,,): 81,41 P(.sst. a
(2)
where 7; = (81.0: 04,0, r..0- ... +1) denotes trajectory in D, (i,t) denotes timestep in episode and 7, denotes the length of r;- Note that la can be mixture of multiple policies. For brevity. omit the episode index in the subscript of state and actions, unless necessary. Generically, offline RL algorithms learn based on actor-critic methods that train a Q-value function Q S A and π in parallel. The Q-value ((s, a) predicts the expected return of taking action a at state s and following later; a maximizes the expected Q-value over D. Q and are trained through alternating between policy evaluation (Equation 3) and policy improvement (Equation 4) steps shown below:
Q + arg min I
(re + 7Ea' [Q(s+11, d')l -st,, a+))
Uni(D)
(3)
Q
arg max E ((s):a) a)
Uni(D), ((1ss))
(4)
where E[ Uni( denotes an expectation over uniform sampling of transitions.
3 PROBLEM FORMULATION
Most offline RL algorithms are anchored the behavior policy. This beneficial when the dataset behavior policy is high-pes nigh-performing while detrimental when the behavior policy is low- low-performing. We consider mixed datasets consisting of mostly low-performing trajectories and handful of high- performing trajectories. In such datasets, it is possible to exploit the rare high-performing trajectories, yet the anchoring restrains these algorithms from making sizable policy improvements over the behavior policy of the mixed dataset. We formally define the return positive-sided variance (RPSV) of a dataset in Section 3.1 and illustrate why the performance of offline RL algorithms could be limited on high-RPSV datasets in Section 3.2.
3.1 POSITIVE- VARIANCE
Formally, we concerned with dataset D (To, ". potentially collected by various be- havior policies (Po. (11 ".. (NN-1) and constituted of empirical returns [G(r)),G(n)) ((TT),..G(TTT- G(TN-1)) where T; is generated by (4, N is the number of trajectories, T, denotes the length of 7, and G(T,) = - Ts To study the distribution of return, we equip ourselves with statistical quantity: the positive-sided variance (PSV) of random variable X:
Definition (Positive-sided variance). The positive-sided variance (PSV) of a random variable X is the second-order moment of the positive component of X - E[X]:
[+] E (x- - with x4 = max O.
(5)
The return PSV (RPSV) of D aims capturing the positive dispersion of the distribution of the trajectory returns. An interesting question to ask is: what distribution leads to high RPSV? We simplify sampling trajectories collected by a novice and an expert as sampling from a Bernoulli distribution B, and suppose that the novice policy always yields O return, while the expert always yields return. Figure visualizes V [B(p)] p(1 p)?, which is the Bernoulli distribution's PSV as function of its parameter where p the probability of choosing an expert trajectory. We see that maximal PSV is achieved for p = J. Both p = (pure novice) and p (pure expert) leads to zero PSV. This observation indicates that mixed datasets tend to have higher RPSV than dataset collected by a single policy. We present the return distribution of datasets at varying RPSV in Figure 1. Low-RPSV datasets have their highest returns that remain close to the mean return. which limits the opportunity for policy improvement. In contrast, the return distribution of high-RPSV datasets disperses away from the mean toward the positive end.
3

---

Published as a conference paper at ICLR 2023
RPSV= 0.00, 0.0.07
RPSV =.08. 6
Density
Density
0.05
0.6
0.0
0.2
0.4
0.6
0.8
0.
0.0
0.2
0.4
0.6
0.8
1.0
Normalized return G(t))
Normalized return G(T)
(a) Beroulli distribution PSV
(b) Low RPSV
(c) High RPSV
(a) Bernoulli distribution PSV: V [+[ p(1 p)3 (b-c) The return distribution of datasets with (b) low and (c) high return positive-sided variance (RPSV) (Section 3.1), where RPSV measures the positive contributions the variance of trajectory returns a dataset and G denotes the average episodic returns (dashed line) of the dataset. Intuitively, a high RPSV implies some trajectories have far higher returns than the average.
Figure 1:
3.2 OFFLINE RL. FAILS TO UTILIZE DATA IN HIGH-RPSV DATASETS
High-RPSV datasets (Figure Ic) have handful of high-return trajectories, yet the anchoring of offline RL algorithms behavior policy inhibits offline RL from utilizing these high-return data to their fullest. Predominating low-return trajectories in a high-RPSV dataset restrain offline RL algorithms from learning non-trivial policy close the best trajectories in D due these algorithms' pessimistic and/or conservative nature. High RPSV implies that the average episodic return is far from the best return in D (see Figure 1c). The average episodic return reflects the performance (μ) (formally justified in Section 4. 4.1) of the behavior policy that collected DD where p is mixture of {Ho, M1, 1 (Section 3.1).
Pessimistic algorithms (Petrik et al., 2016; Kumar et al., 2020b; Buckman et al., 2020) strive to guarantee the algorithm returns such that ( 2 J(x,, but this guarantee is loose when (n) is low. Conservative algorithms (Laroche et al., 2019; Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019a) restrict to behave closely to μ to prevent exploiting poorly estimated Q-values on out-of- cit-of-distributtinn state-action pairs in actor-critic updates (i.e., ($141, a') & D in Equation 3), hence restricting J(r) from deviating too much from J(u). Similarly, one-step algorithms (Brandfonbrener et al., ;021; Kostrikov et al., 2022) that perform only single step of policy improvement return target policy subject to constraints that enforces (Peters & Schaal, 2007; Peng al., 2019). As consequence, offline RL algorithms are restrained by (μ) and fail to utilize high-return data far from (()) in high-RPSV datasets.
be close to J
On the contrary, in low-RPSV datasets (Figure lb), pessimistic, conservative, and one-step algorithms do not have this severe under-utilizatio issue since the return distribution concentrates around or below the average episodic return, and there are very few to better trajectories exploit. We will show, in Section 5.2, that no sampling strategy makes offline RL algorithms perform better in extremely low-RPSV datasets, while in high-RPSV datasets, methods (Sections 4.2 and 4.3) outperform typical uniform sampling substantially.
METHOD
Section 3 explains why behavior policy anchoring prevents offline RL algorithms from exploiting high-RPSV datasets to their fullest. To overcome this issue, the question that needs to be answered is: can we improve the performance of the behavior policy without collecting additional data? To do so. propose to implicitly alter it through re- weighgtting of the transitions in the dataset. Indeed, show that weighted sampling can emulate sampling transitions with a different behavior policy. We analyze the connection between weighted sampling and performance of the implicit behavior policy in Section 1. and then present two weighted sampling strategies in Sections 4.2 and 4.3.
4.1 ANALYSIS
We start by showing how re- reighighting the transitions in dataset emulates sampling transitions generated by an implicit mixture behavior policy different from the one that collected the dataset. implicit because the policy is defined by the weights of transitions in the dataset. As suggested
4

---

Published as a conference paper at ICLR 2023
in Peng et al. (2019), sampling transitions from D defined in Section is equivalent to sampling state-action pairs from a weighted joint state-action occupancy: dw(s, a) Σ N- w.dyy (s)p,, _
where w; is the weight of trajectory i (each T; is collected by μ4), W = ( -N-- -1). and dz. (s) denotes the unnormalized state occupancy measure (Laroche et al., 2022) in the rollout of p4. Tweaking weighting W effectively alters dw and thus the transition distribution during sampling. As Peng et al. (2019) suggested, a weighting W also induces a weighted behavior policy: (s.a)
(w((as) = SN=1 wid,, (s) Uniform sampling te; N. Vw; E W is equivalent to sampling from the joint state-action occupancy of the original mixture behavior policy μ that collected D. To obtain well-defined sampling distribution over transitions, we need to convert these trajectory weights w; to transitions sample weights Wa,t, Vt E [0, 1]:
TT-1 N-1 T
W1
Σ Σ
Wie
W
1.
(6)
Tww
Tiw;
1=
i=
i=0
Thus, we formulate our goal as finding W {ui)eeiu.N-1 E AN such that J(Jw) 2 J(N), where AN denotes the simplex of size N. Naturally, we can write J(uw)= Ei w,J(m). The remaining question is then to estimate J(u)). The episodic return G(T,) can be treated as a sample of J(G). As result, we can concentrate J(ww) near the weighted sum of returns with direct application of Hoeffding's inequality (Serfling, 1974):
PP [J(w))-EA -N-1 w;;((T)) >
z S 2xx
2ea
(7)
G N w
where G, = GMAX - GMIN is the return interval amplitude (see Hoeffding's 's inequality). For complete- ness, the soundness of the method is proved for any policy and MDP with discount factor (Sutton & Barto, 2018) less than 1 in Appendix A. Equation 7 tells us that we have a consistent estimator for J(Jw) as long as too much mass has not been assigned to a small set of trajectories.
Since goal is to obtain a behavior policy with higher performance, we would like to give high weights w, to high performance μt. However, it is worth noting that setting wy as a function of G; could induce bias in the estimator of J(ww due the stochasticity in the trajectory generation, stemming from Po. P. and/or 14;- In that case, Equation concentration bound would not be valid anymore. To demonstrate and illustrate the bias, we provide counterexample in Appendix A.3. The following section addresses this issue by making the strong assumption of determinism of the environment, and applying a trick remove the stochasticity from the behavior policy μ4, Section 4.3 then relaxes the requirement for Po to be deterministic by using the return advantage instead of the absolute return.
4.2 RETURN- WEIGHTING
In this section, we are making the strong assumption that the MDP is deterministic (i.e., the transition dynamics P and the initial state distribution po is Dirac delta distribution). This assumption allows us to obtain that G(T,) J(4)), where 14 is the deterministic policy taking the actions in trajectory T Since the performance of the target policy is anchored on the performance of a behavior policy,
we find weighting distribution W to maximize (pw):
max
(8)
WEAN
where wy corresponds to the unnormalized weight assigned to each transition in episode i. However, the resulting solution is trivially assigning all the weights to the transitions in episode TÜ with maximum return. This trivial solution would indeed be optimal in the deterministic setting we consider but would fail otherwise. To prevent this from happening, we classically incorporate entropy regularization and turn Equation 8 into:
N
max Σ
G.G
w; log Wi.
(9)
WEAN
1=0
k=0
'The formalization of μ4 is provided in Appendix A.1
5

---

Published as a conference paper at ICLR 2023
where a E R* is temperature parameter that controls the strength of regularization. interpolates the solution from delta distribution (a - 0) to an uniform distribution (a + oc). As the optimal solution to Equation is Boltzmann distribution of G(T,)), the resulting weighting distribution W is:
exp G(T:)//x
W/ Σ TED exp G(T,)/aa
(10)
The temperature parameter a (Equations 10 and 11) is fixed hyperparameter. As the choice of a is dependent on the scale of episodic returns, which varies across environments, we normalize G(Ti) -(z.)-minn G(T,)
using a max-min normalization: G(T,) max, ((T,) miny G(Ty))
4.3 ADVANTAGEE-WIGGITTN
In this section, allow the initial state distribution Po to be stochastic. The return-weighting strategy in Section 4.2 could be biased toward some trajectories starting from lucky initial states that yield higher returns than other initial states. Thus, we change the objective of Equation 9 to maximizing the weighted episodic advantage Σv.ew wT,)(T,) with entropy regularization. A(T.) denotes the
episodic advantage of T; and is defined as A(T:) = G(T:) - (s..)). (*r(,,,) is the estimated expected return of following μ starting from 9,,0, using regression: ÇP + arg miny [[(((T)) - (z Uni(D)]. Substituting G(T.) with A(T:) in Equation 9 and solving for W, we obtain the following weighting distribution:
exp A(Ti)/a A(T)) G(T,) (s..))))-
Wh; =
exp Ex A(T:)/a²
(11)
5 EXPERIMENTS
Our experiments answer the following primary questions: (i) Do our methods enable offline RL algorithms to achieve better performance in datasets with sparse high-return trajectories? (ii) Does method benefit from high RPSV? (iii) Can method also perform well in regular datasets? (iv) Is our method robust to stochasticity in an MDP?
5.1 SETUP
Implementation. We implement our weighted-sampling strategy and baselines in the following offline RL algorithms: implicit Q-learning (IQL) (Kostrikov et al., 2022), conservative Q-learning (CQL) (Kumar et al., 2020b), TD3+BC (Fujimoto Gu, 2021). and behavior cloning (BC). IQL. CQL, and TD3+BC were chosen to cover various approaches of offline RI, including one-step. pessimistic, and conservative algorithms. Note that though BC an imitation learning algorithm, we include it since BC clones the behavior policy, which is the object we directly alter, and BC is also common baseline in offline RL research (Kumar et al., 2020b; Kostrikov et al., 2022).
Baselines. We compare our weighted sampling against uniform sampling (denoted Uniform), percentage filtering (Chen et al., 2021) (denoted as Top-x%), and half-sampling (denoted as Half). Percentage filtering only filtering as baseline since it similarly increases the expected return of the behavior policy by discarding some data. In the following, we compare our method against Top-10% since 10% the best configum configuration found in the hyperparameter search (Appendix A. A.1 Half-sampling samples half of transitions from high-return and low-return trajectories. respectively. Half is simple workaround to avoid over-sampling low-return data in datasets consisting of only sparse high-return trajectories. Note that Half requires the extra assumption of separating a dataset into high-return and low-return partitions, while our methods do not need this. Our return-weightt and advantage-weighe strategies are denoted as RW and AW, respectively, for which we use the same hyperparameter a in all the environments (see Appendix A.7).
episodes with top-2%% returns for training. We consider percentage
Datasets and environments. (i.e., the combination of offline RL algorithm and sampling strategy) in MuloCo locomotion
We evaluate the performance of each algorithm+sampler variant

---

Published as a conference paper at ICLR 2023
COL
TD3+BC
0.4
50%%
50%
Rato d%
Ratio 5%
Ratio 0%
Ratio a%
Unttom
Hal
10%
RW (curs)
AW (ours)
Figure 2: Our RW and AW sampling strategies achieve higher returns (y-axis) than all baselines (color) on average consistently for all algorithms CQL. IQL, BC, and TD3+BC, all datasets with varying high-return data ratios %% (x-axis). Remarkably, our performances in four algorithms exceed or match the average returns (dashed lines) of these algorithms trained with uniform sampling in full expert datasets. The substantial performance gain over Uniform at low ratios (1% S 0% S 10%) shows the advantage of our methods in datasets with sparse high-return trajectories.
environments of D4RL benchmarks (Fu et al., 2020) and stochastic classic control benchmarks. Each environment is regarded as an MDP and can have multiple datasets in a benchmark suite. The dataset choices are described in the respective sections of the experimental results. We evaluate our method in stochastic classic control to investigate if stochastic dynamics break our weighted sampling strategies. The implementation of stochastic dynamics is presented in Appendix A.6.
Evaluation metric. An algorithm+sanple variant is trained for one million batches of updates in five random seeds for each dataset and environment. Its performance measured by the average normalized episodic return of running the trained policy over 20 episodes in the environment. As suggested Fu et al. (2020), we normalize the performance using (X X Random)) / Expert X Rundom) where X, X Random. and X Expert denote the performance of an algorithm-sampler variant, the random policy, and the expert one, respectively.
5.2 RESULTS IN MIXED DATASETS WITH SPARSE RETHREETUNN TRAJECTORIES
To answer whether our weighted sampling methods improve the performance of uniform sampling in datasets with sparse high-return trajectories, create mixed datasets with varying ratios of high-return data. We test each algorithn+sampler variant in four MuJoCo locomotion environments and eight mixed datasets, and one non-mixed dataset for each environment. The mixed datasets are created by mixing 5% of either an expert or medi um datasets (high-return) with (1 o%) of a random dataset (low-retum), for four ratios, o E {1, 5, 10, 50). The expert, medi um, and random datasets are generated by an expert policy, a policy with 1/3 of the expert policy performance, and a random policy, respectively. We test all the variants in those 32 mixed datasets and random dataset.
Figure shows the mean normalized performance (y-axis) of each algorithm+sampler (color) variant at varying a (x-axis). Each algorithm+sampler variant's performance is measured in the interquartile mean (IQM) (also known as 25%-trimmed mean) of average return (see Section 5.1) since IQM is less susceptible to the outlier performance suggested in Agarwal et al. (2021). Appendix A.8 details the evaluation protocol.
It can be seen that in Figure 2 our RW and AW strategies significantly outperform the baselines Uniform, Top-10%, and Half for all algorithms at at all expert/medium data ratio 0%. Remarkably, methods even exceed or match the performance of each algorithm trained in full expert datasets with uniform sampling (dashed lines) This implies that our methods enable offline RL algorithms achieve expert level of performance by 5% to 10% of medium or expert trajectories. Uniform fails to exploit to the fullest of the datasets when high-perforning trajectories are sparse (i.e., low σ). Top-10% slightly improves the performance, yet fails Uniform in low ratios (0% = 1%). which implies the best filltering percentage might be dataset-dependent Half consistently improves Uniform slightly all ratios, yet the amounts of performance gain are far below ours. Overall, these results suggest that up-weighting high-return trajectories in a dataset with low ratios of high-return data benefits performance while naively filtering out low-return episodes, as Top-10% does, does not consistently improve performance. Moreover, AW and RW do not show visible differences, likely because the initial state distribution is narrow in MuJoCo locomotion environments. We also include the average returns in each environment and dataset in Appendix A.13. In addition to average return, we also evaluate our methods in the probability of improvements (Agarwal et al., 2021) over uniform sampling and show statistically significant improvements in Appendix A. 10.
7

---

Published as a conference paper at ICLR 2023
CoL| RW lours)
IOL RN (ours)
BC| RW (ours)
TD3-BC BC] RW tours)
10-4
1.0
01866 CAA.
20-1
14
CoL| (ours)
|AW (ours)
BC| (ours)
TD3+BC|
11240 is.
0990999
10-1
14
10-4
-10
0.0
0.0 0.1 0.2
0.2
0.3
0.0
0.2
0.3 0.4
Average
dataset
dataset
Figure 3: The left end (average return between 0 to 0.1) of each plot shows that for all offline RL algorithms (CQL, IQL, and TD3+BC) and BC, our AW and RW sampling methods' performance gain grows when RPSV increases. The color denotes the performance (average return) gain over the uniform sampling baseline in the mixed datasets and environments tested in Section 5.2; the x x-axis and y-axis indicate the average return of a dataset and RPSV of the dataset, respectively.
5.3 ANALYSIS OF PERFORMANCE GAIN IN MIXED DATASETS
We hypothesize that methods' performance gain over uniform sampling results from increased RPSV in the datasets. The design of a robust predictor for the performance gain of a sampling strategy is not trivial since offline RL's performance is influenced by several factors, including the environment and the offline RL algorithm that it is paired with. We focus on two statistical factors that are easy to estimate from the dataset: (i) the mean return of a dataset and (ii) RPSV. Although dependent on each other, these two factors have a good variability in our experiments since increasing the ratio of expert/medium data would increase not only RPSV but also the mean return of a dataset.
We show the relationship between the performance gain over uniform sampling (represented by the color of the dot in the plots below), datasets' mean return (x-axis), and RPSV (y-axis, in log scale) in Figure 3. Each dot denotes the average performance gain in tuple of environment, dataset, and a. It can be seen that at similar mean returns (x-axis), our methods' performance gain grows evidently (color gets closer to red) when RPSV increases (y-axis). This observation indicates that the performance gain with low a (cxpert/medium data ratio) in Figure 2 can be related to the performance gain at high RPSV since most datasets with low mean returns have high RPSV in experiments. We also notice that a high dataset average return may temper our advantage. The reason is that offline RL with uniform sampling is already quite efficient the settings where o is in high range, such as 50%, and that the room for additional improvement over it is therefore limited.
5.4 RESULTS IN REGULAR DATASETS WITH MORE HIGHIRETTUN TRAJECTORIES
Datasets in Section 5.2 are adversarially created to test the performance with extremely sparse high-return trajectories. However, we show in Figure 5 that such challenging return distributions are not common in regular datasets in D4RL benchmarks. As a result, regular datasets are easier than mixed datasets with sparse high-return trajectories for the uniform sampling baseline. To show that our method does not lose performance in regular datasets with more high-return trajectories, we also evaluate our method in 30 regular datasets from D4RL benchmark (Fu et al., 2020) using the same evaluation metric in Section 5.1, and present the results in Figure 4a. It can be seen that our methods both exhibit performance on par with the baselines in regular datasets. confirming that our method does not lose performance. Note that we do not compare with Half since regular datasets collected by multiple policies cannot be split into two buffers. Notably, we find that with our RW and AW, BC achieves competitive performance with other offline RL. algorithms (i.e., CQL, IQL, and TD3+BC). The substantial improvement over uniform sampling in BC aligns with our analysis (Section 4. since the performance of BC solely depends on the performance of behavior policy and hence the average returns of sampled trajectories. Nonetheless, paired with RW and AW, offline RL algorithms (i.e., CQL, IQL, and TD3+BC) still outperform BC. This suggests that our weighted sampling strategies do not overshadow the advantage of offline RL over BC. The complete performance table can be found in Appendix 1.13. We also evaluate our methods' probability of
8

---

Published as a conference paper at ICLR 2023
Normalized average retur
0.8
1.0
Normalzzdd return
0.8
0.6
Uniform AW (ours)
0.2
0.9
0.4
COL
TD3+BC
5%
10%
50%
Uniform
Top 10%
RW (curs)
AW (ours)
Ratio 0%
Figure 4: (a) Our method matches the Uniform's return (y-axis). It indicates that methods do not lose performance in datasets consisting of sufficient high-return trajectories. These datasets are regular datasets in D4RL without adversarially mixing low-return trajectories do in Section 5.2. (b) Performance in classic control tasks with stochastic dynamics. Our method outperforms the baselines, showing that stochasticity do not break our methods.
improvements (Agarwal et al., 2021) over uniform sampling, showing that methods are than baselines in Appendix A.8.1.
5.5 RESULTS IN STOCHASTIC MDPs
As our weighted- weighted-sampling strategy theoretically requires a deterministic MDP, we investigate stochastic dynamics (i.e., stochastic state transitions) break our method by evaluating it in stochastic control environments. The details of their implementation can be found in Appendix A.6. We use the evaluation metric deseribed in Section 5.1 and present the results in Figure 4b. Both of methods still outperform uniform sampling in stochastic dynamics, suggesting that stochasticity does not break them. Note that only report the results with CQL since IQL. and TD3+BC not compatible with the discrete action space used in stochastic classic control.
RELATED WORKS
Our weighted sampling strategies and non-uniform experience replay online RL aim to improve uniform sample selection. Prior works prioritize uncertain data (Schaul et al., 2015; Horgan et al., 2018; Lahire et al., 2021), attending on nearly on-policy samples (Sinha et al., 2022), or select samples by topological order Hong al. (2022); Kumar et al. (2020a). However, these approaches do not take the performance of implicit behavioral policy induced by sampling into account and hence are unlikely to tackle the issue mixed offline RL datasets.
Offline imitation learning (IL) (Kim et al., 2021; Ma et al., 2022; Xu et al., 2022) consider training an expert policy from a dataset consisting of handful of expert data and plenty of random data. They train model to discriminate
transition is from an expert and learn nearly expert policy from the discriminator's predictions. Conceptually, our methods and offline IL aim to capitalize advantageous data (i.e., sparse high-return/expert data) in dataset despite different problem settings. Offline IL require that expert and random data are given two separated buffer, but do not need reward labels. In contrast, do not require separable datasets but require reward labels to find advantageous data.
DISCUSSION
Importance of learning sparse high-reture trajectories. Though most regular datasets in main- stream offline RL benchmarks such as D4RL have more high-return trajectories than mixed datasets studied in Section 5.2, it should be noted that collecting these high-return data tedious and could be expensive realistic domains (e.g., health care). Thus, enabling offline RL to learn from datasets with limited amount of high-return trajectories is crucial for deploying offline RL in more realistic tasks. The significance of our work is simple technique to enable offline RL to learn from a handful of high-return trajectories.
Limitation. As our methods require trajectory returns to compute the sample weights, datasets cannot be partially fragmented trajectories, and each trajectory needs to start from states in the initial state distribution; otherwise, trajectory return cannot be estimated, One possible approach to lift this limitation estimating the sample weight using learned value function so that one can estimate the expected return of state without complete trajectories.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank members the Improbable feedback. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing resoures.. This research was supported in part by the MIT-IBM Watson AI Lab, an AWS MLRA research grant, Google cloud credits provided as part of Google-MTT support, DARPA Machine Common Sense Program, ARO MURT under Grant Number W91 NN-21111---22 ONR MURI under Grant Number N00014-22-1-2740, and by the United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750- 118750-19-2-100 19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the Army Research Office the United States Air Force the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
and Microsoft Research Montreal for helpful discussions and
AUTHOR CONTRIBUTIONS
Zhang-Wai Wei Hong conceived the problem of mixed datasct in offiine RL and the idea of trajectory reweighting. implemented the algorithms, ran the experiments, and wrote the paper.
Pulkit Agrawal provided feedback on the idea and experiment designs.
Rémi Tachet des Combes provided feedback on the idea and experiment designs, came up with the examples in Appendix A.3, and revised the paper.
Romain Laroche formulated the analysis in Section 4, came up with RPSV metrics, and formulated the idea and revised the paper.
REPRODUCIBILLT STATEMENT
We have included the implementation details in Appendix A.7 and the source code in the supplemen- tary material.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
Eitan Altman. Constrained Markov Decision Processes. CRC Press, 1999.
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. In Proceedings of the 35th Advances in Neural Information Processing Systems (NeurIPS), 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-condition supervised learning work for offline reinforcement learning? arXiv preprint wwXiv:2206.01079 2022.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs//606.011400, 2016. URL http: http:pp////xvvooo/ org/ abs/1606 01540.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint Xix::2009006799 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distrbuution-aware margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Jinglin Chen and Nan Jiang. Information--theortiic considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042-1051. PMLR, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv: 106.0/000601344 2021.
10

---

Published as a conference paper at ICLR 2023
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Clas-balaaceed loss based on effective number of samples. In Proceedings the IEEE/CVP conference on computer vision and pattern recognition, pp.
2019.
Qi Dong, Shaogang Gong, and Xiatian Zhu. Imbalanced deep learning by minority class incremental rectification. IEEE transactions on pattern analysis and machine intelligence, 41(6):1367-1338 2018.
Dylan Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp.
PMLR, 2019.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364 2021.
Salvador García and Francisco Herrera. Evolutionary undersampling for classification with imbal- anced datasets: Proposals and taxonomy. Evolutionary computation, 17(3):275-306, 2009.
Zhang-Wo Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, and Pulkit Agrawal. Topological experi- ence replay. In In Proceedings The Tenth International Conference on Learning Representations, 2022. URL https: : / opwnreview n forum? d==ORZZeMOOI
Dan Horgan, John Quan, David Budden, Gabriel
Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018. URL https: / www./eeeeeeeeeeeeeeeeeeeeeeeeeee net./ forum ?
0%.
Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang. and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q-learning. In International Conference on Learning Representations, 2022. URL https: // /ooppneevieee iew. net/ mett/oorumiieeeeeeeeeeeeeeeeeeee (oormmidd=6n2299000
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-leanning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465, 2019b.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33:
18560-18572, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 200 Red Hook, NY, USA, 2020b. Curran Associates Inc ISBN 9781713829546.
Thibault Lahire, Matthieu Geist, and Emmanuel Rachelson. Large batch experience replay. arXiv preprint arXiv:2110.01522 2021.
Romain Laroche, Paul Trichelair, and Rémi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
11

---

Published as a conference paper at ICLR 2023
Romain Laroche, Jacob Buckman, and Rémi Tachet des Combes. About non-Markovian policies stater-action visitation density. Proceedings of the Sth Reinforcement Learning and Decision Making (RLDM), 2022.
Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine URL https: http::::s/ppooeednnggg//.
Learning Research, pp. 14639-14663. PMLR, 17-23 Jul 2022.
mlr. press/ v1 62 ma 22. html.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp- 745-750, 2007.
Marek Petrik, Mohammad Ghavamzadeh, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Proceedings of the 29th Advances in Neural Information Processing Systems (NIPS), 2016.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: 15X.0595 2015.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-juu just map them to actions. arXiv preprint
2019.
Robert Serfling. Probability inequalities for the sum in sampling without replacement. The Annals of Statistics, pp. 39-48, 1974.
Thiago D. Simāo, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with estimated baseline bootstrapping. In Proceedings of the 19th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), 2020.
Samarth Sinha, Jiaming Song, Animesh Garg, and Stefano Ermon. Experience replay with likelihood- free importance weights. Learning for Dynamics and Control Conference, pp. 110-123. PMLR. 2022.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz. Wojeiech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877 2019.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density- based weighting for imbalanced regression. Machine Learning, 110(8):117-22111, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Michita Imai Takuma Seno. d3rlpy: offline deep reinforcement library. NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Philip S Thomas. Safe reinforcement learning. PhD thesis, Stanford university, 2015.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 1rXiv:1805.01954. 2018
Tanja Urbancic. Reconstructing human skill with machine learning. In Proceedings of the 11th European Conference on Artificial Intelligence (ECAI). pp. 492-502. John Wiley, 1994.
Chenjun Xiao, Ilbin Lee, Bo Dai. Dale Schuurmans, and Csaba Szepesvari. The curse of passive data collection in batch reinforcement learning. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
12

---

Published as a conference paper at ICLR 2023
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator- seerrmmnattr-weighteedd offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp- 24725-24742 PMLR, 2022.
Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning (ICML), 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILED ANALYSIS
We consider the definitions of policy and a Markovian policy (Laroche al., 2022):
Definition 2 (Policy). policy a represents any finctiin mapping its trajectory history ha (80: 0o, TD St-1, Pe-1: to a distribution over actions T T--(he) E A where A denotes the simplex over A. Let 11 denote the space of policies, and 11 the space of deterministic policies.
Definition 3 (Markovian policy). Policy is said to be Markovian if its action probabilities only depend on the current state St: x(-[h+) = (T-(ss) E AA. Otherwise, policy π is non-Markovian. We let Hl denote the space of Markovian policies, and No the space of deterministic Markovian policies.
We make no assumption on the behavior policy 3, i.e. 3 E II. We notice that:
[(B) = [R, ~o. 3.p [R, B, B, ~ po. pl E [J(,,) Br B]. (12)
Equation 12 is trick that has already been used in Peng et al. (2019). We go bit further by constraining 8, to be a deterministic policy sampled at the start of the episode, which may be programmatically interpreted sampling the random seed used for the full trajectory. With trajectory-wise reweighting W, we obtain:
N J(Bw) Σ ((r.) ==0
(13)
Furthermore, Altman (1999) tell us that there exists Markovian policy 6 with the same occupancy measure as 3w and the same performance when y in MDPs with countable state space. Laroche et al. (2022) generalize this theorem to any MDP (including on uncountable state spaces) as long as y < Simão et al. (2020) prove in Theorem 3.2 Eq. (12) that, in finite MDPs, any Markovian behavior policy SM can be cloned with policy B%) from a dataset of trajectory up to accuracy of
2rt
3|S []S|||A|+44og 1 with high probability 1- - 6, where 2r, is the reward function amplitude. 2N2 N- wf
1-y1
A.2
ADDITIONAL RELATED WORKS: IMBALANCE CLASSIFCAATION/REGGEESSN
Mixed datasets with high RPSV are closely related imbalanced datasets in supervised learning. Supervised learning approaches either over-sample minority classes (Cui et al., 2019; Cao et al., 2019; Dong et al., 2018) or sample data inversely proportional to the target value densities (Yang et al., 2021; Steininger et al., 2021). Other works (Chawla et al., 2002; García & Herrera, 2009) synthesize samples by interpolating data points nearby the minority data. RL, on the other hand, over-sampling minority data (trajectories) can be harmful if the trajectory is low-return and does not cover high-performing policies' trajectories; in other words, naive application of over-sampling techniques from supervised learning can hurt in an RL setting they are agnostic the notion of return.
A.3 EXAMPLE OF BIAS WITH WEIGHTS THAT DEPEND ON THE REALIZATION OF THE TRAJECTORY
We will consider minimal example consisting of a stateless MDP (multi-arm bandit) and 2 actions A {aI, a2). Action a yields a deterministic reward of 0.6. Action 02 Bernouilli distribution reward with parameter p 0.5. In other words, 02 is coin flip: with 50% chance, no reward is received and with 50% chance, maximal reward 1. We collect a dataset containing some number of samples for each action. Now consider weights w; such that:
[ > 0.8]
(14)
Σ, 1G(T) > 0.8
We implicitly replace in the denominator their unweigthed term N with its weigthed version 2 N-1 wi.
14

---

Published as a conference paper at ICLR 2023
Then,
N-1
J(ww) = Σ
W.T (14) = I(H((2) = 1) = 0.5
(15)
1=0
N-1
Σ (T. > 0.8 i=0
w,G((T)
Σ, (T))) 0.8 1,
(16)
1=0
showing a counter-example for the concentration bound proposed in equation 7.
A.4 DATASET AVERAGE RETURN
We plot the average return of mixed and regular datasets in Figure 5. It can be that mixed datasets used in Section 5.2 have lower average return than regular datasets on average. Also, we study the relationship between average return of dataset and offline RL performance Figure showing that increasing average return of dataset improves offline RL performance. Interestingly, there is sweet spot where increasing datasets' average return starts hurting the performance. We hypothesize that it is due to insufficient state-aciton coverage of datasets. We also present the return distribution for each mixed dataset in Figure
3
Density
2
1
-0.5
0.0
0.5
1.0
1.5
Average return dataset
Sparse (mixed)
Dense (regular)
Figure 5: Average return of datasets. We see that mixed datasets used in Section 5.2 have lower average return than regular datasets average.
0.6
0.2
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Ar age return of dataset
Average return of dataset.
Average return of dataset
Average return of dataset
Un form
Figure 6: The relationship between average return of dataset (x-axis) and the performance (y-axis) of uniform sampling with an offline RL. algorithm.
A.5 DATASET RPSV
We list the PSV of each dataset Table I.
A.6 STOCHASTIC CLASSIC CONTROL.
We adapt CartPole-vi. Acrobot-vl, and MountainCar--a in classic control environments in Open AI gym (Brockman et al., 2016). For each timestep, an agent's 'ss actions has 10% chance to be replaced with noisy action a A. As such, the transitions dynamics turns to be stochastic.
15

---

Published as a conference paper at ICLR 2023
RPSV
ant-expert-v2 v2 (a
0.002886
ent- (σ 5)
0,013152
ant-expert-v2 (a 10)
0.026764
ant-cxpert-v2 (a 50)
0.151138
anl-expert-v2
0.015052
ant-fuullrepllay-22
0.096415
ant-mediuum-xppert-2
0.042611
amt-mediumm-epplay-22
0.038834
ant-medium-v2 (a = 1)
0.001334
ant-medium-v2 5)
0.006703
ant-medium-VV 10)
0,013846
ant-medium-v?? (a = 50)
0.075325
ant-medium-v2
0.020586
ant-random-v2
0.000092
untmaaz--ange-diivees--0
0.015378
autmazeelange-playyv00
0.004538
antiaazenmedduu--diveee-e-
0.072896
piimmazz-mmcdiu--laa--000
0.006583
uuthaaze-umazzeddiviis-vv0
0.029304
antmaze-umaze-v0
0.016956
halfchesttah-experrt-222 (ar 1)
0.008236
halfcheettah-eepperr-222 (ar 5)
0.035784
halfcheettab-eepport-222 (ar 10)
0.063523
halfchhettab-expeert-22 (a 50)
0.097583
hallcheetah-exxeertv22
0.000130
halfchhettah-full-eepaaa--2
0.006521
aafichetta--mcddiu--exxee---
0.028588
aafcchettah-mccduu--rrpaaayy-
0.005747
halichhetah-mmcddu--226e (a 1)
0.001771
halfcheetah--meddun--222 (a 5)
0.007604
halfcheetab-mmedium--22 (a 10)
0.013685
halichectah-mediiun-222 (a 50)
0.021007
halfcheetah medium-v2 nalfcheetahmmediun-22
0.0001 10
balfcheetah-raadddm-27
0.000019
hopper-expert-v22 (a 1)
0.000318
hopper-expert-v22 (a = 5)
0.001436
hopper-expert-v2 (a 10)
0.002948
hopper-expert-v2 50)
0.024691
hopper-expert-v2
0.000771
hopper-full-reelay--22
0.041835
hogper-meddium--exeett1222
0.064473
hopper-mcciuum-coolayy222
0.018824
hopper-medium--22 (a
0.000129
hopper-medium-v2 (a
0.000506
hopper-medium-y2 (a 10)
0.001055
hopper-medium-v2 (a 50)
0.008475
hopper-meddium-v2
0.006648
hopper-random-12
0,000025
walker2d-exxertt-22 (a = 1)
0.000264
walker20-exxerrt-222 (a 5)
0.001261
walker2dd-cxpert-v2 (Ø 10)
0.002601
walker2dd-cxpent-v22 50)
0.022037
walker2d-cxxert-v2
0.000030
walkeer2d-ffll--epplayyy
0.070379
walkeer2l-medium---xpetttt
0.027597
walkeer2U-meddiu--rrplaaaay
0.029698
walker22l-medium-22 (o =
0.000128
malker2d-medium-i (o = 55)
0.000559
malker2d-medium- (o 10)
0.001233
walker2d walker2d-medium- 500)
0.010165
walker22d-medium-y2
0.018411
walker24-random--22
0.000001
Table 1: RPSV calculated using normalized return.
16

---

Published as a conference paper at ICLR 2023
A.7 DETAILS OF IMPLEMENTATION
Temperature a. For RW and AW, we use a = 0.1 for IQL and TD3+BC, and a 0.2 for CQL.
Trajectory advantage. We use linear regression approximate VP. We make training set (sr.0, G(T:)) VT; D, and train regression model on the training set. Implementation. We use the public codebase, d3rlpy (Takuma Seno, 2021). For each algorithm, we use the hyperparamters
is.
A.8 DETAILS OF EVALUATION PROTOCOL
Performance logging. Given an environment E and dataset DE, for each trial (i.e., random seed) we train each algorithm+sampler variant for one million batches using DE and rollout the learned policy the environment E for 20 episodes. The average return of the 20 episodes are booked as the performance of the trial.
Performance metric. Given list of empirical returns of eacy trial [g] 92, :..], interquantile mean (IQM) (Agarwal al., 2021) discards the bottom 25% and top 25% samples and calculate the mean.
A.8.1 PROBABILITY OF IMPROVEMENT
According to Agarwal et al. (2021), the probability of improvement in an environment is defined as:
N
N
1, Imms > ym..j
P(Xm 2 Ym
N
(1m Bin)), S(Em,i Mm..)
=
2 Fm, = Um..j
0, Tm.i ym.j+
where m denote an environment index, Im, and ym denote the samples of mean of return in trials of algorithms X and Y, respectively. We report the average probability of improvements
..MM1 P(Xm. 2 ) and its 95%-cconfeence interval using bootstrapping. MI Em
PI cannot be directly translated into "number of winning" since PI takes the stochasticity resulting from random seeds into account, measuring the probability of improvements in trial with a randomly selected seed, dataset, and environment. For example, in Figure 9, we show that our methods attain 70% probability of improvements over uniform sampling, while this does not mean sampling in 70% of datasets and environments. From the complete score table in Appendix A.13, we that AW and RW strategies outperform uniform sampling in at least 80% of datasets.
beat uniform
We want to highlight that probability of improvement (PI) measures the robustness of a method, conveying different messages than the average performance shown in Figure 2 and Figure 4a. PI measures "how likely a method to perform better than uniform sampling in a randomly selected environment, dataset, and random seed?" PT captures the uncertainty among random seeds while aggregated metrics like average performance does not. For example, suppose we have 5 trials with different random seeds on the same environment and dataset for two methods A and B. The fact that A has higher average return than B, does not follow that A always performs better than B in all trials. It is possible that worse than in some trials. Comparing only the average return, one would mis-conclude that A is certainly better than B. Instead, PI answers "how likely is A to be better than B?
PI is important for algorithm selection since it measures the robustness of method. One can have extremely high performance gain in few tasks and lose baselines in the majority of tasks. so, this new method would not always outperform baselines, which makes it not robust. A robust method should consistently improve baselines and not lose performance in most tasks.
Robustness of method is important for user to decide whether or not to prefer the new method over the existing method (i.e., baselines). As offline RL algorithms' performance interplay with several factors (e.g., dataset properties, environment dynamics, reward functions, etc). it is unlikely to accurately predict what conditions make the new method perform the best. Lacking of perfect knowledge of the best condition for the new method, it is unclear whether user should deploy the
17

---

Published as a conference paper at ICLR 2023
new method on task that does not have benchmarking results yet. As result, robustness is crucial when selecting an algorithm for a new task. If the new method is shown to be robust and perform better than the baseline in most trials (i.e., high PD, it would be worth preferring the new method over the baseline. In contrast, it's not worth using the new method the new method has PI below 50
A.9 SENSITIVITY TEMPERATURE
The temperature (Section 4) is important hyperparameter of our method. We investigate how sensitive the choice of temperature algorithms is. Using the evaluation metric shown Section 5.1, compare the performance of RW and AW paired with IQL at varying temperature in Figure 8, where 0.1 is the temperature used in Sections 5.2, 5.4, and 5.5. Our methods outperform uniform sampling in range of temperatures and hence not overly sensitive temperature. The full results in other offline algorithms are presented in Appendix A.12.
A.10 PROBABILITY OF IMPROVEMENTS
In addition to average performance, the recent study by Agarwal et al. (2021) highlights the importance of measuring the robustness of an algorithm by its probability of improvement (PI) since outliers could dominate the average performance. necessarily perform better than baselines in the majority of environments. Therefore, evaluate our method in both regular (Section 5.4) and mixed (Section 5.2) datasets D4RL using probability of per- forming better than uniform sampling: PI(X Uniform), where X {Half. Top- 1op-10%, AW). The bottom row of Figure 9 shows that our method achieves above 70% chance of outperforming uniform sampling in mixed datasets with sparse high-return trajectories. Moreover, the lower bounds of the confidence interval are clearly above 50%, which indicates that the improvements are significant according to Agarwal et al. (2021). On the other hand, in regular datasets with abundant high-return data, PI > Uniform) and PI(RW > Uniform) are around 50%, suggesting that our methods match uniform sampling baseline. Note that PI(RW> > Uniform) 70% does not imply our method only beats uniform sampling in 70% of datasets and enviromments. The calculation of PI is detailed in Appendix A.8.1.
algorithm with higher average performance does not
A.11 ADDITIONAL RESULTS OF PERCENTAGE FILTERING
Figure 10 presents the additional results at varying percentage for percentage filtering.
A.12 ADDITIONAL RESULTS OF TEMPERATURE SENSITIVITY
Figure 1l presents the full results at varying temperatures.
A.13
FULL RESULTS
We list the full benchmark results in Tables 3, 4, and 5, where bold text denotes a score higher than Uniform and agn sign indicates the maximum score in dataset and environment (row).
A. 14 RESULTS OFFICIAL IQL CODEBASE
As the performance of our IQL implementation slightly mismatches the official implementation' we run the experiments in Sections 5.2 and 5.4 based on the official codebase and report the results in Figure 12. It can be seen that our methods still exhibit similar amounts of performance gain shown in Figure 2 and Figure 4a, indicating that the performance gain of our methods are independent of implementation.
thw.. http://///aiiiooooo////a/reaaaaa
18

---

Published as a conference paper at ICLR 2023
AW (ours)
RW (ours)
Top-10%
Uniform
Dataset
ent-expert-V2
0.31
0.31
0.05
0.30
(a (1t-medium-<220
0.34
0.35*
0.06
0.31
halfchectth--epper-2222 (ar
0.02
0.03*
0.01
0.02
halfchectah-mmddiu-222
0.02
0.03*
0.02
0.02
hopper-expert--v2 (o
0.17
0.16
0.03
0.04
hopper-mediumm-22 (a
0.22
0.34*
0.04
0.04
walker2d-exxpert-v2 (or =1
0.02
0.02
0.01
0.01
walker2d-meciuum-222 (Ø 1)
0.03
0.05
0.02
0.01
ant-cxpert-v2
1.01
1.08*
0.04
0.31
ant-mcdium-v200 (a 5)
0.86
0.86
0.10
0.31
halfcheetall-expert--22 (a 5)
0.14*
0.11
0.02
0.02
halfcheetah--medium-22 (a 5)
0.36*
0.36
0.14
0.02
hopper--xxpert-<2266 (or
0.97
0.87
0.05
0.04
hopper-medium--V2 (a
0.54*
0.49
0.15
0.05
walker2d-eexpert-22 (a = 5)
1.03*
1.01
0.01
0.01
walker20-meedium-222 (o 5)
0.60*
0,56
0.11
0.02
ant-expert-v2 10)
1.19
1.19
0.15
0.34
ant-medium--22 10)
0.85
0.91
0.27
0.45
halfchhctal--cxpeet-22666 (Ø 10)
0.68
0.76
0.77
0.02
hallcheeral--meddum--2266. 10)
0.40
0,42
0,42
0.02
hopper-exxpert-22 10)
1.03
1.05
0.06
0.05
hopper-meedium--2 (a 10)
0.57
0.57
0.23
0.05
walker2d-eexpert-222 10)
1.08
1.06
0.01
0.03
walker2d-medium--00 (o 10)
0.68
0.62
0.30
0.06
ant
exteexxprt-v22 50)
1.24
1.21
1.26
0.44
ant met-mmddum--2 50)
0.88
0.89*
0.89
0.73
halfchectta--expeet-22 (ar 50)
0.92
0.92*
0.40
0.80
halfcheetal--meddiu-222 (a 50)
0.42*
0.42
0.41
0.10
hopper-exxpent-v22 (or 50)
1.10*
1.10
0.86
0.04
hopper-mediumm-v2 (a 50)
0.56
0,57
0.51
0.03
walker2d-exxpert-22 50)
1.08*
1.08
0.25
0.01
walker2d-mediumm-e (o 50)
0.71
0.72
0.59
0.10
ant-expert-v2
1.25*
1.24
1.23
1.25
ant-full-replay-v22
1.26
1.28
1.25
1.18
ant-medium-expent
1.25
1.27
1.23
1.17
ant-medium--replay--22
0.80
0.79
0.66
0.67
ant-medium-v2
0.86
0.88
0.93*
0.86
ant-random-v2
0.30
0,29
0.06
0.32*
antmaze-large saarg-diivers-vee
0.12
0.16*
0.00
0.00
antmaze-large
laa----rag-e--y--y-- play-v0
0.09
0.13*
0.00
0.00
antmaze-medium
-iiee---iiiiiddide---dddd diverse-v0
0.10
0.29*
0.01
0.00
antmaze-meedimmpplay--a
0.13
0.21*
0.01
0.00
antmaze-umare-divvree.
0.58
0.65*
0.51
0.54
antmaze-umaze-vO
0.59
0.60
0.65*
0.49
halfcheetah--exeert-22
0.92
0.92
0.70
0.92
0.67
0.67
0.64
0.62
halfcheetah- satittaah-eeium--eee -expetttt--eee-2-2222
0.92
0.92*
0.88
0.58
halfchectah- medium- -22
0.38
0.40*
0.31
0.34
halfchectah medium-v2
malfchectahmmeciu--222
0.42
0.42
0.42
0.43*
halfchectah- random-v2
0.02
0.02
0.02
0.02*
hopper-expert-v2
1.08
1.11*
1.09
hopper-full-replla--22
0.98
1.01
0.93
0.31
hopper-mediimm-expert-22
1.10
1.10*
1.09
0.53
hopper-mediuum-eellay...
0.72*
0.67
0.63
0.27
hopper-medium-
0.56
0.54
0.58*
0.52
hopper-random--2
0.05*
0.05
0.05
0.04
walker2d-experrtv22
1.09*
1.08
1.03
1.08
walker-ad-full--eppay--22
0.84
0.84
0.88*
0.27
walker22d-mediuu--spprryyy
1.08
1,08
1.08
0.97
walkerz2o-meediun--eelayyyy
0.56
0.56*
0.51
0.19
walker2d-mmedium-22
0.70
0.71
0.70
0.64
walker2d-random--
0.01
0.01
0.01
0.01
Num. win Uniform
58
58
39
Table 2: BC results
19

---

Published as a conference paper at ICLR 2023
AW (ours)
RW (ours)
Top-10%
Uniform
Dataset
ent-expert-V2
0.54*
0.54
0.06
0.25
(a (1t-medium-v22
0.82
0.82*
0.07
0.39
halfchectta--epper-2222 (ar
0.03
0.04
0.03
0.03
halfchectah-mmcdiu-222
0.16
0.20
0.07
0.21
hopper-expert--v2 (o
0.55*
0.50
0.11
0.12
hopper-mediumm-22 (a
0.39
0.50
0.30
0.53I
walker2d-exxpert-v2 (or =)
0.29
0.28
0.10
0.12
walker2d-meciuum-222 (Ø
0.46
0.48
0.12
0.36
ant-cxpert-v2
1.16
1.19*
0.12
0.60
ant-mcdium-v2(0 (a 5)
0.88
0.91
0.28
0.73
halfcheetall-expert--22 (a 5)
0.66
0.71
0.04
0.04
halfcheetah--medium-222 (o 5)
0.43*
0.41
0.30
0.33
hopper--expert-v2266 (or
1.02*
0.87
0.18
0.12
hopper-mediumm-V2 (a
0.53
0.56*
0.48
0.49
walker2d-eexpert-22 (a = 5)
1.08
1.08
0.53
0.25
walker20-meeiium-222 (o 5)
0.58
0.61
0.49
0.58
ant-expert-v2 10)
1.18
1.23*
0.40
0.74
ant-medium--22 10)
0.87
0.90*
0.62
0.79
halfchhctaah-cxpeet-22666 (Ø 10)
0.85
0.89*
0.85
hallcheeral--meddiu--2226 10)
0.45
0.45
0.45*
0.40
hopper-exxpert-22 10)
1.00
1
0.36
0.18
hopper-meedimm-22 (a 10)
0.55
0.60*
0.54
0.46
walker2d-eexpert-222 10)
1.09*
1.08
0.92
0.66
walker2d-medium--00 (o 10)
0.67*
0.64
0.66
0.65
ant
ext-exxprt-v22 50)
1.21
1.21
1.24
1.07
ant met-mmddum--2 50)
0.92
0.94
0.96*
0.91
halfchectaa--expeet-22 (ar 50)
0.91
0.94*
0.70
0.50
halfcheetal--meddiu-222 (ar 50)
0.47
0.47*
0.44
0.45
hopper-exxpent-v22 (or 50)
1.06
1.07
0,43
0.50
hopper-mediumm-v2 (a 50)
0.54
0.52
0.62
0.56
walker2d-exxpert-22 50)
1.09*
1.09
1.08
1.08
walker2d-mediumm-V (o 50)
0.63
0.63
0.71
ant-expert-v2
1.25
1.26
1.27*
1.13
ant-full-replay-v22
1.29*
1.27
1.25
1.24
iit-medium-experte
1,23
1,32
1.28
1.14
ant-medium--replay--22
0.83
0.86
0.72
0.83
ant-medium-v2
0.97
0.98
0.88
0.99*
ant-random-v2
0.14*
0.14
0.06
0.12
antmaze-large saarg-ddivers-vee
0.40*
0.24
0.04
0.00
antmaze-large
laae---aagee-----y--- play-v0
0.18
0.33*
0.00
0.00
antmaze-medium
-eiii--iii-idddi----dddd diverse-v0
0.24
0.31
0.02
0.05
antmaze-meddimmpplay-aa
0.41
0.43*
0.03
antmaze-umare-divvree.
0.54
0.40
0.61
0.46
antmaze-umaze-vO
0.87*
0.86
0.72
0.85
halfcheetah--exeert-22
0.94
0.93
0.74
0.93
halfcheetah-full rull------y--
0.75#
0.75
0.71
0.70
halfcheetah- weteeeatah-eeiim----e -xxpeett--ee
0.94*
0.93
0.89
0.71
halfchectah- medium- -22
0.44*
0.44
0.35
0.44
halfchectah
malfchectahmmeciu--222 medium-v2
0.47
0.47*
0.45
0.47
halfchectah- random-v2
0.07
0.07
0.04
0.12
hopper-expert-v2
1.00
0.96
0.92
0.95
hopper-full-replla--22
0.97*
0.79
0.97
0.89
hopper-mediim--expert-22
0.99
1.01
1.05
0.59
hopper-mediuum-replay...
0.84
0.86
0.85
0.83
0.57
0.57
0.65*
0.55
hopper-random--2
0.06
0.07
0.08
0.09*
walker2d-experrtv22
1.09*
1.09
1.09
1.09
walker-ad-full--pppay--22
0.75
0.83
0.76
0.92*
walker2dd-mediuu-eeppeey
1.10*
1.09
1.09
1.06
walkerz2o-meddiun--eelayyyy
0.47
0.37
0.51
0.69
walker2d-mmedium-22
0.69
0.66
0.65
0.75*
walker2d-random--
0.03
0.04
0.11*
0.06
Num. win Uniform
48
47
29
Table 3: IQL results
20

---

Published as a conference paper at ICLR 2023
AW (ours)
RW (ours)
Top-10%
Uniform
Dataset
ent-expert-V2
0.80*
0.74
0.04
0.08
(a (1--medium-v22
0.79
0.80*
0.06
0.14
halfchectta--exper--222 (ar
0.39*
0.38
0.09
0.25
halfchectah-mmcdiu-222
0.41
0.42*
0.26
0.40
hopper-expert--v2 (o
0.84*
0.52
0.15
0.09
hopper-mediumm-22 (a
0.61*
0.58
0.43
0.30
walker2d-exxpert-v2 (or =)
0.57
0.54
0.01
0.03
walker2d-meciuum-222 (Ø 1)
0.46
0.48
0.12
0.03
ant-cxpert-v2
1.06
1.06*
0.35
0.15
ant-mcdium-v200 (a 5)
0.88
0.93*
0.57
0.50
halfcheetall-expert--22 (a 5)
0.67
0.66
0.10
0.30
halfcheetah--medium-222 (o 5)
0.47*
0.47
0.44
0.47
hopperr-xxpert-<2266 (or
0.99*
0.99
0.31
0.10
hopper-medium--V2 (a
0.65
0.68*
0.63
walker2d-eexpert-22 (a = 5)
1.06*
1.02
0.00
0.03
walker2d-mediium-222 (o 5)
0.76
0.75
0.41
0.04
ant-expert-v2 10)
1.11
1.16
0.93
0.21
ant-medium--22 10)
0.91
0.96*
0.91
0.57
halfcheceta--cxxeett-2666 (Ø 10)
0.76
0.76*
0.74
0.29
hallcheeral--medium--2266 10)
0.48
0.48
0.47
0.48
hopper-exxpert-22 10)
1.08
1.03
0.37
0.14
hopper-meedium-22 (a 10)
0.68
0.71
0.72
0.62
walker2d-eexpert-222
10)
1.02
1.09*
0.12
walker2d-medium--00 (o 10)
0.79*
0.79
0.68
0.24
ant ext-exxprt-v22 (a 50)
1.26
0.53
1.27
0.83
ant met-mmddum--2 50)
0.97
0.94
0.98
0.82
halfchectaa--expeet-22 (ar 50)
0.80
0.86
0.56
0.57
halfcheetal--meddiu-222 (ar 50)
0.49
0.49
0.47
0.49*
hopper-exxpent-v22 (or 50)
1.02
1.04
1.04*
0.82
hopper-mediumm-V2 (a 50)
0.63
0.38
0.72
0.73*
walker2d-exxpert-22 50)
1.09*
1.08
0.96
0.24
walker2d-mediumm-e (o 50)
0.82
0.83*
0.81
ant-expert-v2
1.27*
1.27
1.16
1.21
ant-full-replay-v22
1.27
1.24
1.31
1.22
ant-medium-expent
1,28
1.22
1.23
1.17
ant-medium--replay--22
0.91
0.97
0.83
0.90
ant-medium-v2
0.95
0.97
0.96
0.98
ant-random-v2
0.08
0.08
0.06
0.11*
antmaze-large saarg-ddivers-vee
0.00
0,00
0.08
0.00
antmaze-large
laag--arag--eyy----- play-v0
0.00
0
0.00
0.00
antmaze-medium
-iier---iiiidddede---dddd diverse-v0
0.00
0.08*
0.00
0.00
untmaze-meedimmpplay--a
0.00
0.03*
0.01
0.00
antmaze-umare-divvree.
0.22
0.10
0.23*
0.05
antmaze-umaze-vO
0.65
0.71*
0.25
0.65
halfcheetah--exeert-22
0.89*
0.88
0.77
0.79
0.79
0.79*
0.77
halfcheetah- weteeeatah-eeiim----e -expettt-2eeeeee-2222
0.84
0.85*
0.70
0.63
halfchectah- medium- r2
0.47
0.47
0,42
0.47*
halfchectah medium-v2
malfchectahmmeciu--222
0.49
0.49*
0.47
0.49
halfchectah- random-v2
0.17
0.17
0.03
0.23*
hopper-expert-v2
1.05
1.01
1.05
hopper-full-replla--22
1.03
1.07
1.01
1.00
hopper-mediim--expert-22
0.91
1.00
1.04
0.99
hopper-mediuum-eellay...
0.99*
0.99
0.96
0.95
hopper-medium-
0.71
0.73
0.71
0.74
hopper-random--2
0.27*
0.24
0.08
0.12
walker2d-experrtv22
1.09
1,09*
1.09
1.09
walker-ad-full--eppay--22
0.86
0.85
0.91
0.94*
walker22d-mcdiuu--epppryyy
1.09
1.08
1,09*
1.08
walkerz2o-meediuu---epaayyy
0.87#
0.86
0.71
0.84
walker2d-mmedium-22
0.83*
0.82
0.74
0.83
walker2d-random--
0.14
0.14
0.17*
0.03
Num. win Uniform
50
50
33
Table 4: CQL results
21

---

Published as a conference paper at ICLR 2023
AW (ours)
RW (ours)
Top-10%
Uniform
Dataset
ent-expert-V2
0.18
0.17
0.03
0.25*
(a (1t-medium-x22
0,48
0,46
0.01
0.39
halfchectta--epper-2222 (ar
0.04
0.04
0.08
0.04
halfchectah-mmcdiu-222
0.21
0.18
0.31
0.16
hopper-expert--v2 (o
0.57
0.52
0.12
hopper-mediumm-22 (a
0.53
0.54*
0.21
0.30
walker2d-exxpert-22 (or 1)
0.69
0.90*
0.03
0.05
walker2d-medfuum-222 (Ø 1)
0.18
0.17
0.06
0.13
ant-cxpert-v2
0.44
0.40
0.08
0.32
ant-mcdium-v200 (a 5)
0.99*
0.81
0.25
0,58
halfcheetall-expert--22 (a 5)
0.59
0.62
0.08
0.14
halfcheetah--medium-222 (o 5)
0.47*
0.47
0.45
0.30
hopper--xxpert-<2266 (or
0.97*
0.88
0.44
0.42
hopper-mediumm-V2 (a
0.55
0.55*
0.42
0.38
walker2d-eexpert-22 (a = 5)
0.66
0.70*
0,02
0.13
walker20-meedium-222 (o 5)
0.64
0.78
0,04
0.05
ant-expert-v2 10)
0.53*
0.46
0.16
0.46
ant-medium-v2 10)
1.09*
1.05
0.39
0.60
halfchhctal--cxpeet-22666 (Ø 10)
0.79
0.82*
0.81
0.31
hallcheeral--meddiu--222 10)
0.48
0.47
0.48
0.43
hopper-exxpert-22 10)
1.02
1.05
0.64
0.65
hopper-meedium--2 (a 10)
0.57
0.57
0.53
0.24
walker2d-ecxpert-22 10)
1.10
0.81
0.03
0.08
walker2d-medium--00 (o 10)
0.76
0.62
0.03
0.06
ant
ext-exxprt-v22 50)
0,48
0,57
0.47
0.51
ant met-mmddum--2 50)
1.11
1.06
1.00
0.90
halfchectaa--expeet-22 (ar 50)
0.94
0.95*
0.69
0.80
halfcheetal--meddiu-222 (a 50)
0.48*
0.48
0.47
0.48
hopper-exxpent-v22 (or 50)
1.11*
1.10
1.01
0.97
hopper-meddumm-v2 (a 50)
0.62*
0.60
0.52
0.44
walker2d-exxpert-22 50)
1.10*
1.10
0.07
0.08
walker2d-mediumm-e (o 50)
0.80
0.81
0.34
0.07
ant-expert-v2
0.96
0.98*
0.83
0.55
ant-full-replay-v22
1.34
1,38
1.28
1.34
cit-medium-experte
1.13*
1.12
0.95
0.70
ant-medium--replay--22
0.96
1.00
0.64
1.06
ant-medium-v2
1.17
1.16
1.20
ant-random-v2
0,22
0,36
-0.02
0.37
antmaze-large saarg-ddivers-vee
0.00
0.01
0.03*
0.00
antmaze-large laae---aage--------e--
0.00
0.00*
0.00*
0.00
antmaze-medium
-iier--eiiiidddede---dddd diverse-v0
0.10*
0.05
0.07
antmaze-meddummpplay-ae
0.07*
0.01
0.04
0.00
antmaze-umarz-diveree.
0.52*
0.26
0.17
0.47
antmaze-umaze-vO
0.50
0.72
0.51
0.79*
halfcheetab--exeert-22
0.97
0.97
0.84
0.95
halfcheetah-full-reee full-replay--2
0.78*
0.77
0.76
halfcheetah- weweeeathh-eeiim----e expertt-22
0.97*
0.96
0.86
0.87
halfchectah- medium- r2
0.44
0.45*
0.34
0.44
halfchectah
malfchectahmmeciu-->22 medium-v2
0.48
0.48
0.48
0.48
halfchectah- random-v2
0.11
0.11
0.12*
0.11
hopper-expert-v2
1.05
1.06
hopper-full-replla--22
1.04*
1,03
1.02
0.73
hopper-mediimm-expert-22
1.03
1.10*
1.10
0.96
hopper-mediium-eeplay...
0.93
0.95*
0.85
0.59
hopper-medium-
0.62
0.62
0.64
0.56
hopper-random--2
0.07
0.07
0.08
0.10*
walker2d-experrtv22
1.10*
1.10
1.10
1.10
walker-ad-full--eppay--22
0.96*
0.96
0.92
0.94
walker22d-mcdduu--eepprtyy
1.10
1.10
1.10*
1.10
walkerr2--mediiu--eepay--2
0.80*
0.75
0.78
0.80
walker2d-mmedium-22
0.80
0.81
0.67
0.82*
walker2d-random-y
0.03
0.02
0.02
0.04*
Num. win Uniform
49
48
25
Table 5: TD3+BC results
22

---

Published as a conference paper at ICLR 2023
ha.kihecat-mrrsiime (owu)
bakthetdhnmmtmmm Jam201
m lowst)
Tuttchethhheel unis 2001
12
-022 0:6
402
20
low
holltodaal ospett 12
bu www.aaaa..a...ooooo.o.. 100
Nolhhech CRouriN tom
hafthhttthcoettteeoooo fow 1000
1.7
5.
-0.32 -022 0.66
101
3.00
0.00
0.23
250
0.75
1.0
0.8
2.5
16
00
3.0
6.5
13
32
04
0.6
0.8
(a) Trajectory return distribution of HalfCheetah
art-mallam (s
on-mmcclmma22 (om
atc modiimm-m [o
arkmellumst (pw 1951
15.0
032
L15 oxpet1-22 fo
0nt-uap-r002 1o
urt.cepoo--222 (ow 101
unticepettv (ow50)
a05 wwc-sssotoo v2 to 100)
0.32
(b) Trajectory return distribution of Ant
Bww2e meciumm2 (p
wwker22 mo 1g
Belle2 mediu Jp: 13)
berker mediumi2 mV 50)
willoee medium 22 jo 130)
402
150
300
0..0000.302 0.202 0:222 8.05
1.00
M4 42 (ow
MJ www.sppe--/aeeooo (pw1)
kc sapprt-p
W. www.o2dloogg sepesppeettt (ow501
9e Re 22-03p51-1000 (ow 200)
100
120
100
30
200
10
20005 0.005 0035 0.53 0.05
300
025
035
115
500 029
0.99
6.7
3,00
5.25
0.38
375
3.00
(c) Trajectory return distribution of Walker2d
hontemedumm0222 (g
22spn-medmm22 1p
hoppe-meetammi222 (p-
aeppen umm-0 um-40
500
hep heporemmdium--0 to 1031
320
Topoer www.otoootooooooooo
o2 (ow
hosper oo 10w
Tepaer expurt12 fow
1oppor-copptoo 2001
2.00
32.5
32.0
7.5
Roum
Robm
Nouu
1ctum
Trajectory return distribution of Hopper
Figure 7: Trajectory return distribution of each mixed dataset.
23

---

Published as a conference paper at ICLR 2023
RW
AW
1.0
Normalized return
0.8
Uniform
0.01
0.6
0.1
0.5
0.4
1.0
0.2
0.0
0.2
0.4
0.0
0.2
0.4
Ratio
Ratio
Figure 8: Performance of our method with varying temperature a (Sections 4.2 and 4.3), where color denotes Both AW and RW achieve higher returns than the baselines in wide range of temperatures [0.01, 1.0]. our methods are not overly sensitive choice of temperature.
Regular (dense high-return data)
Mixed (sparse high-return data)
100%
100%
PIX Uniform)
75%
75%
50%
50%
Top-10%
RW (ours)
AW (ours)
Figure 9: Probability of improvement over uniform sampling (Agarwal et al., 2021) in 32 mixed (lower row) and 30 regular datasets (upper row). In mixed datasets with sparse high-return data, our methods attains above 75% of the probability of improvements with lower bound of confidence interval clearly above 50%, suggesting statistically significant improvements over uniform sampling. On the other hand, in regular datasets with abundant high-return data, PI(AW > Uniform) and PI(RW Uniform) are around 50%, suggesting that our methods match uniform sampling baseline.
IQL
BC
Normalized retur
0.8
0.6
Uniform
Top-50%
0.4
Top-30%
Top-10%
0.2
Top-1%
0.0
0.0
0.2
0.4
0.0
0.2
0.4
Ratio
Ratio
Figure 10: Performance of percentage-filtenn sampling with varying percentages.
24

---

Published as a conference paper at ICLR 2023
Algo IQL RW
Algo IQL AW
1.0
Normalized return
0.8
0.6
0.4
0.2
Algo = CQL RW
Algo CQL AW
1.0
Normalized return
0.8
Uniform
0.01
0.6
0.1
0.2
0.4
0.5
1.0
0.2
Algo TD3PlusBC RW
Algo TD3PlusBC AW
1.0
Normalized return
0.8
0.6
0.4
0.2
0.0
0.2
0.4
0.0
0.2
0.4
Ratio
Ratio
Figure 11: Performance at varying temperature.
25

---

Published as a conference paper at ICLR 2023
Normalized average return
1.0
Uniform Top-10%
0.8
Half
(ours)
Normalized average reture
0.6
AW (ours)
Sg
50%
Ratio
Top-10%
(ouns)
(ours)
(b)
Figure 12: Our results official implementation shows similar results to the results from our implementation in Sections 5.2 and 5.4. (a) The average return of cach sampling strategy in mixed dataszets (Section 5.2) official IQL codebase. We see that our methods also show higher average return than the baselines in similar trend shown in Figure 2, suggesting that the performance gain of our methods are independent of implementation choices. (b) The average returns of each sampling methods in regular datasets (Section 5.4). We see that methods show insignificantly performance in over uniform sampling, similar to the results in Figure 4a.
26


Reviews:
[{"review_id": "zABxdzVCs", "reviewer": "Reviewer_C84m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to reweight episodes in the replay by their advantage values. The paper claims that doing so is especially useful when there are many more low return episodes in the replay buffer compared to high return episodes. ", "summary_of_the_review": "I look forward to the rebuttal. The idea is simple, and may therefore be used widely. But as it currently stands, it is unclear if the method outperforms simple and common baselines.", "strength_and_weaknesses": "Strength\n\nThe proposed idea is intuitive and the claim that the idea is useful when there are many more low return episodes than high return episodes makes sense. \n\nWeaknesses\n\nLack of comparison to simple baseline. The issue that the paper studies is well-known, and a common technique to tackle it is to ensure each batch of episodes used to update the networks contain half successful, and half failed episodes. I would have liked to see the comparison to this simple and common baselines.\n\nParts of the writing are not clear. For example:\n\n- \"cold start performance boost\" in the Introduction.\n\n- \"Most offline RL algorithms are anchored to the behavior policy\". What are anchors mean, and where is the evidence that support this claims?\n\n- How should I interpret figure 2? The caption states that the proposed methods outperform baselines, but how is this conclusion supported by the figure?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WMx6UuDNpx", "reviewer": "Reviewer_BiNt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores a particular setting of low sparse high reward trajectories in an offline RL datasets. Offline RL algorithms that explicitly do Behavioral Cloning (BC) have a hard time learning good policies in these settings. The authors propose a simple reweighing scheme to induce an artificial dataset with a higher performing behavioral policy. They also show theoretical analysis regarding the bias of this estimate for deterministic settings and also introduce Return Positive Sided Variance as a measure of quantifying the variance in the returns of trajectories among the dataset. Finally the experimental results show that the weighted sampling of the dataset significantly helps in the artificially generated datasets with varying RPSV however the performance gains on true D4RL tasks are marginal especially for methods without explicit BC like CQL and IQL.", "summary_of_the_review": "Overall the paper presents a relative simple weighted sampling method which can be a valuable tool in an offline-rl toolkit, however, the metric RPSV presented in the paper fails to capture the scenario where the method can/should be applied. This can be an interesting paper with a better metric / analysis on when the approach should be considered during offline RL, however, under the current version, I am leaning toward a rejection.", "strength_and_weaknesses": "Strengths:\n\n- The weighted sampling helps combating the need for a near expert dataset for approaches with explicit behavioral cloning terms.\n    \n- The weighted sampling approach has been demonstrated to not hurt the performance even for well balance datasets across various modern offline rl approaches (CQL and IQL).\n    \n\nWeakness:\n\n- The results while impressive is not surprising.  While the gains are signficant for BC and TD3+BC, it is well established that these methods will suffer in sparse high reward trajectory settings. The gains over CQL and IQL are marginal for the current D4RL benchmark.\n    \n- While the authors introduce RPSV as a measure of tracking the variance in the dataset, it does not answer the question of “When is it necessary to do weighted sampling?”. i.e. The RPSV metric follows a inverted U curve as we increase sigma from 1% to 100% in the artifical datasets that were created.\n    \n- For this approach to be applicable, the data needs to be collected such that the trajectories are complete; i.e. The behavioral policies have to start from the starting state distribution. While this is not that hard of a constraint it is to be noted that this approach does not make sense if the collected policies start from different parts of the state-space. (i.e. trajectory stitching settings.)\n    \n- Probability of improvement over uniform sampling 3(b) is very skewed compared to the actual amount of improvement we see in figure 3(a). Here it seems that our approach is equally valid for both IQL and TD3+BC according to fig 3(b) whereas 3(a) tells a different story. overall I feel fig 3(b) does not add much value.\n\nMinor points.\n\n- ““However, offline IL requires expert and random data to be separated while we neither assume separated datasets nor access to expert data.” ([pdf](zotero://open-pdf/library/items/BFZJEQLU?page=9)) ” This is not fully accurate as we are using the rewards to rescale sampling we are assuming proxies to both a “separate dataset” and access to “expert data” offline IL does not require a expert data, its just that it wont work without one, as with the case with this approach.-  \n    \n- sigma has different scale in page 14 and 15. (0.01 and 1 )\n    \n- The formatting /Bolding has errors in the tables in appendices.  \n    \n- It would be wonderful to include plans to release the sampling code snippets for the public.\n    \n- IQL numbers for hopper is not matching with the author provided numbers, It might be good idea to doublecheck them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AnyEinotOIg", "reviewer": "Reviewer_56VB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle a long-known problem in offline learning where the performance of the RL agent is highly dependent on the distribution of high and low return trajectories. It follows that in datasets that contain a high percentage of low-returning trajectories and a low percentage of high-returning trajectories. The Agent will be overly restrained by the low-return trajectories. \n\nThe authors tackle this problem by implementing trajectory weighting, where different weights are assigned to different trajectories of different importance, naturally a more important trajectory would have a more important weight. After assigning weights to each trajectory, the authors add an entropy regularization term.", "summary_of_the_review": "About Challenge: \n- Without collecting additional data, how can the performance of the policy be improved? Previous works have tackled this by skewing their sampling.\n\nAbout Contribution: \n- The authors reweight the transitions in the dataset, and regularize it by adding an entropy regularization term. Then they further change the reweighting term to equation 11 in the paper.", "strength_and_weaknesses": "Main concern: \n- How prevalent is this problem in offline RL? I see the table in A.3, but it does not give the whole image of this problem. Is this problem seen in other mainstream datasets?\n- How does this method compare to the other sampling strategy methods? Why should this method be used over the other ones?\n- Would making the model learn the policy on offline datasets make the model overfit?\n- Is this method robust under different distributions of dataset? For example in Figure 1, what if the dataset has two density spikes, on 0.0, and 0.8.\n- In the result part, there is no comparison between the two methods, the authors only gave experiments on their method vs the vanilla method of no weighted distributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0ow0s2EUt1-", "reviewer": "Reviewer_Z9su", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is trying to solve the problem that when there exists a scenario in the offline RL dataset: Mostly low return trajectory, few high return trajectory, indicating the mixed policies lying behind, the results could be not ideal.\n\nBy introducing the RPSV, the variance of the positive case and negative cases are well captured in the original dataset and is helping to infer the distribution of high-return and low-return policies.\n\nAnd the author base on the variance of PSV proposed two weighting methods. Abundant experiment on CQL, IQL, TD3+BC, sole BC, etc. from various environments like classic control has justified the effectiveness of different methods. Detailed explanations of experiments and deductions are shown in the appendix.\n", "summary_of_the_review": "The author is trying to use novel ways to analyze the distribution of offline training datasets. Which helped to improve the performance of the offline RL methods and also justify the applicability to other methods. The paper is presented in an explicit way, also with necessary details from the appendix part. \n\nIt’s meaningful work to help offline RL algorithms improve the exploitation in the limited amount of high return trajectories.\n", "strength_and_weaknesses": "Strength:\n\n- The novelty is clear. The author is trying to analyze the mixed dataset’s efficiency in a theoretical way, representing and capturing the tendency of PSV.\n\n- The author proposed the two methods in a rigorous logic, return-weighting and advantage weighting are straightforward by the definition. \n\n- The sufficient experiments are evidently showing that the proposed two methods on different baseline methods, could better capture the dataset’s trajectory original feature, and by reweighting the importance of the high-return trajectory, the method’s effectiveness is well improved.\n\nWeakness:\n\n- It’s noticeable from Figure2 that the two weighting methods sometimes have a different appearance, it would be better if the author could analyze the different suitable situations for the above different two methods.\n\n- A little confusing part may be the introduction of negative-sided variance(NSV), because the majority part of the paper is mainly focusing on the RPSV, is NAV the NSV used in the implementation part of two re-weighting methods? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster