ConserWeightive Behavioral Cloning for Reliable Offline Reinforcement Learning
Authors: Tung Nguyen
Qinqing Zheng
Aditya Grover
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline RL
behavioral cloning
conservatism

Abstract: The goal of offline reinforcement learning (RL) is to learn near-optimal policies from static logged datasets, thus sidestepping expensive online interactions. Behavioral cloning (BC) provides a straightforward solution to offline RL by mimicking offline trajectories via supervised learning. Recent advances~\cite{chen2021decision, janner2021offline, emmons2021rvs} have shown that by conditioning on desired future returns, BC can perform competitively to their value-based counterparts, while enjoying much more simplicity and training stability. However, the distribution of returns in the offline dataset can be arbitrarily skewed and suboptimal, which poses a unique challenge for conditioning BC on expert returns at test-time. We propose ConserWeightive Behavioral Cloning (\name), a simple and effective method for improving the performance of conditional BC for offline RL with two key components: trajectory weighting and conservative regularization. Trajectory weighting addresses the bias-variance tradeoff in conditional BC and provides a principled mechanism to learn from both low return trajectories (typically plentiful) and high return trajectories (typically few). Further, we analyze the notion of conservatism in existing BC methods, and propose a novel conservative regularizer that explicitly encourages the policy to stay close to the data distribution. The regularizer helps achieve more reliable performance, and removes the need for ad-hoc tuning of the conditioning value during evaluation. We instantiate \name{} in the context of Reinforcement Learning via Supervised Learning (RvS)~\cite{emmons2021rvs} and Decision Transformer (DT)~\citep{chen2021decision}, and empirically show that it significantly boosts the performance and stability of prior methods on various offline RL benchmarks.

Under review as a conference paper at ICLR 2023
CONSERWEIGHTIVE BEHAVIORAL CLONING FOR RELIABLE OFFLINE REINNORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of offline reinforcement learning (RL) is learn near-optimal policies from static logged datasets, thus sidestepping expensive online interactions. Behav- ioral cloning (BC) provides a straightforward solution to offline RL by mimicking offline trajectories via supervised learning. Recent advances (Chen et al., 2021; Jan- ner et al., 2021; Emmons et al., 2021) have shown that by conditioning on desired future returns, BC can perform competitively to their value-based counterparts, while enjoying much more simplicity and training stability. However, the distri- bution of returns in the offline dataset can be arbitrarily skewed and suboptimal, which poses a unique challenge for conditioning BC on expert returns at test-time. We propose ConserWeightive Behavioral Cloning (CWBC), simple and effective method for improving the performance of conditional BC for offline RI. with two key components: trajectory weighting and conservative regularization. Trajectory weighting addresses the bias-variance tradeoff in conditional BC and provides a principled mechanism to learn from both low return trajectories (typically plentiful) and high return trajectories (typically few). Further, we analyze the notion of conservatism in existing BC methods, and propose novel conservative regularizer that explicitly encourages the policy to stay close to the data distribution. The regularizer helps achieve more reliable performance, and removes the need for ad-hoc tuning of the conditioning value during evaluation. We instantiate CWBC in the context of Reinforcement Learning via Supervised Learning (RvS) (Emmons et al., 2021) and Decision Transformer (DT) (Chen al., 2021), and empirically show that it significantly boosts the performance and stability of prior methods on various offline RL benchmarks.
INTRODUCTION
In many real-world applications such as education, healtheare and autonomous driving, collecting data via online interactions be expensive historical logged datasets in these domains that have been collected previously by some unknown policies. The goal of offline reinforcement learning (RI.) is to directly learn effective agent policies from such datasets, without additional online interactions (Lange et al., 2012; Levine et al., 2020). Many online RL algorithms have been adapted to work in the offline setting, including value-based methods (Fujimoto et al., 2019; Ghasemipour et al., 2021; Wu et al., 2019; Jaques et al., 2019; Kumar al., 2020; Fujimoto & Gu, 2021; Kostrikov et al., 2021a) well as model-based methods (Yu et al., 2020; Kidambi et al., 2020). The key challenge in all these methods is to generalize the value or dynamics to state-action pairs outside the offline dataset.
dangerous. However, often have access to
An alternative way to approach offline RL is via approaches derived from behavioral cloning (BC) (Bain & Sammut, 1995). BC is a supervised learning technique that was initially devel- oped for imitation learning, where the goal is to learn policy that mimics the expert demonstrations. Recently, a number of works propose to formulate offline RL as supervised learning problems (Chen et al., 2021; Janner et al., 2021; Emmons et al., 2021). Since offline RL datasets usually do not have expert demonstrations, these works condition BC on extra context information to specify target out- comes such returns and goals. Compared with the based approaches, the empirical evidence has shown that these conditional BC approaches perform competitively, and they additionally enjoy the enhanced simplicity and training stability of supervised learning.

---

Under review as a conference paper at ICLR 2023
As commonly observed for supervised learning approaches, the performance of conditional BC is often limited by the suboptimility of the offline dataset, which particularly can be probed through the distribution of returns in the dataset. There are two related challenges in this regard for offline RL.
First, there is a unique mias-variance tradcoff in learning that arises due to the mismatch between the training and test distribution of returns. Typically, offline datasets in the real world mostly contain trajectories with low returns, whereas at test time, we are interested in conditioning on high returns. Simply filtering the offline dataset to contain high return trajectories not always viable, as the number of such high-return trajectories can be very low leading to high variance during learning.
Second, the maximum return in the offline trajectories is often far below the desired expert returns. This implies that at test time, we need to condition our agent on out-of-distribttion (ood) expert returns. Interestingly, we find that existing BC methods have significantly different behaviors when conditioning on ood returns. While DT (Chen et al., 2021) enjoys stable performance, RvS (Emmons et al., 2021) highly sensitive to such ood conditioning and exhibits vast drops in peak performance for such ood inputs. Therefore, the current practice for setting the conditioning return at test-time in RvS is based on careful tuning with online rollouts, which is often tedious, impractical, and inconsistent with the promise of offline RL to minimize online interactions.
We propose SonserWeightive Behavior Cloning (CWBC), a new BC-based approach for offline RL that mitigates the aforementioned challenges. CWBC consists of 2 key components: trajectory weighting and conservative regularization. With trajectory weighting, we strive to balance the bias- variance trade-off in learning by proposing scheme for downweighting the low-return trajectories, but at the same time, we do not filter them for data efficiency. Moreover, we introduce a notion of conservatism for ood sensitve BC methods such as RvS, which encourages the policy stay close to the data distribution when conditioning on large returns. We take trajectories with high returns from the dataset and add positive noise to their returns. which generates trajectories with large ood returns. We predict actions conditioning on the perturbed returns and project them to the original actions by penalizing the & distance. By imposing such a regularizer, we can condition the policy on large, unseen target returns at test-time, sidestepping tedious manual tuning and online interactions.
Our proposed algorithm is simple and easy to implement. Empirically, we instantiate our framework in the context of RvS (Emmons et al., 2021) and DT (Chen et al., 2021). two state-of-the-as BC methods for offline RL. CWBC significantly improves the performance of RvS and DT in D4RL (Fu et al., 2020) locomotion tasks by 18% and 8%, respectively, without any hand picking of the value of the conditioning returns at test-time.
2 RELATED WORK
Offline Temporal Difference Learning Most of the existing off-policy RL methods are often based on temporal difference (TD) updates. key challenge of directly applying them in the offline setting is the extrapolation error: the value function is poorly estimated at unseen state-action pairs. To remedy this issue, various forms of conservatism have been introduced to off-policy RL methods that exploits temporal difference updates, with the purpose of encouraging the learned policy to stay close to the behavior policy that generates the data. For instance, Fujimoto et al. (2019); Ghasemipour al. (2021) use certain policy parameterizations specifically tailored for offline RL. Wu et al. (2019); Jaques et al. (2019); Kumar et al. (2019) penalize the divergence-basee policy and the behavior policy. Fujimoto & Gu (2021) propose an extra behavior cloning term to regularize the policy. This regularizer is simply the & distance between predicted actions and the truth. yet surprisingly effective for porting off-policy TD methods the offline setting. Instead of regularizing the policy, several other works have sought to incorporate divergence regularizations into the value function estimation, e.g., (Nachum et al., 2019; Kumar et al., 2020; Kostrikov et al., 202 Another recent work by Kostrikov et al. (2021b) predicts the Q function via expectile regression, where the estimation of the maximum Q-value constrained to be in the dataset.
distances between the learned
Behavior Cloning Approaches for Offline RL Recently, there is surge of interest in converting offline RL into supervised learning paradigms (Chen et al., 2021; Janner et al., 2021; Emmons et al.. 2021). In essence, these approaches conduct behavior cloning (Bain & Sammut, 1995) by additionally conditioning on extra information such as goals or rewards. Among these works, Chen et al. (2021) and Janner et al. (2021) have formulated offline RI. sequence modeling problems
2

---

Under review as a conference paper at ICLR 2023
and train transformer architectures (Vaswani et al., 2017) in similar fashion to language and vision (Radford et al., 2018; Chen et al., 2020; Brown et al., 2020; Lu et al., 2022; Yan et al., 2021). Extensions have also been proposed in the context of sequential decision making for offline black-box optimization (Nguyen & Grover, 2022; Krishnamoorthy et al., 2022). A recent work by Emmons al. (2021) further shows that conditional BC achieve competitive performance even with simple but carefully designed MLP network. Earlier, similar ideas have also been proposed for online RL, where the policy trained via supervised learning techniques fit the data stored in the replay buffer (Schmidhuber, 2019; Srivastava et al., 2019; Ghosh et al., 2019).
Data Exploration for Offline RL Recent research efforts have also been made towards under- standing properties and limitations of datasets used for offline RL (Yarats al., 2022; Lambert et al., 2022; Guo et al., 2021), particularly focusing on exploration techniques during data collection. Both Yarats et al. (2022) and Lambert et al. (2022) collect datasets using task-agnostic exploration strategies (Laskin et al., 2021), relabel the rewards and train offline RL. algorithms on them. Yarats al. (2022) benchmark multiple offline RL algorithms on different tasks including transferring, whereas Lambert et al. (2022) focus on improving the exploration method.
3 PRELIMINARIES
We model our environment as a Markov decision process (MDP) (Bellman, 1957), which can be described by tuple .M AS,A. P. R, 7), where 8 is the state space, A is the action space, p(s1) is the distribution of the initial state, P(s++1|$t, ax) is the transition probability distribution, R(s,, ax) is the deterministic reward function, and 7 the discount factor. At each timestep the agent observes state E S and takes an action a₂ E A. This moves the agent to the next state 8141 P(-sst at) and provides the agent with a reward rt ((s),a)).
Offline RL. We are interested in learning a (near-)optimal policy from static offline dataset of trajectories collected by unknown policies, denoted as Tottine. We assume that these trajectories are i.i.d samples drawn from some unknown static distribution T. We use denote trajectory and use |r] to denote its length. Following Chen et al. (2021), the returm-toge (RTG) for a trajectory at timestep is defined as the sum of rewards starting from / until the end of the trajectory: gr = re. This means the initial RTG gi is equal to the total return of the trajectory r, =yln, Tt.
Decision Transformer (DT). DT (Chen et al., 2021) solves offline RL via sequence modeling. Specifically, DT employs transformer architecture that generates actions given sequence of historical states and RTGs. To do that, DT first transforms each trajectory in the dataset into a sequence of returns-to-go, states, and actions:
== (g1, $1, 01 92. $2 02 91-1 S/11: 9|11)
(I)
DT trains a policy that generates action a, at each timestep t conditioned on the history of RTGs It-K;t, states St-K.t. and actions a wherein K is the context length of the transformer. The learning objective a simple mean square error between the predicted actions and the ground truths:
min (D) ET-T E| (0g Te (:t Kit at- 1.
During evaluation, DT starts with an initial state 81 and a target RTG 91. At each step I, the agent generates an action a,, receives reward r, and observes the next state 8++1- DT updates its RTG Gt+1 = gt Tr and generates next action d:+1- This process is repeated until the end of the episode.
Reinforcement Learning via Supervised Learning (RvS). Emmons al. (2021) conduct thorough empirical study of conditional BC methods under the umbrella of Reinforcement Learning via Supervised Learning (RvS), and show that even simple models such multi-layer perceptrons (MLP) can perform well. With carefully chosen architecture and hyperparameters. they exhibit performance that matches or exceeds the performance of transformer-base models. There are two main differences between RvS and DT. First, RvS conditions on the average reward W/ into the future instead of the sum of future rewards:
Wi
TH ----1:
3

---

Under review as a conference paper at ICLR 2023
suboptimal data distribution expert distribution
original transformed f 6,
0.06
density
0.04
0.02
0.0
20
20
40
60
80
100
0.00
20
normalized return fr
40
80
100
nomalized return
Offline data distribution the expert dis- tribution.
(b) The original return distribution 7 and the trans- formed distribution T.
Figure I: The suboptimality of offline datasets (left) and the effeet of trajectory weighting on the return distribution (right). We illustrate on walker2 d-med- wwmeddreelaaae For weighting, we use B 20, x 0.01, f f90, where P90 is the 90-th percentile of the returns in the offline dataset.
where H/ is the maximum episode length. Intuitively, wi is RTG normalized by the number of remaining steps. Second, RvS employs a simple MLP architecture, which generates action a, at step t based on only the current state 8, and expected outcome wi. RvS minimizes a mean square error:
min Lavs (f) E,-T I E
(a+ To (s))), }.
At evaluation time, RvS performs a repeating process similarly to DT, except that the expected outcome now updated as w (g: rt)/ -)).
4 CONSERVATIVE BEHAVIORAL CLONING WITH TRAJECTORY WEIGHTING
A key challenge that behavioral cloning faces in an offline setting is the suboptimality of the dataset, which we can characterize via the distribution of trajectory returns. An ideal offline dataset consists of sufficiently many high-quality trajectories, which have returns matching those of dataset of expert demonstrations. For such an idealized scenario, offline RL reduces to a vanilla imitation learning problem. In practice, however, observe that the return distribution for typical dataset of offline trajectories spread over a wide range of returns and is highly non-uniform. Figure la illustrates the return distribution of the different from the expert distribution. Therefore, from a return perspective, the trajectories in the offline dataset can be of varying importance for learning. which leads to bias-variance trade-off. Further, for conditoondiio methods including conditional BC, is unclear how the policy will behave when conditioned on o.o.d. returns at test-time. We study mitigation techniques for both these challenges in the following sections.
dataset (Fu et al., 2020), which is significantly
4.1 CONTROLLING BIAS- VARIANCE TRADEOFF VIA TRAJECTORY WEIGHTING
To formalize our discussion, recall that r, denotes the return of trajectory and r sup, T- be the maximum expert return, which is assumed to be known in prior works on conditional BC (Chen et al., 2021; Emmons al., 2021). We know that the optimal offline data distribution, denoted by T is simply the distribution of demonstrations rolled out from the optimal policy. Typically, the offline trajectory distribution T will be biased w.r.t. T*, During learning, this leads a bias-variance tradeoff, wherein ideally want to learn our BC agent condition the expert returns, but is forced to minimize the empirical risk on a biased data distribution.
The core idea of our approach to transform T into a new distribution 7 that better estimates T More concretely, T should concentrate on high-return trajectories, which mitigates the bias. One naive strategy to simply filter out small fraction of high return trajectories from the offline dataset. However, since we expect the original dataset to contain very few high return trajectories. filtering trajectories will increase the variance for downstream BC. To balance the bias-variance trade-off, we propose weight the trajectories based on their returns. Let fr R - R., be the density function of r, where
T. We consider the transformed distribution 7 whose density function p7 is
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Weighted Trajectory Sampling
Input: offline dataset Tomlnne number of bins B, smoothing parameters X, Ai
Compute the returns: F,- Σl= rt, VT Jettine-
Group the trajectories into B equal-sized bins according to ry.
Sample bin b E [B] with probability (b) defined in Equation (6).
Sample trajectory r in bin b uniformly at random.
Output: 7
Table The normalized return on D4RL locomotion tasks of RvS and DT with trajectory weighting. We use +W shorthand for weighting. We #wins to denote the number of datasets where the variant outperforms the original model. The results are averaged over 10 seeds.
DT
DT+W
RvS
RyS-W
wa modlkka2dmmdinnn walkes2d-med-2 replay
71.3 3.9
70.4
73.3 1 5.7
54.5 7.7
53.1 12.
60.5 : 8.9
51.0 + 12.1
61.2 14.7
wa kalllkk-2ddmmm -d--dd-me--xpp-
99.8 1 21.3 213
108.2 0.8
102.2 :: 104.1
104.1 + 0.5
hoppez-medium
59.9 4.9
63.9 + 44
56.6 5.5
62.5 + 7.1
hepper>ed--roplay
56.4 20.1
76.9 5.9
87.7 1 9.7
92.4 1 6.1
heppez-med expert
95.1 11.3
108.8 0.9
108.4
haolfcheetahmmedium
42.5 0.6
41.6 1.7
16.2 1: 4.5
4.0 1: 5.4
hal wwi.fehhaahmmddrrpll...
34.5 4.2
36,9 2.2
-0.4
-0.8 1. 2.2
87.2 2.7
85.6 2.0
83.4 + 2.1
69.1 +: 3.7
# wins average
66.7
71.9
64.6
61.7
trajectory weight
(T)
(r(r.) [r(+)++X exp (
(5)
where X, E R_ are two hyperparameters. A larger value of leads a smaller value upweights the high-return trajectories. In contrast, a smaller value of gives more weights to high-return trajectories, while a larger value makes 7 closer to T. Our trajectory weighting motivated by a similar scheme proposed for model-based optimization (Kumar & Levine, 2020), where the authors use it to balance the bias and variance for gradient approximation for surrogates to black-box functions, and theoretically establish the optimality of the proposed distribution.
uniform T. whereas
4.1.1 IMPLEMENTATION DETAILS
In practice, the dataset Joftine only contains finite number of samples and the density function Py in equation (5) cannot be computed exactly. Following Kumar & Levine (2020), we sample from discretized approximation of T. We first group the trajectories in Toftiine into cqual-sized bins according to the return r,. To sample trajectory, we first sample bin index be {l..... B} and then uniformly sample a trajectory inside bin b. We use [b] to denote the size of bin b. Let / = 1/|b] Eee, Fr the average return of the trajectories in bin b, f* be the highest return in the dataset Joffiine and define /Tmm (b) |6/ [b//[Tomine).. As a discretized version of equation (5), the bins are weighted by their average returns with probability
Fromie (b) FTatin (D)+X
Pbin(b) O
exp
(6)
Algorithm summarizes the data sampling procedure when trajectory weighting is used. Figure 1b illustrates the impact of trajectory weighting on the return distribution of the med-replay dataset for the walker2d environment. We plot the histograms before and after transformation, where the density curves are estimated by kernel density estimators.
4.1.2 EMPIRICAL RESULTS
Dataset We evaluate the effectiveness of trajectory weighting on three locomotion tasks with dense rewards from the D4RL benchmark (Fu et al., 2020): hopper, wa malkerdd and hal fcheetah.
5

---

Under review as a conference paper at ICLR 2023
ker2d medium
willer2mmedium
300
80
20
20
20
heppormedum
hopse-medum-eg
126
120
120
330
300
300
Nutum
40
120
140
100
140
100
180
100
hattcheetaa--me
Taltchestat- www.aa--ediimteeee
halfcheetahh Seu---ree
320
120
320
1400
00
25
50
100
323
150
115
25
50
500
325
350
175
25
53
75
100
325
350
115
D7
01-W)
RvS
RvS+W
return offline data
return
Figure 2: Performance of RvS and DT when conditioning on different evaluation RTGs, We report the mean and standard deviation of 10 seeds.
For each task, we consider the v2 medi um, med-replay and med- med-expert rt. offline datasets. The medí um dataset contains 1M samples from a policy trained to approximately } 3 the performance of expert policy. The med-replat dataset uses the replay buffer of policy trained up to the performance of a medium um policy. The med-expert dataset contains 1M samples generated by medi um policy and 1M samples generated by expert policy.
Baselines We apply trajectory weighting to RvS (Emmons et al., 2021) and DT (Chen et al., 2021), I), two state-of-the-ans BC methods. We compare their performance when trained on the original distribution and on the transformed distribution induced by our trajectory weighting (denoted as +W).
Hyperparameters For all datasets, use B 20 and X 0.01, and we set the temperature parameter x to be the difference between the highest return and the 90-th percentile: P* - Poo. whose value varies across the datasets. At test time, set the evaluation RTG to be the expert return for each environment. The model architecture and the other hyperparamters are identical to what were used in the original paper. We provide a complete list of hyperparameters in Appendix B.2 and additional ablation experiments on À and λ in Appendix C.
Results Table shows the performance of RvS and DT and their variants. DT+W outperforms the original DT in 6/9 datasets, achieving an average improvement of 8%. The improvement is significant in low-quality datasets (med-replay) yy) which agrees with our analysis. Unlike in DT, trajectory weighting in RvS has varying effects, and the average performance of RyS+W is not better than that of RvS. To better understand this, we plot the achieved returns of RyS and DT when conditioning on different values of RTG. Figure 2 shows an interesting difference between behaviors of DT and RvS. DT insensitive to the conditioning RTG, and continues performing stably even when conditioning on out-of-
RTGs. In contrast, the performance of RvS highly correlates with the evaluation RTG, but degrades quickly after a certain threshold. The performance crash problem of RvS shadows the improvement made by trajectory weighting.
4.2 RELIABLE EVALUATION VIA CONSERVATISM
The results in Section 4.1.2 introduce another challenging problem for return-conditioned BC in offline RL: generalization to out-of-distribution (ood) returns. While strong generalization beyond the offline dataset remains an ongoing challenge for the offline RL community (Wang et al., 2020; Zanette, 2021; Foster et al., 2021), we require the policy be reliable and at least stay close to the data distribution to avoid catastrophic failure when conditioned on ood returns. In other words, we want the policy to be conservative. Figure 2 shows that DT enjoys self-conservatism, while RvS does

---

Under review as a conference paper at ICLR 2023
honpeemedumerrky
halftheee tommeddddd
Patuum
Eva
DT-Concat
offiine
Figure 3: Performance of DT when the state and RTG tokens are concatenated. We report the mean and standard deviation of 10 seeds.
Algorithm/ 2: ConserWeightive Behavioral Cloning (CWBC) for RyS
Input: dataset Tottines number iterations batch size S, regularization coefficient a, initial parameters 8g
for ileration = I,...,
Sample batch of trajectories (T(1), ...., (5)3 from Jomine using Algorithm
for every sampled trajectory 7(6) do Samplie noise as described in Section 4.2.1.
Compute noisy RTGs: of - 9r + + tt |r(1]
// loss and regularizer defined in Equation (4) and (7)
Perform gradient update of 0 by minimizing the regularized empirical risk
CR.s( a C (0).
Output: T0
not. We hypothesize that the conservative behavior of DT comes from the transformer architecture. As the policy conditions on a sequence of both state tokens and RTG tokens to predict next action, the attention layers can choose to ignore the ood RTG tokens while still obtaining good prediction loss. To test this hypothesis, we experiment with a slightly modified version DT, where we concatenate the state and RTG each timestep instead of treating them separate tokens. By doing this, the model cannot ignore the RTG information in the sequence. We call this version D'T-Concat. Figure shows that the performance of DT-Concat is strongly correlated with the conditioning RTG, and degrades quickly when the target return is out-of
This result confirms our hypothesis.
However, conservatism does not have come from the architecture, but can also emerge from proper objective function, as commonly done in conservative value-based methods (Kumar et al., 2020; Fujimoto Gu, 2021). this section, we propose a novel conservative regularization for BC that explicitly encourages the policy to stay close to the data distribution. The intuition is to enforce predicted actions when conditioning on large ood returns to stay close to the in-distribution actions. To do that, for trajectory with high return, we inject positive random noise e & to its RTGs, and penalize the le distance between the predicted action and the ground truth. Specifically, to guarantee we generate large ood returns, choose noise distribution E such that the perturbed initial RTG 9a + & is at least f* the highest return in the dataset. The next subsections instantiate the conservative regularizer in the context of RvS, and empirically evaluate its performance.
4.2.1 IMPLEMENTATION DETAILS
We apply conservative regularization to trajectories whose returns are above Fg.. the q-th percentile of returns in the dataset. This makes sure that when conditioned on ood returns, the policy behaves similarly to high-return trajectories and not random trajectory in the dataset. We sample scalar noise &, and offset the RTG of T at every timestep by e: gi gt + ee I,..., |T|. resulting in the conservative regularizer:
CR.S (0) Er-T, eve, [_,,2,, म Eii (a. - wf)
where w/ (g.+e)/(H -- +) (cf. Equation (3)) is the noisy average RTG at timestep t. We observe that using the 95-th percentile of Pas generally works well across different environments and datasets.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of the normalized return on the D4RL locomotion benchmark. For BC and TD3+BC, we get the numbers from (Emmons et al., 2021). For IQL., we get the numbers from (Kostrikov et al., 2021b). For TTO, we get the numbers from (Janner et al., 2021). The results are averaged over 10 seeds.
RvS
ReS+W
RYS+C:
RoS+WaC:
DT
BC
TD3+BC
CQL
IQL.
TIO
velkerrddimddin see like rarddndddreee my
1.77
515 1.51777
71.31
73.6
71.5 3.3
75.3
53.7
82.9
78.3
51.3
510 12.1
6122 1477
120:1135
72.8
53:4::12.22
26.0
81.8
86.1
73.0
78:4
12.8
102.2
10L1 0.0.
102.1 10.2
107.6 0.05
90.8
107.5
110.1
109.5
109.6
10.8
hogper-nedi:mm
56.6
8.1
6.3
62.0 1.
30.0 4.0
52.9
59.3
61.6
66.3
67.4 11.3
hogge n-neec- negi
9.77
6.1
91.5 3.5
87.7 +
36.4 20.1
18.1
00.9
07.8
94.7
90.4
12.6
hogper neee-eett
108.8 0.0
108.4 a 1.8
101.0 1: 13.4
110.0 1 2.8
96.4 11.3
52.5
08.0
102.0
91.5
106.0 1: 1.1
hall rcheetat:mediinm he Feheet wh-ned----eee repi by
16.2
4.0 5.4
10.7 1: 1.0
42.2 1: 0.7
42.5 1 0.6
42.6
68.3
49.1
47.4
41.0
2.4427
-08 2.2
36.8 + 1.5
= 0.8
34.5 =2
36.6
44.6
44.2
hell www.eeattneed-eeeeee
2.4
69.1 1:7
91.2 1: 1.0
91.1
87.2 2:7.
55.2
00.7
85.8
80.7
40.8 2 8.7
average
616
61.7
76.5
75.3
77.0
We use the noise distribution & = Uniform[4:: uz], where the lower bound 1, = F* " 5o that the perturbed initial RTG of ry + E is no less than f*, and the upper bound u, = f -r, + v 22 so that the standard deviation of E, is equal σ. We emphasize our conservative regularizer is distinct from the other conservative components proposed for the value-based offline RL methods. While those usually try to regularize the value function estimation to prevent extrapolation error (Fujimoto et al., 2019), we perturb the returns to generate ood conditioning and regularize the predicted actions.
When the conservative regularizer is used, the final objective for training RvS is LRvs + a CRvs(O), in which a is the regularization coefficient. When trajectory reweighting is used in conjunction with the conservative regularizer, we obtain ConserWeightive Behavioral Cloning (CWBC), which combines the best of both components. We provide a pseudo code for CWBC in Algorithm 2.
4.2.2 EMPIRICAL. RESULTS
Dataset We evaluate the effectiveness the conservative regularizer, as well as the performance of CWBC as a whole on the D4RL datasets (Fu et al., 2020) for the gym locomotion tasks.
Baselines We apply the conservative regularizer, which we denote as +C, both RvS and RyS+W. In addition, report the performance of three value- methods: TD3+BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), and IQL (Kostrikov et al., 2021b) as a reference.
Hyperparameters We apply our conservatism regularization to trajectories whose returns are above the q = 95-th percentile return in the dataset, and perturb their RTGs as described in Section 4.2.1. We use regularization coefficient of a 1. The evaluation protocol is similar to Section 4.2.
Results Table 2 reports the performance of different methods we consider. Our proposed framework CWBC with all components enabled (RvS+Wa W+C) significantly outperforms the original RvS 9/9 datasets, with an average improvement of 18 18% over RvS. RvS+W+C also the best performing BC method in the table, and is competitive with the value-based methods. Conservative regularization consistently improves the results for both RvS and RvS+W. Trajectory weighting on its own can have varying effects on performance, but is synergistic when combined with RvS+C leading to our best performing model in RvS+W+C.
To better understand the impact of each component, we plot the achieved returns of RvS and other variants when conditioning on different values of conditioned RTG. Figure 4 shows that RvS generalizes poorly to -iitt--iistiittiionn RTGs, which leads significant performance drop when the evaluation RTG is larger than the best return in the dataset. Figure 4 illustrates the significant importance of encouraging conservatism for RvS, where RvS+C has much more stable performance, even when the evaluation RTG is 2x the expert return. By explicitly asking the model to stay close to the data distribution, we achieve more reliable out-of-distributte performance crash problem. This leads to absolute performance improvement of RvS+C in Table 2. CWBC combines the best of both weighting and conservatism, which enjoys good performance when conditioning on high RTG values, well as better robustness to large, duit----iitiiuution RTGs.
performance, and avoid the
8

---

Under review as a conference paper at ICLR 2023
ker2d modmmmedum
waller25-meumm
www.er2d-medumreeee
1.03
Netum
46-
124
teppa-medium
120
hopper redd Artheep
123
Tappae- med.umeddmmm-m- sogeet
300
100
180
1u5um
338
125
200
75
100
125
50
75
170
230
balKheeaah-mede medium
taftheelah meduum
ha meduum-epprt
120
120
120
30
75
100
155
100
125
110
250
115
25
220
325
100
175
280
Rv5
RvS+C
RvS+W+C
offline
Figure 4: Performance of RvS and its variants when conditioning on different evaluation RTGs. We report the mean and standard deviation of 10 seeds.
In addition to the main results, we include ablations for different choices conservative percentile q and regularization coefficient in Appendix C. Finally, also evaluate CWBC in two more benchmarks: Atari games (Bellemare et al., 2013) and the D4RL, Antmaze datasets. We present these results in Appendix D and E respectively.
CONCLUSION
We proposed ConserWeightive Behavioral Cloning (CWBC), new framework that extends BC for offline RL with two novel components: trajectory weighting and conservative regularization. Trajectory weighting balances the bias-variance tradeoff that arises in learning from a suboptimal dataset, improving the performance of both DT and RvS. Next, we showed that while DT is self- conservative due its attention architecture, we can recover this desired behavior even for RvS using our proposed conservative regularizer. Confirmed by the experiments, CWBC significantly improves the performance and stability of RvS.
While we made good progress for BC, advanced value-based methods such as CQL and IQL are still ahead and we believe further understanding of the tradeoffs in both kinds of approaches is important future work. Another promising direction from data perspective is how to combine datasets from multiple environments to obtain diverse, high-quality data. Recent works have shown promising results in this direction (Reed et al., 2022). Last but not least, while CWBC significantly improves the performance and reliability RvS, is not able to extrapolate beyond the offline dataset. How to obtain extrapolation, whether is possible, is still open question, and poses persistent research opportunity for not only CWBC but the whole offline RI. community.
REPRODUCIBILITY STATEMENT
We present the practical implementation of our framework in Section 4.1.1 and Section 4. We include the implementation details of our paper in Appendix we use, the open sourced code we base on, and the list of hyperparameters we use to reproduce our results. Finally, we submitted the source code in the supplementary material.
which contains information about the datasets

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Michael Bain and Claude Sammut. A framework for behavioural clonnin In Machine Intelligence 15, pp. 103-129, 1995.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Richard Bellman. markovian decision process. Indiana Univ. Math. J., 1957.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR, 2020.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 10751, 2021.
Dylan Foster, Akshay Krishnamurthy, David Simchi-Levi, and Yunzong Xu. Offline reinforcement learning: Fundamental barriers for value function approximation. arXiv preprinl arX1v:2111.10919 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learming operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06008. 2019.
Wenshuo Guo, Kumar Krishna Agrawal, Aditya Grover, Vidya Muthukumar, and Ashwin Pananjady, Learning from exploring demonstrator: Optimal reward estimation for bandits. arXiv preprint (0w.22/06.14666, 2021.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off- off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 1r0iv:1907.0000 0046000466 2019.
10

---

Under review as a conference paper at ICLR 2023
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169) 2021b.
Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Generative pretraining for black-box optmiiattion arXiv preprint 1rXiv 2022.
Aviral Kumar and Sergey Levine. Model inversion networks for model- optimization. Advances in Neural Information Processing Systems, 33:5126-5137, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-lcarning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems. 33:1179-1191, 2020.
Nathan Lambert, Markus Wulfmeier, William Whitney Arunkumar Byravan, Michael Bloesch, Vibhavari Dasagi, Tim Hertweck, and Martin Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201. 122200111611 1861, 2022.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Iearning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005. 0/005, 2020.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et Human-level control through deep reinforcement learning. nature, 51 518(7540):529-55333 2015.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprini arXiv: /9/i:02022.0 2019.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-awar meta learning sequence modeling. arXiv preprint arXiv:2207.041 04779, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06115. 2022.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-juss map them to actions. arXiv preprint arXiv: 1912.02855 2019.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: 1912:1112..2877, 2019.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ruosong Wang, Dean P Foster, and Sham M Kakade. What are the statistical limits of offline rl with linear function approximation? arXiv preprini arXiv:2010. 102000011899 1895, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: /911v:/911.113330 2019.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv: 11X::2204411115, 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't 't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint orXiv:2201.13422 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neurul Information Processing Systems, 33:14122 13:14129: 13:1419--4442 2020.
Andrea Zanette. Exponential lower bounds for batch reinforcement learning: Batch rl can be exponentially harder than online rl. In International Conference on Machine Learning, pp. 12287- 12297. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A
LIST OF SYMBOLS
Table 3: Important symbols used in this paper.
Symbol
Meaning
Definition
S
state space action space
A
T
trajectory
|
trajectory length
T
distribution of trajectories
J afttine
offline dataset
policy
policy parameters
state at timestep t
action at timestep reward timestep trajectory return
re
Σ
return- to-n--to-goo-o at timestep
rl
H
maximum trajectory length
we
average retur-to--o timestep empirical risk of RvS conservative regularizer for Rv$ probability density of trajectory probability density of trajectory T ~ 7 index of bin of trajectories in the offline dataset size of bin
Ge / - 1)
Cxs
Equation (4)
CRUS
Equation (7)
(T(T)
T
(T)
Equation (5)
b
Jamt (b)
proportion trajectories probability that bin is sampled average return of trajectories in bin b highest return offline dataset q-th percentile of the returns in the offline datasct
b / Jaminee
Poi((b)
Equation (6)
Pg
B IMPLEMENTATION DETAILS
B.1 DATASETS AND SOURCE CODE
We train and evaluate our models on the D4RL (Fu al., 2020) and Atari (Agarwal et al., 2020) benchmarks, which are available at https://cithub com com/rail-berreeeee..eeooa /d4rl and https: h/tps/eeree.eeeeeeeeeeeeeeeceeee google/ weall//ttoos/daaaeettetee /t////nnnrreee replay. respectively. Our code- wwwpp/ettttt/nnneeeeeeeee
base is largely based on the RvS (Emmons et al., 2021) official implementation at https: //gi //github. com/ httt://cootttos//tssssse and DT (Chen et al., 2021) official implementation github..ttte..g http:///igcccc/////////sssttoooo cew.eeeciiiiittttataaor.eeete
13

---

Under review as a conference paper at ICLR 2023
B.2
DEFAULT HYPERPARAMETERS
Table 4: Hyperparameters used for locomotion experiments.
Hyperparameter
Value
Context length K (DT)
Number attention heads (DT)
Model
Hidden layers
2 for RvS, 3 for DT
Hidden dimension
1024 for RvS, 128 for DT ReLU
Activation function
Dropout
0.0 RvS, 0.1 DT
Conservative percentile
Conservative regularizer
Noise standard deviation a Regularization coefficient
1000
1.0
bins B
20
Trajectory weighting
Smoothing parameter À
0.01
Smoothing parameter
ft Puo
Batch size
64
Learning
le for RvS, le for DT
Optimization
Weight decay
Training iterations
100000
Evaluation
Target return
1 Expert return
Table 5: Hyperparameters used for Atari experiments.
Hyperparameter
Value
Encoder channels
32, 32, 64
Encoder filter sizes
8x8844 x x
Encoder strides
4, 1
Model
Hidden layers
4
Hidden dimension
1024
Activation function
ReLU
Dropout
0.1
Conservative percentile q
95
50 for Breakout, Pong
Conservative regularization
Noise
500 for Qbert. Seaquest
Conservative weight
0.1
# bins B
20
Trajectory weighting
X
0.1
K
Feo
Batch size
128
Learning rate
6e
Optimization
Weight decay
le - 4
Training itcrations
25000
90 for Breakout (1 x max in dataset)
2500 for Qbert (5x max in dataset)
Evaluation
Target return
20 for Pong (1x max in dataset)
1450 for Seaquest (5x max dataset)
14

---

Under review as a conference paper at ICLR 2023
C ABLATION ANALYSIS
In this section, we investigate the impact of each of those hyperparameters CWBC to give insights on what values work well in practice. We use the walker2d environment and the three related datasets for illustration. In all the experiments, when we vary one hyperparameter, the other hyperparameters are kept as Table
C.I TRAJECTORY WEIGHTING: SMOOTHING PARAMETERS À AND K
Two hyperparameters &. and À in Equation (6) affect the probability a bin index is sampled:
Futtieee (b)
Pain CC
Frattire 1b)+}
exp
In practice, have observed that the performance of CWBC is considerably robust to wide range of values of x and A.
The impact of K The smoothing parameter & controls how we weight the trajectories based on their relative returns. Intuitively, smaller gives more weights to high-return bins (and thus their trajectories), and larger и makes the transformed distribution more uniform. We illustrate the effect of к the transformed distribution and the performance of CWBC in Figure 5. As in Section 4.1.2, we set A to be the difference between the empirical highest return f and the 2-th percentile return in fr fz. and vary the values of This allows the actual value of to adapt to
the dataset: different datasets.
Figure shows the results. The top row plots the distributions of returns before and after trajectory weighting for varying values of s. We tested four values E (99, 90, 50, 0j. which correspond to four increasing values of к. We mark the actual values of in each dataset in the top tow!. For each dataset, we can see the transformed distribution using small и (orange) highly concentrates on high returns; as к increases, the density for low returns increases and the distribution becomes more and more uniform. The bottom row plots the corresponding performance of CWBC with different choices of k. We select RvS+C as our baseline model, which does not have trajectory weighting but has the conservative regularization enabled. We can see that relatively small values of к (based on F99 P. and F50) perform well on all the three datasets, whereas large values (based on fo) hurt the performance for the med-expert expert dataset, and even underperform the bascline RvS+C.
waliterdd medium reply
walkerdd medium
waller2d midilum expert
3.05
orig nal
0.12
2=62
orginal
0.5
2 0.5
mag.mal
0.10
2-5.5
2-1
$2.5
2515
26.3
21.2
922
0.3
6-100.
0.02
0.01
3.00
0.0
100
0
20
40
60
80
100
0
20
40
80
100
Normalized return C.
120
80
02
40
60
60
50-
20
IUG
BIG
Esal RTG
RvS+C
P
K K=
K
Figure 5: The influence of x on the transformed distribution (top) and on the performance of CWBC (bottom). The legend in each panel (top) shows the absolute values of for easier comparison. In the bottom row, we also plot the results of RvS+C (no trajectory weighting) as a baseline.
1np defined he the lowest return in the dataset: fo minreT Tr-
15

---

Under review as a conference paper at ICLR 2023
The impact of A
To better understand the role of X, we rewrite Equation (6)
T1
T2
Pbin (b) f (b) exp | -
{1/
Jame (b) A)}
Clearly, only T2 depends on A. When O, T2 canceled out and the above equation reduces to
Pain (b) x exp ( -----"))
which purely depends the relative return. As À increases, T2 is less sensitive to fram.. (b), and finally becomes the same for every e [B] as X - oo. In that scenario, Pom(() only depends on T1, which is the original frequency fm. (b) weighted by the relative return.
The top row of Figure plots the distributions of returns before and after trajectory weighting with different values of A. When À 0, the distributions concentrate on high returns. As X increases, the distributions are more correlated with the original one, but still weights more on the high-return region compared the original distribution due the exponential term in T1. The bottom of Figure plots the actual performance of CWBC À varies. All values of À produce similar results, which are consistently better than or comparable to training on the original datset (RvS+C).
wokker26-meddm--eptty
8.05
0.125
0.04
0.4
2.03
0.075
0.02
0.050
0.01
0.00
0.000
40
60
80
20
80
80
100
20
40
86
Kremelowed return
lnmmeltted neturr
Normalees lted refurn
walker2d www.mmmmodmmmmtoo replay
walker2dmedium
walter25mediamm expert
50
100
60-
40
50
40
50
100
150
200
50
1000
150
200
50
100
150
200
RyS+C
A- 0.0
0.0.11
1.0
A- 100.0
Figure 6: The influence of À on the transformed distribution (top) and on the performance of CWBC (bottom). We plot the result RvS+C as the baseline.
C.2 CONSERVATIVE REGULARIZATION PERCENTILE
We only apply the conservative regularization to trajectories whose return is above the q-th percentile of the returns the dataset. Intuitively, a larger applies the regularization fewer trajectories. We test four values for q: 0, 50, 95, and 99. For q = 0. our regularization applies to all the trajectories in the dataset. Figure demonstrates the impact of q on the performance of CWBC. q = 95 and 99 perform well on all the three datasets, while = 50 and O lead to poor results for the med-rep redlay dataset. This because, when the regularization applies to trajectories low returns, the regularizer will force the policy conditioned on out-of -iit--d-iittiittiion RTGs stay close to the actions from low return trajectories. Since the med-reol ay dataset contains many low return trajectories (see Figure 5), such regularization results in poor performance. In contrast, medi um and med-expert datasets contain a much larger portion of high return trajectories, and they are less sensitive the choice of q.
16

---

Under review as a conference paper at ICLR 2023
walkerrd medium-restyy
walke/2dmed.mm
meller22mmslumespeer
80
100
70
80
9atumm
20-
50-
50
150
150
200
50
150
150
200
50
100
150
200
RTG
Eval
Figure 7: Performance of CWBC with different values of the conservative percentile q.
C.3 REGULARIZATION COFFFICIENT
The hyperparameter a controls the weight of the conservative regularization in the final objective function of CWBC LRV in Figure 8. Not using any regularization (a 0) suffers from the performance crash problem, while overly aggressive regularization (a 10) also hurts the performance. CWBC is robust to the other non-extreme values of a
CRUS- We show the performance of CWBC with different values of
reslay
expert
120
80
80
60
80
40
20-
20
RTG
RTG
Eval
a=
0=1.1
a=5 0.5
d 1.0
-= 10.0
Figure 8: Performance of CWBC with different values of a.
D ADDITIONAL RESULTS ON ATARI GAMES
In addition D4RL,, consider 4 games from the Atari benchmark (Bellemare al., 2013): Breakout, Qbert, Pong, and Seaquest. Similar to (Chen et al., 2021), for each game, we train our method on 500000 transitions sampled from the DQN-replay dataset, which consists of 50 million transitions of an online DQN agent (Mnih et al., 2015). Due to the varying performance of the DQN agent different games, the quality of the datasets also varies. While Breakout and Pong datasets are high-quality with many expert transitions, Qbert and Seaquest datasets are highly suboptimal.
Hyperparameters For trajectory weighting, we use B 20 bins, 0.1, and к f As. We apply conservative regularization with coefficient a 0.1 to trajectories whose returns are above 195- The standard deviation of the noise distribution varies across datasets, as each different games have very different return ranges. During evaluation, we set the target return to 5 x F* for Qbert and Seaquest, and to 1 f* for Breakout and Pong.
RVS
RuS WW
RiSiC
RuSeWiC
maa roturn in offline data
Figure 9: Performance of RvS and its variants on Atari games when conditioning on different evaluation RTGs.
17

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison of the normalized return on Atari games. The results are averaged over seeds. We include the results of DT, CQL, and BC from (Chen et al., 2021) for reference.
RvS
RyS+W
RyS+C
RyS+W+C
DT
CQL
BC
Breakout
126.9 1 38.0
120.1 i 28.43
163.0 50.4
237.3 1 82.1
267.5 97.5
211.1
138.9 1 61.7
Qbert
-0.4
0.0 =
12.4 + 8.6
19.1 +
15.4 11.4
104.2
17.3
14.7
Pong
73.7 1
90.7 1 6.4
84.1 + 9.6
90.4 1: 1.9
106.1 8.1
111.9
85.2 1 20.0
Seaquest
0.2 0.2
0.1 0.1
1.6 +: 0.2
1.4 0.3
2.5 1 0.4
1.7
2.1 0.3
wins
average
50.6
52.7
62.3
87.1
97.9
107.2
60.9
Results Table 6 summarizes the performance of RvS and its variants. CWBC (RvS+W+C) is the best method, outperforming the original RvS by 72% on average. Figure 9 clearly shows the effectiveness of the conservative regularization (+C). In two low-quality datasets Qbert and Seaquest, the performance of RvS degrades quickly when conditioning on out-of-distribution RTGs. By regularizing the policy stay close the data distribution, we achieve a much more stable performance. The trajectory weighting component (+W) alone has varying effects on performance because of the performance crash problem, but achieves state-of-the thrtr-rrttttttttt when used in conjunction with conservative regularization.
is also worth noting that in both Qbert and Seaquest, CWBC achieves returns that are much higher than the best return in the offline dataset. This shows that while conservatism encourages the policy to stay close to the data distribution, does not prohibit extrapolation. There is always trade-off between optimizing the original supervised objective (which presumably allows extrapolation) and the conservative objective. This is very similar to other conservative regularizations used in value-based such CQL or TD3+BC, where there is a trade-off between learning the value function and staying close to the data distribution.
E ADDITIONAL RESULTS ON D4RL ANTMAZE
Our proposed conservative regularization is especially important in dense reward environments such gym locomotion tasks Atari games, where choosing the target return during evaluation is difficult problem. On the other hand, trajectory weighting is generally useful whenever the offline dataset contains both low-return and high-return trajectories. In this section, we consider Antmaze (Fu et al., 2020), a sparse reward environment in the D4RL benchmark to evaluate the generality of CWBC. Antmaze is navigation domain in which the task is to control complex 8-DoF "Ant" quadruped robot to reach a goal location. We consider maze layouts: uma ze, med i um, and large, and 3 dataset flavors: vo, diverse, and play. We use the same hyperparameters as mentioned in B.2.
Table 7: Comparison of the success rate the Antmaze environment. The results averaged over 3 seeds. We include the results of DT,, CQL, and BC from (Emmons et al., 2021) for reference.
RvS
RyS+W
RvS+C
RvS+W+C
DT
CQL
BC
umaze-v0
54.0 13.56
65.0 18.03
58.0 t 8.72
65.0 t 12.85
65.6
44.8
54.6
umaze-diverse
55.0 15.65
46.0 16.85
50.0 + 10.95
42.0 + 7.48
51.2
23.4
45.6
medium-play
0.0 100000 0.0
26.0 12.0
0.0 0.0
25.0 = 13.6
1.0
0.0
0.0
mediur-diverse
1.0 3.0
24.0 + 15.62
1.0 + 3.0
23.0 : 11.0
0.6
0.0
0.0
large-play
0.0 0.0+0.00
4.0 4.9
0.0 0.0
5.0 6.71
0.0
0.0
0.0
large-diverse
0.0 0.0
10.0 + 10.0
0.0 + 0.0
17.0 + 11.87
0.2
0.0
0.0
wins average
1
18.3
29.2
18.2
29.5
11.4
16.7
Results Table summarizes the results. As expected, the conservative regularization is not important in these tasks, as the target return is either 0 (fail) or (success). However, the trajectory weighting significantly boosts performance, resulting in average of 60% improvement over the original RvS.
18

---

Under review as a conference paper at ICLR 2023
F TRAJECTORY WEIGHTING VERSUS HARD FILTERING
An alternative to trajectory weighting hard filtering (+F), where we train the model on only top 10% trajectories with the highest returns. Filtering can be considered hard weighting mechanism, wherein the transformed distribution only has support over trajectories with returns above a certain threshold.
F.1 HARD FILTERING FOR RvS
When using hard filtering for RvS, we also consider combining with the conservative regularization. Table 8 and Figure 10 compare the performance of trajectory weighting and hard filtering when applied RvS. While RvS+F+C also gains notable improvements it lags behind RvS+W+C and seems to erode the benefits of conservatism alone in RvS+C. This agrees with analysis in Section 4.1. While hard filtering achieves the same effect of reducing bias, completely removes the low-return trajectories, resulting in highly increased variance. Our trajectory weighting upweights the good trajectories but aims to stay close to the original data distribution, balancing this bias-variance tradeoff. This is clearly shown in Figure 10, where RyS+W+C has much smaller variance when conditioning on large RTGs.
Table 8: Comparison of trajectory weighting (+W) and hard filtering (+F) on D4RL locomotion benchmarks. The results are averaged over 10 seeds.
RvS
RvS+W
RvS+C
RyS+W+C
RyS+F
RyS+F+C
walker2dmeedimm
73.3 5.7
54.5 7.7
71.3 = 4.9
73.6 1 5.4
60.9 : 4.9
68.2 :: 7.1
lelkkerrddnee--eepp...
$1. 12.1
61.2 14.7
62.0 13.5
72.8 + 7.5
47.1 = 7.7
53.9 + 11.0
walker2d-ned-exper
102.2
101.1 0.5
102.1 10.2
107.6 0.5
101.7 3.3
105.4 + 0.6
hopper-medium
5.5
62.5 = 7.1
61.0 = 5.3
62.9 + 3.6
62.4 :: 5.0
65.7 + 6.4
hopper-med-replay
87.7
92.4 6.1
91.5 = 3.5
87.7 + 4.2
91.2 : 5.3
92.1 : 2.9
hopper-med-expert
108.8 0.9
108.4 = 1.8
101.0 + 13.4
110.0 + 2.8
97.5 = 15.0
106.8 3.5
16.2
1:
0.7
2.5
halfcchettah-mee--eee
-04 2.7
2.2
36.8 1.5
10.1 0.8
3.5
35.7 2.8
expeet
83.1 2.1
69.1 3.7
91.2 = 1.0
91.1 + 2.0
16.0 = 1.5
83.2 4: 5.0
# wins
9
5
average
64.6
61.7
73.1
76.5
56.5
71.8
Nulkor2d
Welker 2e cxpert
25
15
106
125
175
200
25
24
75
100
125
150
115
200
25
53
15
106
125
124
12
204
Sxrpetmattummetee
120
111
22
halthera--ree mererrtt---euiummmmm
s4Pchest' medum replay
haltchatahhhme mww.umecmmmmmmmmmmmmmmmm a0sssar aspert
122
122
20
110
300
125
320
135
Evoi
EVN RTG
EU0
BvS+WW
ReS-W-C
RvS+F
RvS4
return oiflinn datla
return
Figure 10: Comparison of trajectory weighting and hard filtering.
19

---

Under review as a conference paper at ICLR 2023
F.2
HARD FILTERING FOR UNCONDITIONAL BC
Hard filtering can also be applied to ordinary BC. This is equivalent to Filtered BC in (Emmons et al., 2021). Table 9 compares Filtered BC and CWBC. CWBC performs comparably well in medi um and med-expert rt datasets, and outperforms Filtered BC significantly with an average improvement of 12% in med-replay datasets. We believe that low-quality datasets, even when we filter out 90% percent of the data, the quality of the remaining trajectories is still very diverse that simple imitation learning is not good enough. CWBC able learn from such diverse data, and by conditioning on expert return at test time, we can recover an efficient policy.
Table 9; The normalized return D4RL for Filtered BC, RvS, and CWBC. For Filtered BC, we numbers from (Emmons al., 2021).
Filtered
RvS
RvS+W+C
walker2d-medium hopper-medium
75.0
73.3 + 5.7
73.6 + 5.4
56.9
56.6 + 5.5
62.9 + 3.6
halffchettah-medi.mm
42.5
16.2 + 4.5
42.2 + 0.7
medium averae
58.1
48.7
59.6
relply
62.5
54.0 + 12.1
72.8 + 7.5
hopper-med-zreplay
75.9
87.7 + 9.7
87.7 5.2
40.6
-0.4 2.7
40.4 + 0.8
med-replay
59.7
47.1
67.0
ex.eer
109.0
102.2 + 2.3
107.6 + 0.5
hopper-med-expert
110.9
108.8 + 0.9
110.0 = 2.8
halfcceettbb-mdd-xxpett
92.94
83.4 2.1
91.1 + 2.0
med-expert average
104.3
98.1
102.9
average
74.0
64.6
76.5
20

---

Under review as a conference paper at ICLR 2023
G BIAS- TRADEOFF ANALYSIS
We formalize our discussion on the bias-variance tradeoff when learning from a suboptimal distri- bution mentioned in Section 4.1. The objective functions for training DT (2) and RvS (4) can be rewritten as:
min pp (0) = E, [T(T, To)]
(8)
= E Er-po (r), (D(T,TO))
(9)
In which, PD(r) is the data distribution over trajectory returns, Tr is uniform distribution over the set of trajectories whose return is r, and [ me)) is the supervised loss function with respect to the sampled trajectory т. For DT, D(T; TT)
(o+- TT Kith KttKit (t) " and for RvS, (TT, m)) ir Σ (a. (0( wi)) Equation (9) is equivalent to first sampling return r, then sampling a trajectory t whose return is r, and calculating the loss on т. Ideally, we want to train the model from optimal return distribution p"(r), which is centered around the expert return r*:
min Lx- (0) = (x)pp(()) [T,, mo)].
(10)
In practice. we only have access the suboptimal return distribution (r()), which leads to biased training objective with respect to p" (r). While the dataset is fixed, we can transform the data distribution Pr(r) (r) that better estimates the ideal distribution p" (r). The objective function with respect to g is:
min (0) = Br [D(T, mD) 0
(11)
q(r) [PD(F)
D(r, TO)
(12)
In the extreme case, g(r) 1[r r* which means only train the policy trajectories whose return matches the expert return r". However, since offline datasets often contain very few expert trajectories, this y leads to very high-variance training objective. An optimal distribution q should lead to a training objective that balances the bias-variance tradeoff. We quantify this by measuring the tg of the difference between the gradient of C. and the gradient of the optimal objective function Cp. (0). Analogous to Kumar & Levine (2020), we can prove that for some constants C₁, C22, C3. with high confidence:
1 Nr
de(gllpD) C3-DrV(P .q) (13) [D]
(l) VoL. (0)11] & C1 C.Brrgg())
+ C2
In which, N, is the number of trajectories in dataset D whose return is r, d2 is the exponentiated Renyi divergence, and Dry is the total variation divergence. The right hand side of inequality (13) shows that an optimal distribution q should be close the data distribution pD reduce variance, while approximating well p" to reduce bias. As shown in Kumar & Levine (2020), q(r)x N. exp(( (e-r'l) minimizes this bound, which inspires our trajectory weighting. N.*K
21


Reviews:
[{"review_id": "rOpb-3jbvrE", "reviewer": "Reviewer_VCrN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two techniques, trajectory weighting and conservative regularization, to overcome two challenges, the bias-variance tradeoff and out-of-distribution return condition, respectively. Empirical study is conducted on top of two conditional BC offline RL algorithms, Reinforcement Learning via Supervised Learning (RvS) and Decision Transformer (DT). ", "summary_of_the_review": "Overall, the depth of this contribution and novelty may not quite yet reach the high bar set for ICLR. The improvement is significant while the improved performance still underperforms the SOTA offline RL algorithms. Only D4RL locomotion tasks are experimented, and adding experiments on more challenging tasks would make the claims more convincing. ", "strength_and_weaknesses": "Strengths:\n1. The paper is easy to follow and well structured. \n2. Experiment results support the authors claims. \n\nWeaknesses:\n1. Lack of novelty. The trajectory weighting technique is originated from Kumar & Levine, 2020. The solution for out-of-distribution (ood) returns is naïve and simple by adding positive random noise. The conditional BC offline RL algorithms are prior works, RvS and DT. \n2. The contributions are not very strong in the current state of the paper. The paper develops two techniques specifically for DT and RvS, which are limited in contributions. \n3. Improvement is not consistent via trajectory weighting and the final performance still underperforms SOTA methods. \n4. Experiments are not sufficient. The authors validate their techniques on D4RL locomotion tasks, which are known to be easy tasks. Better add more experiments such as AntMaze, Adroit or Kitchen to make the claims more convincing. \n\nAviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126–5137, 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VXCar7M46v", "reviewer": "Reviewer_rbiQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper identifies the skewed distribution of returns in the offline dataset as a challenge to return-conditioned methods, and proposes to weight the trajectories in the dataset based on their returns. \n", "summary_of_the_review": "I recommend a weak reject, because it is unclear why we should be interested in return-conditioned methods, especially when they face their own sets of unique challenges, and when enhanced with proposals such as the ones in this paper, still under-perform value-based methods. What are their unique selling points?", "strength_and_weaknesses": "Strengths\n\n1. The paper is relatively well-written and easy to follow.\n\n2. The proposed idea is intuitive, whose benefits are demonstrated on two different classes of return-conditioned methods (DT and RvS).\n\nWeaknesses\n\n1. The paper makes repeated reference to a \"bias-variance trade-off\" to motivate their methods. AFAIK, what this trade-off means exactly, and a theoretical or rigorous analyses as to why their method leads to a more favorable trade-off are missing completely.\n\n2. Skewed distribution of returns is a well-known problem in RL, and common technique to tackle this problem is to ensure each batch consists of equal amount of high return and low return episodes (so equivalently, use two bins only). I would have liked to see a comparison to such a simple baseline.\n\n3. My biggest concern is the motivation behind studying return-condition method? Because AFAIK, they under-performs value-based methods, especially on tasks that require stitching (such as antmaze navigation and notshown in the paper). Even in table 2, the performance of return-conditioned methods are under-whelming. I am not saying that the paper should not be accepted because it does not out-perform state-of-the-art, but there should be a reason as to why the class of return-conditioned method is interesting relative to value-based methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K0UyJ2rRHZg", "reviewer": "Reviewer_uS7R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the offline RL problem and proposed two techniques (trajectory weighting, and reward perturbation) to improve Decision Transformer (DT) and Reinforcement Learning via Supervised Learning (RvS).\nSpecifically, trajectory weighting transforms the trajectory distribution of the original offline dataset. So the low-return trajectories are down-weighted in the new distribution while the target policy can still learn useful information from these low-return trajectories. Reward perturbation helps improve the performance of RvS when the target return-to-go is too high at test time. This approach is to perturb the return-to-go during RvS training, so that the learned policy yields actions from high-return trajectories when conditioned on out-of-distribution high return-to-go.\nThe experiments are conducted on the standard offline RL dataset, D4RL. On Hopper, HalfCheetah, and Walker2d, the trajectory-weighting technique turns out to be beneficial for both DT and RvS, and reward perturbation shows improvement for RvS.\n", "summary_of_the_review": "To sum up, this paper proposes simple techniques to improve DT and RvS with extensive experiments. However, the scope is kind of limited, and the performance is not SOTA.\n", "strength_and_weaknesses": "Strengths:\nThis paper is generally well-written. The presentation is clear and easy to follow.\nThe proposed method is technically simple and easy to implement\nThere are extensive experiments on different environments, locomotion tasks, Atari games, and AntMaze (see Appendix). The comparison with hard-filtering BC in Appendix is a good supplement for the main text.\n\nWeaknesses:\nThe paper is mainly considering some tricks to improve DT and RvS, which seems incremental. The proposed idea is not general and enlightening enough to inspire follow-up works.\nThe proposed methods are more like engineering techniques, instead of a principled approach with theoretical foundations or brilliant insight.\nThe comparison with baselines is not thorough. For example, the trajectory transformer is missing in Table 1&2. Table 6&7 do not have BC, IQL, etc. The proposed method fails to beat trajectory transformer as a SOTA method on locomotion tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6hLYmgT1b6x", "reviewer": "Reviewer_vF8h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper considers offline RL with return conditioning, similar to the recent line of works on decision transformers. It considers the problem of return conditioning especially when rewards can be arbitrary depending on the offline data distribution, even if considering expert trajectories. It considers two key methods to tackle arbitrary reward conditioning - trajectory weighting and conservative regularization. These ideas are proposed assuming that expert trajectories can still contain low and high return trajectories. Experimental results are claimed to achieve significant gains of 18% and 8% depending on the state of the art offline RL algorithm, conditioning on returns, being used. ", "summary_of_the_review": "\nThe paper considers an important problem but the novelty, either algorithmic, or significance of experimental gains are rather limited. While the paper is easy to follow and argues for the need for such weighting - it seems like the two contributions are rather forced together to make this into a complete paper. It is not clear why conservative updates are still needed when trajectories are re-weighted, and the two additions to existing baselines are probably not the most important needed to tackle this problem. \n\nI think this paper is a useful contribution, but perhaps not significantly novel for acceptance in a venue like ICLR. \n\nI am however willing to re-consider my review and score if there is a more detailed justification of the proposed approach, and the importance of it, to improve existing performance on benchmarks. \n\n", "strength_and_weaknesses": "\n\t1. The paper tackles an interesting problem of return conditioning in offline RL, where even if there exists expert trajectories, the returns from offline dataset at test time can still be smaller compared to the expert data collecting policy. It argues that conditional BC can be improved to tackle this problem, by a simple mechanism of trajectory reweighting. \n\t2. Figure 1 well motivates the problem this paper tries to tackle. However, it would be helpful to understand more about this phenomenon - ie, to what extent does performance actually differ between the original and transformed distributions. This figure tries to argue that distribution of returns can be different depending on the type of data distribution. Existing benchmarks like D4RL considers expert/medium/medium-expert datasets, and it would be interesting to see the significance of this phenomenon in actual benchmark datasets used in practice. \n\t3. The core idea proposed in the paper is to consider a reweighted trajectory distribution that conditions more on the high return trajectories, such that conditioning BC on this distribution can improve performance. This leads to the problem of high variance though, since there may be fewer high return samples available. \n\t4. Equation 5 is a mechanism inspired from previous work for reweighting the trajectories. I think the main problem is the approximation of equation 5 in this case that is proposed. Moreover, equation 5 in its current form is not completely novel and is a modification of existing approaches that typically re-weights training data distribution as well; \n\t5. The mechanism is applied to existing baselines to show that performance can be marginally improved due to the addition of this reweighting mechanism. It is not well explained in the paper why such a form of regularization is needed, given the reweighting mechanism already being proposed. Are there simple demonstrations explaining the need for it? I ask this because regularization schemes in offline RL has been well studied in recent past, all on D4RL too - so unless a different form of dataset or other benchmarks are being used, this conservatism based update seems like a rather forced introduction to this paper?\nThe paper tries to well motivate the problem, but I am not sure if the experimental results presented in this form are fully convincing or not. I understand the limitations of existing offline RL experiments to d4rl benchmarks only, but recently there are new benchmarks such as v-d4rl as well, with visual observations, that might be worth considering to further make additions to experiments in this paper. Current results comparing to only two baselines seems rather limited. Results on the Atari domains, figure 9 shows marginal improvements only?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject