When does return-conditioned supervised learning work for offline reinforcement learning?
Authors: David Brandfonbrener
Alberto Bietti
Jacob Buckman
Romain Laroche
Joan Bruna
Venue: NeurIPS 2022 Accept (2022)

Abstract: Several recent works have proposed a class of algorithms for the offline reinforcement learning (RL) problem that we will refer to as return-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning on achieving high return. In this paper, we provide a rigorous study of the capabilities and limitations of RCSL something which is crucially missing in previous work. We find that RCSL returns the optimal policy under a set of assumptions that are stronger than those needed for the more traditional dynamic programming-based algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple point-mass environments and on datasets from the D4RL benchmark.

When does return-conditioned supervised learning work for offline reinforcement learning?
David Brandfonbrener
Alberto Bietti New York University
Jacob Buckman MILA
New York University
david. www.annfoobbrrerrnnu.......
Romain Laroche Microsoft Research
Joan Bruna New York University
Abstract
Several recent works have proposed class of algorithms for the offline reinforce- ment learning (RL) problem that we will refer to as ceturn-conditioned supervised learning (RCSL). RCSL algorithms learn the distribution of actions conditioned on both the state and the return of the trajectory. Then they define a policy by conditioning achieving high return. In this paper, provide rigorous study the capabilities and limitations RCSL, something which is crucially miss- ing in previous work. We find that RCSL returns the optimal policy under set of assumptions that are stronger than those needed for the more traditional dy- algorithms. We provide specific examples of MDPs and datasets that illustrate the necessity of these assumptions and the limits of RCSL. Finally, we present empirical evidence that these limitations will also cause issues in practice by providing illustrative experiments in simple posst-mass environments and on datasets from the D4RL benchmark.
namic programming-bast
Introduction
In recent years, deep learning has proven to be an exceptionally powerful generic algorithm for solving supervised learning (SL) tasks. These approaches tend to be stable, and scale well with compute and data [1].. In contrast, deep reinforcement learning algorithms seem to lack these nice properties; results well known to be sensitive to hyperparameters and difficult to replicate. In spite of this, deep reinforcement learning (RI.) has achieved impressive feats, such as defeating human champions at Go (25). This juxtaposition of success and instability has inspired researchers to explore alternative approaches to reinforcement learning that more closely resemble supervised learning in hopes of making deep RL. well-behaved as deep SL.
One family of algorithms that has garnered great interest recently is return-condittiond supervised learning (RCSL). The core idea of RCSL to learn the return-conditional distribution of actions in each state, and then define policy by sampling from the distribution of actions that receive high return. This was first proposed for the online RL setting by work on Upside Down RL [2 26 and Reward Conditioned Policies 21.. The idea was extended to the offline RL setting using transformers that condition on the entire history states rather than just the current Markovian state in the Decision Transformer (DT) work & [2 Recent work RL via Supervised Learning (RvS)
9 unifies and simplifies ideas from these prior works with ideas about goal-conditioned policies.
Importantly, none this prior work provides theoretical guarantees or analysis of the failure modes of the neturn-conditionas approach. In contrast. the more established dynamic programming (DP) algorithms for RL are better understood theoretically. This paper attempts to address this gap in
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

understanding, in order to assess when RCSL is a reliable approach for offline RL. Specifically, we answer the following questions:
What optimality guarantees can we make for RCSL? Under what conditions are they nec- essary and sufficient?
In what situations does RCSL fail in theory and in practice?
How does RCSL relate to other approaches, such as DP and behavior cloning (BC)?
We find that although RCSL does select a near-optimal policy under certain conditions, the necessary assumptions are more strict than those for DP. In particular, RCSL (but not DP) requires nearly deterministic dynamics in the MDP, knowledge of the proper value condition conditioning value to be supported by the distribution returns in the dataset. We provide simple tabular examples to demonstrate the necessity of these assumptions. The shortcomings of RCSL that we identify in theory are verified empirically with some simple experiments using neural models on ad-hoc example problems well as benchmark datasets. We conclude that RCSL alone is unlikely to be a general solution for offline RL problems, but does show promise in some specific situations such deterministic MDPs with high-quality behavior data.
and for the
2 Preliminaries
2.1 Setup
We will consider an offline RL setup where we are given dataset 'D of trajectories (01,01, (o."". oHHaHH TH) of observations Or O, actions at E A, and rewards rt E [o. 1] generated by some behavior policy / interacting with finite horizon MDP with horizon H. Let g(r) Σ t= P, denote the cumulative return of the trajectory (we will just use when the trajectory is clear from context). And let ((T) = E,-= [g(])] be the expected return of a policy т. We then let the state representation &t E S be any function of the history of observations, actions, and rewards up to step along with Oş- To simplify notation in the finite horizon setting, we will sometimes drop the timestep from refer generic states and assume that we can access the timestep from the state representation as tss). Let Pa denote the joint distribution over states, actions, rewards, and returns induced by any policy ı.
In this paper, we focus the RCSL approach that learns by retum--ondittoneed supervised learning. Explicitly, at training time this method minimizes the empirical negative log likelihood loss:
((*) -
Σ
log (ogelessee)()))
(1)
H
Then at test time, an algorithm takes the learned policy along with conditioning function f(s) to define the test-time policy y as:
(a(ss) [aa)s s. f())).
Nota bene: the Decision Transformer 8 is captured in this framework by defining the state space that the state 8t at time also contains all past Og,, ar, and re for t' < t. In prior work, f is usually chosen be constant at the initial state and to decrease with observed reward along trajectory, which is captured by a state representation that includes the history of rewards.
2.2 The RCSL policy
To better understand the objective. it is useful to first consider its optimum in the case of infinite data. It is clear that our loss function attempts to learn Pa(a]s, g) where 33 is the behavior policy that generated the data (and recall that Ps refers to the distribution over states, actions, and returns induced by 39.. Factoring this distribution, quickly that the optimal policy n RCSL for specific conditioning function f can be written
RCSL (als) Pa(als, f())) Ps(cls) B(als) ( f
(3)
()( Essentially, the RCSL policy re-weights the behavior based on the distribution of future returns.
P=(s())lss

---

Connection to distributional RL. In distributional RL A. the distribution of future returns under a policy from state and action is defined as: (s) g = -(=(()) T T ~ n. 8t(s) = de(a) = The RCSL policy is precisely proportional to the product of the behavior policy and the density of the distributional Q function of the behavior policy (i.e. Po(gls,a)).
2.3 Related work
As noted in the introduction, work is in direct response to the recent line of literature on RCSL
12 2 Specifically, we will focus on the DT S and RvS [ formulations in our experiments since they also focus on the offline RL setting. Note that another recent work introduced the Trajectory Transformer [[5] which does not fall under the RCSL umbrella since it performs
2 26. 21
planning in the learned model to define policy.
Another relevant predecessor of RCSL comes from work on goal-based RL [16 Compared to RCSL, this line of work replaces the target return y the empirical loss function by goal state. One instantiation is hindsight experience replay (HER) where each trajectory in the replay buffer relabeled as the observed final state was in fact the goal state Another instance goal- conditioned supervised learning [GCSL., [K which provides more careful analysis and guarantees, but the guarantees (I) are not transferable to the cruuun-coddtioned setting, (2) assume bounds on Loo errors in TV distance instead of dealing with expected loss functions that can be estimated from data, and (3) do not provide analysis the tightness of the bounds.
Concurrent work 34
also all raise the issue of RCSL stochastic environments with infinite 22 3
data, and present some algorithmic solutions. However, none of this work addresses the potentially more fundamental issue of sample complexity that arises from the requirement of return coverage that we discuss in Section 4.
3 When does RCSL find the optimal policy?
We begin by exploring how RCSL behaves with infinite data and a fully expressive policy class. In this setting, classic DP algorithms (e.g. Q-learning) guaranteed to converge to the optimal policy under coverage assumptions 27. But we now show that this is not the case for RCSL, which requires additional assumptions for a similar guarantee. Our approach is to first derive positive result: under certain assumptions, the policy which optimizes the RCSL objective (Section 2.2 is guaranteed to be near-optimal. We then illustrate the limitations of RCSL by providing simple examples that are nonetheless challenging for these methods in order to demonstrate why our assumptions are necessary and that our bound is tight.
Theorem (Alignment with respect to the conditioning function). Consider an MDP, behavior and conditioning function f. Assume the following:
1. Return coverage: P3(g {(=)){=1) 2 a for all initial states 81-
2. Near determinism: P(r + r(s, a) or s' + T(s, alssa))ssoo a) S at all s, for some functions T and r. Note that this does not constrain the stochasticity of the initial state.
3. Consistency of f: f(s)= f(s') for all
Then
Ea,1[(())) JJ (T Se (=++2 H?.
af
Moreover, there exist problems where the bound is light up to constant factors.
The proof is in Appendix C.1 Note that the quantity E., [s11] is specific to the structure of RCSL algorithms and captures the notion that the ideal RCSL policy will be able to reproduce policies of any value when given different conditioning functions (with appropriate data). The theorem immediately yields the following corollaries (with proof in Appendix C.I.
Note this can be exactly enforced (as in prior work) by augmenting the state space to include the cumulative reward observed far.
3

---

Corollary 1. Under the assumptions of Theorem
there exists conditioning function f such that
+3 (1+3) H².
J(T') - J Sc
(5)
a
Corollary 2. af > 0, and [(61) (v (or all initial states st, then ( J (TTT).
The corollaries tell us that in near determinisitc environments with the proper conditioning functions and data coverage, is possible for RCSL to recover near optimal policies. These assumptions are somewhat strong compared to those needed for DP-based approaches, they are necessary for our analysis.
will now explain why
Tightness. To demonstrate tightness we will consider the simple examples in Figure | These MDPs and behavior policies demonstrate tightness and a up constant factors, and provide insight into how stochastic dynamics lead to suboptimal behavior from RCSL algorithms.
0= r=l-é
8 =
7
B(1
1.0
81
r B(e)
$1
$1
#1
$=
B(e)
0 1=1--- B(e)
(a) An example where the bound is (b) An example where RCSL also tight. B denotes the Bernoulli dis- has large regret. tribution.
(c) An example where RCSL also has large regret any condition- ing function.
Figure 1: Failure modes of RCSL in stochastic environments with infinite data.
First, consider the example in Figure la with conditioning f(s1) 1. There is only one possible policy in this case, and it has (TT) = so that [[]( = 1 c. Note that a c, so we have that e/ay 1. Thus, the bound is tight in e/ay. This example shows that the goal of achieving a specific desired return is incompatible with stochastic environments.
This first example is somewhat silly since there is only one action, so the learned policy does not actually suffer any regret. To show that this issue can in fact lead to regret, consider the example in Figure again with conditioning ((11) 1. Then applying the reasoning from Section 2.2
Pg(g 1| 01)
f
RCSL (acsL(a11s1) = (((1(s1)) Pg = 1|s1) = 0.5 0.5 . = 0.
(6)
So get that [[f(s:)] - order, up to a constant factor). However, in this case the learned policy y
1-c, while c/ay e/(e/2) 2 (which is the same
RCSL suffers substantial
regret since the chosen action 02 has substantially lower expected value than a by 1- 2e.
The issue in the second example could be resolved by changing the conditioning function so that f(s1) 11 c. Now we will consider the example in Figure Ic where we will see that there exist cases where the bias of RCSL in stochastic environments can remain regardless of the conditioning function. In this MDP, the only returns that are supported are g = 0 or = 1. For f(s1) 1, plugging in the formula for n yields
Paly = 1|81, 01) E
RCSL Ff (a (alls)) = ((ai(s))
Pe(g 1[51) e(1 6) + ()- 2
Thus, B[f(s:)] Jmm RSSL) 1/2 and J(T") (TRCSL) 1/2 - c. This shows that merely changing the conditioning function is not enough to overcome the bias of the RCSL method in stochastic environments.
These examples show that even for MDPs that are c-close to being deterministic, the regret of RCSL can be large. But, in the special case of deterministic MDPs we find that RCSL can indeed recover optimal policy. And note that we still allow for stochasticity in the initial states in these deter- ministic MDPs, which provides rich setting for problems like robotics that requires generalization over the state space from finite data. In the next section, we will consider more precisely what happens to RCSL algorithms with finite data and limited model classes.

---

Trajectory stitching, Another issue often discussed in the offline RL literature is the idea of tra- jectory stitching [31 g. Ideally, an offline RL agent can take suboptimal trajectories that overlap and stitch them into a better policy. Clearly, DP-based algorithms can do this, but it is not so clear that RCSL algorithms can. In Appendix B we provide theoretical and empirical evidence that fact they cannot perform stitching in general, even with infinite data. While this does not directly affect our bounds, the failure to perform stitching is an issue of practical importance for RCSL methods.
4 Sample complexity of RCSL
Now that we have a better sense of what policy RCSL will converge to with infinite data, we can consider how quickly (and under what conditions) it will converge to the policy т when given finite data and a limited policy class, as will occur in practice. We will do this via a reduction from the regret relative the infinite data solution π to the expected loss function L minimized at training time by RCSL, which is encoded in the following theorem.
Theorem (Reduction of RCSL. to SL.. Consider any function f S:S - TR such that the following two assumptions hold:
(g Ppts)
1. Bounded occupancy mismatch:
S C for all 8.
2. Return coverage: Pa(g = f(s)[s) 2 a for all s.
Define the expected loss as [(a) By [DKL (PaL(Pa( ( estimated RCSL policy it that conditions on at test time (denoted by ft), we have that
g)](((((ss)))))))) g))] Then for any
Cf H2 22 J J(tt) S af
The proof can be found in Appendix coverage as before to ensure we have sufficient data to define Ty. We also require an assumption on the state occupancy of the idealized policy y relative to This assumption is needed since the loss L(f) is optimized on states sampled from P3. but we care about the expected return of the learned policy relative to that of zy. which can be written as an expectation over states sampled from Px,,
Note that we require a similar assumption of return
This gives us reduction to supervised learning, but to take this all the way to sample complexity bound we need to control the loss L(+) from finite samples. Letting N denote the size of the dataset, the following corollary uses standard uniform convergence results from supervised learning 24 to yield finite sample bounds.
Corollary 3 (Sample complexity of RCSL). To get finite data guarantees, add to the above assump- tions the assumptions that (I) the policy class II is finite, (2) log (a)s, g) -log π (a') s', s',g') S cfor any (a, 9, a', 8', g') and all π E II, and (3) the approximation error of II is bounded by Capprox. L.e. min L(T)) S Eupproa- Then with probability at least 8,
C Ci (va as
log IT/55) | 1/4
(( ( 50
Eapproz
The proof is in Appendix C.41
Analyzing the bound, terms of fourth root rather than the square root, but this comes from the fact that we are optimizing surrogate loss. Namely the learner optimizes KI. divergence, but we ultimately care about regret which we access by using the KL term to bound a TV divergence and thus lose a square root factor. A similar rate appears, for example, when bounding 0-1 classification loss of logistic regression K 5)
see that the dependence on N is in
This corollary also tells us something about how the learner will learn generalize across different values of the return. the policy class is small (for some notion of model complexity) and suffi- ciently structured, then it can information from the given data to generalize across values of g. using low-return trajectories to inform the model on high-return trajectories.
Note that a full sample complexity bound that competes with the optimal policy can be derived by combining this result with Corollary [ as follows:
5

---

Corollary 4 (Sample complexity against the optimal policy). Under all of the assumptions of Corol- laryl 1 and Corollary we ger:
( - H²
(vo(Lo8) log /0 /0 N
1/4
J(x*) J(ay) so
+ V Capproa
c + H2 af :FP)
(10)
af
Tightness. To better understand why the dependence on 1/ay is tight and potentially exponential in the horizon H, even in deterministic environments, we offer the example in Figure Specifically, we claim that any value of f()1) where the policy =f
RRRSLL prefers the good action di from 81 will
require on the order of 10 samples in expectation to recover as क
To see why this is the case, we consider the MDP illus- trated in Figure 2 with horizon H >> The MDP has four states each with two actions. All transitions and re- wards are deterministic. The only actions with non-zero reward are r($2, a1) = 1 and r(83, a1) = 0.5. The inter- esting decision at 81 where a is better than ag.
==
3
0 = 0.5
82
8 0.9
41
Note that for any integer <
1/2,, we have that Ps(g k|s1, 02) 0.5 0.5 0.5 (0.25)*, while Po(g k|s1, 01) 0.5 (0.1)*. Conditioning
81
$4
427
0 0.5
any such & will make us more likely to choose the bad action 02 from $1 The only way increase the likelihood of the good action 01 from 81 and 82 is to condition on ((1 > H/2. Unfortunately for RCSL, the probability of observ- ing 9 > H/2 is extremely small, since for any such f we have P3(g = (8))) & 0.5 (0.1) < 10-H/2 Thus, both ay and the sample complexity of learning for any ʃ that will yield policy better than the behavior is exponential in the horizon H.
$3
= 0.5
01 /
0.5
Figure 2: An example where RCSL has exponential sample complexity in de- terministic environment.
Fundamentally, the problem here is that RCSL uses trajectory-Ievel information instead of per- forming dynamic programming individual transitions. But, collecting enough trajectory-level information can take exponentially many samples in the horizon. In contrast, DP merely requires coverage of transitions in the MDP to perform planning and thus avoids this issue of exponential sample complexity. In the next section we will delve deeper into this comparison with DP-based approaches as well as the simple top- % BC baseline.
Comparing RCSL with bounds for alternative methods
Now that we understand the rate at which we expect RCSL to converge, we briefly present the con- vergence rates of two baseline methods for comparison. In particular, we will leverage an existing analysis of a DP-based algorithm, and conduct a novel analysis of top-% BC. We find that the sam- ple complexity of RCSL has similar rate top- % BC, and is worse than DP due to the potentially exponential dependence on horizon that stems from return coverage.
5.1 Comparison to dynamic programming.
We will compare to the state of the art (to our knowledge) bound for a DP-based offline RI. al- gorithm. Namely, we will look at the results of [32 for pessimistic soft policy iteration. Similar results exist for slightly different algorithms or assumptions 7 30), but we choose this one since it both the tightest and more closely aligns with the practical actor-critic algorithms that we use for our experiments. Their bound makes the following assumptions about the function class F and the dataset (letting T represent the Bellman operator for policy T):
1. Realizability: for any policies π, n' there exists f E F with - S1.
2. Bellman completeness: for any л and f E F there exists f' E F such that |1 T TT,li,, S
3. Coverage:
(s.(a < C for all Pa(xaa)
FExcept for f(+1) = 0, which will yield policy substantially worse than the hehavior.
'The original paper uses slightly tighter notion of coverage, but this bound will suffice for comparison.

---

With these assumptions in place, the sample complexity bound takes the form
=o H² (VClo
1F + H2 C(e +
=0
(11)
N
Note: this is the result for the "information-thhoreee implemented. The paper also provides a "practical" version of the algorithm for which the bound is the same except that the the square root in the first term is replaced with fifth root.
form of the algorithm that cannot be efficiently
There are several points of comparison with analysis (specifically, our Corollary The first thing to note is that for RCSL to compete with the optimal policy, we require nearly deterministic dynamics and a priori knowledge of the optimal conditioning function. These assumptions not required for the DP-based algorithm; this is a critical difference, since it is clear that these conditions often do not hold in practice.
Comparing the coverage assumptions, our C3 becomes nearly equivalent to C. The major difference that our analysis RCSL also requires dependence on return coverage 1/ay. This is problematic since seen in Section 4 this return coverage dependence can be exponential in horizon in cases where the state coverage does not depend on horizon.
Comparing the approximation error assumptions, we see that the realizability and completeness assumptions required for DP are substantially less intuitive than the standard supervised learning approximation error assumption needed for RCSL. These assumptions are not directly comparable, but intuitively the RCSI. approximation error assumption is simpler.
Finally, dependence on H is the same for both methods and dependence on N depends on which version of the DP algorithm we compare to. For the coormationnthhoredte algorithm DP has better dependence on N, but for the practical algorithm RCSL has better dependence. It is not clear whether the dependence on N in either the RCSL analysis or in the analysis of the practical algorithm from 32 is tight, and it is an interesting direction for future work to resolve this issue.
5.2 Comparison to top-% % behavior cloning.
The closest algorithm to RCSL is top-% BC, which was introduced as a baseline for Decision Trans- formers [ This algorithm simply sorts all trajectories in the dataset by return and takes the top fraction of trajectories to run behavior cloning (for p E [0. I).. The most obvious difference between this algorithm and RCSL is that RCSL allows us to plug in different conditioning functions at test time to produce different policies, while top-% % BC learns only one policy. However, if we want to achieve high returns, the two algorithms are quite similar.
The full statements and proofs our theoretical results for top- % BC are deferred to Appendix C.5 The results essentially the same as those for RCSI. except for key modifications:
Defining coverage. The first difference in the analysis is the notion of coverage. For RCSL needed the return distribution to cover the conditioning function f. For top-% BC we instead let Jp be the 1 P quantile of the return distribution over trajectories sampled by the behavior B and then define coverage as Pg(g 2 gpls) 2 ap for all s. This modification is somewhat minor.
Sample size and generalization. The main difference between RCSL and top-% % BC is that the RCSL algorithm attempts transfer information gained from low-return trajectories while the top- % BC algorithm simply throws those trajectories away. This shows up in the formal bounds since for a dataset of size N the top-% BC algorithm only uses p N samples while RCSL uses all N. Depending on the data distribution, competing with the optimal policy may require setting p very close to zero (exponentially small in H) yielding poor sample complexity.
These bounds suggest that RCSL can generalization actoss returns to provide improvements in sample complexity over top-% BC by leveraging all of the data. However, the RCSL model is attempting to learn richer class of functions that conditions on reward, which may require larger policy class negating some of this benefit. Overall, RCSL should expect to beat top-% BC the behavior policy is still providing useful information about how interact with the environment in low-return trajectories that top-% BC would throw away.
'The original paper considers infinite horizon discounted setting. For the purposes of comparison, we will just assume that - can be replaced by H.
7

---

6 Experiments
We have illustrated through theory and some simple examples when we expect RCSL to work, but the theory does not cover all cases that are relevant for practice. In particular, it is not clear how the neural networks trained in practice can leverage generalization across returns. Moreover, one of the key benefits RCSL approaches (as compared to DP) that by avoiding the instabilities of non-linear off-policy DP in favor of supervised learning, one might hope that RCSL is more stable in practice. In this section we attempt to test these capabilities first through targeted experiments in a mass environment and then by comparisons on standard benchmark data.
Throughout this section we will consider six algorithms, two from each of three categories:
Behavior cloning (BC): standard behavior cloning (BC) and percentage behavior cloning (&BC) that runs BC on the trajectories with the highest returns [
2. Dynamic programming (DP): TD3+BC [D] simple DP-based offline RL approach and
IQL, 20
more stable DP-based offline RL approach.
3. Return-conditione supervised learning (RCSL): RvS g an RCSL approach using simple
MLP policies, and DT' an RCSL approach using transformer policies.
All algorithms are implemented in JAX 6 using flax | and the jaxrl framework
[9|
except for and code can
DT which is taken from the original paper. Full details can be found in Appendix D
be found at httttt::/ggtthiie cww hato:/davibbbaaddn http://vvwirraaaaaoooreeeee/eeee ootaraaeeaoobeeeeeeeeeeaaaaaaa
6.1 Point-mass datasets
First, we use targeted experiments to demonstrate how the tabular failure modes illustrated above can arise even in simple deterministic MDPs that may be encountered in continuous control. Specifically, we will focus on the issue of exponential sample complexity discussed in Section ! We build our datasets in environment using the Deepmind control suite (28 and MuJoCo simulator 29] The environment consists of a point-mass navigating in a 2-d plane.
To build an example with exponential sample complexity we construct navigation task with a goal region in the center of the environment. The dataset is constructed by running behavior policy that a random walk that is biased towards the top right of the environment. To construct different levels of reward coverage, consider the environment and dataset under three different reward functions (ordered by probability of sceing a trajectory with high return, from lowest to highest):
(a) The "ring of fire" reward. This reward is within the goal region, -1 in the ring of fire region surrounding the goal, and O otherwise
(b) The sparse reward. This reward is within the goal region and 0 otherwise.
(c) The dense reward. This reward function is within the goal region and gradually decays with the Euclidean distance outside of it.
Intuitively, the ring of fire reward will cause serious problems for RCSL approaches when combined with the random walk behavior policy. The issue is that any random walk which reached the goal region is highly likely to spend more time in the region of negative rewards than in the actual goal states, since the ring of fire has larger area than the goal. As result, while they are technically supported by the distribution, it is unlikely to find many trajectories (if any at all) with positive returns in the dataset, let alone near-optimal returns. As a result, the RCSL-based approaches are not even able to learn to achieve positive returns, as seen in Figure
The sparse reward is also difficult for the RCSL-based algorithms, for similar reasons; however the problem is less extreme since any trajectory that gets positive reward must go to the goal, so there is signal in the returns indicating where the goal is. In contrast, the dense reward provides enough signal in the returns that RCSL approaches are able perform well, although still not as well as IQL. It is also worth noting that because the datset still does not have full coverage of the state-space, simple basea algorithms like TD3+BC can struggle with training instability.
6.2 Benchmark data
In addition to our targeted experiments we also ran our candidate algorithms on some datasets from the D4RL benchmark [10]. These are meant to provide more realistic and larger-scal scale data scenarios.
8

---

-41
4.4
BC
% BC
TD3+BC (DP)
(DP)
100
Rv5 (RCSL)
DT (RCSL)
(a) Ring of fire
(b) Sparse
(c) Dense
Figure 3: RCSL fails under reward functions that lead to exponentially small probability of sampling good trajectories, but can generalize when the reward is dense. Error bars show standard deviation across three seeds. BC methods are in blue, DP methods in brown, and RCSL methods in green.
While this also makes these experiments less targeted, we can still see that the insights that gained in simpler problems can be useful in these larger settings. We attempt to choose subset of the datasets with very different properties from eachother. For example, the play data on the ant-maze environment is very diverse and plentiful while the human demonstration data the pen environment has poor coverage but high values. Results are shown Figure M And additional results leading to similar conclusions can be found in Appendix A
We find that for most of the datasets DP-based algorithms TD3+BC and IQL outperform both the BC-based algorithms and RCSL-based al- gorithms. This is especially stark on the ANTMAZE datasets where the behavior pol- icy highly stochastic, requiring the learner to stitch together trajectories to achieve good performance. While none of these tasks has stochastic dynamies, the issues of return cov- erage and trajectory stitching persist.
BC
% BC
TD3+BC (DP)
IOL IDF:
RUS IRCSLI
DT (RCSL)
AM med phy
HC mos no
Piumann
In contrast, RCSL performs well when the behavior policy is already high quality, but optimal (as in the PEN-HUMAN task). Since the data is suboptimal and reward is dense, there opportunity for RCSL out- perform the BC-based methods. Moreover, since the data has poor coverage, standard DP approaches like TD3+BC are highly unstable. IQL is more stable and performs similarly to RCSL-based algorithms, outperformed by DT (perhaps due to the use of History-dependee policies).
ANTMAZE CITMAZZE-UMZZE HALFCHEETAH- Figure 4: Data from ANTMAZE-MEDIUME MEDIUM REPLAY, and PEN-HUMAN. Error bars show standard deviation across three seeds. Each algorithm is tuned over values and best performance is reported.
7 Discussion
Looking back at our results, we can better place RCSL in relation to the more classical BC and DP algorithms. Like BC, RCSL relies supervised learning and thus inherits its simplicity, ele- gance, and ease implementation and debugging. However, also inherits BC's dependence on the quality of the behavior policy. This dependence can be somewhat reduced in (nearly) deterministic environments, where conditioning on high returns can break the bias towards the behavior policy. But, the reliance trajectory-level information still means that RCSL is fundamentally limited by

---

the quality of the best trajectories in the dataset, which can require a sample complexity exponential in horizon in order compete with the optimal policy, even deterministic environments.
In contrast, DP methods are capable of learning good policies even when the dataset does not con- tain any high-return trajectories and the environment is stochastic. This represents a fundamental gap between the two approaches that cannot be bridged within the RCSL paradigm. However, em- pirically, current deep DP algorithms are not well-behaved. These algorithms are often unstable and difficult debug, although recent work has started to alleviate these issues somewhat 20
In sum, for tasks where the requirements for RCSL to perform well met, it is an excellent prac- tical choice, with great advantages in simplicity over DP. Since many real-world tasks of relevance have these attributes, RCSL techniques could have substantial impact. But general learning paradigm, RCSL is fundamentally limited in ways that DP is not.
Acknowledgments
This work was partially supported by NSF RI-1816753, NSF CAREER CIF 1845360, NSF CHS- 1901091, NSF Scale MoDL DMS 2134216, Capital One and Samsung Electronics. DB was sup- ported by the Department of Defense (DoD) through the National Defense Science Engineering Graduate Fellowship (NDSEG) Program.
References
[1] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31. PMLR, 2017.
[2] M. Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and W. 7aremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] L. Bartlett, M. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156 2006.
[4] M. G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning. MIT Press, 2022. http://www. www.:diissriiiionaaaaaaoaa
[5] S. Boucheron, O. Bousquet, and G. Lugosi. Theory of classification: A survey of some recent advances. ESAMM: probability and statistics, 9:323-375, 2005.
[6] Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula,
A. Paszke, J. VanderPlas, S. Wanderman-Milng and Q. Zhang. JAX: composable transfor- mations of Python+NumPy programs, 2018. URL http: http::///hthbb com/ com/google/jax
[7] Chen and N. Jiang. information-theoreer considerations in batch reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 2019.
[8] I.. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint w1Xi::2106.01345, 2021.
[9] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.1075: 2021.
[10] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint wwxiv:2004.07219, 2020.
[11] S. Fujimoto and S. S. Gu. minimalist approach offline reinforcement learning. arXiv preprint 20t.::0006.066000 2021.
[12] H. Furuta, Y. Matsuo, and S. S. Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10336 2021.
[13] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 2019.
[14] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee. Flax: neural network library and ecosystem for JAX, 2020. URI. http:://tttttubb.... http:::///gituuu..comoo/////// www.s/wlwoooooooogeoooooooooooee google/ flax
10

---

[15] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34, 2021.
[16] L. P. Kaelbling. Learning achieve goals. In IJCAI, 1993.
[17] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint 2020.
arXiv:2001.08.3
[18] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint
A4Xiv: 22800, 2014.
[19] I. Kostrikov. Jaxrl: Implementations of reinforcement learning algorithms in jax., 10 2021. URL https://githhbb. com/ikoxtrikov/jurl,, 2021.
[20] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning.
arXiv preprint arXiv:2110.06166 2021.
[21] A. Kumar, X. B. Peng, and S. Levine. Reward-conditioeed policies. arXiv preprint arXiv: 13465, 2019.
[22] K. Paster, S. Mcllraith, and Ba. You can't count on luck: Why decision transformers fail in stochastic environments. arXiv preprint arXiv:2205. 15967, 2022.
[23] Schmidhuber. Reinforcement learning upside down: Don't predict rewards-jus map them to actions. arXiv preprint (ttp:::991..22288 2019.
[24] S. Shalev-Shwartz and S. Ben-enDDaae -Dvidd Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
[25] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-489, 2016.
[26] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaškowski, and J. Schmidhuber: Training agents using upside-down reinforcement learning. arXiv preprint 191208771 2019.
[27] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[28] Y. Tassa, Doron, Muldal, T. Erez, Y. Li, D. d. Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancg, et al. Deepmind control suite. arXiv preprint UrXiv:1801.0069) 2018.
[29] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026- 5033, 2012. doi: 10.1 (O9//RROS.002.663800
[30] R. Wang, P. Foster, and S. M. Kakade. What are the statistical limits of offline rl with linear function approximation?, 2020.
[31] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
N. Siegel, C. Gulcehre, N. Heess,
[32] T. Xie, C.-A. Cheng, N. Jiang P. Mineiro, and A. Agarwal. Bellman-consiste pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[33] M. Yang, D. Schuurmans, P. Abbeel, and O. Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210.13433 2022.
[34] M. Strupl, F. Faccio, D. R. Ashley, J. Schmidhuber, and R. K. Srivastava. Upside-down rein- forcement learning can diverge in stochastic environments with episodic resets, 2022.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix E
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix D
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix D
(c) Did you include any assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "zu_3rwoZkDL", "reviewer": "Reviewer_6EAT", "summary": "This paper discuss the conditions that return-conditioned supervised learning (RCSL) work for offline reinforcement learning (RL). This paper mainly contributes theoretically on the optimality of the RCSL algorithms and the sample complexity of the RCSL algorithms. Under some strong assumptions, they present theoretical guarantees for RCSL given both infinite data and finite data. Furthermore, empirical evidence on failure and success of RCSL methods are provided.", "questions": "The reviewer has the following several questions:\n\na) why discussing the connection of RCSL to distributional RL? They actually lie in quite different categories.\n\nb) the theoretical results in appendix (on top-%BC) also involve the assumption of \"near determinism\". Then will top-%BC fail on stochastic environments? If we include all the data, i.e., top-100% BC or vanilla BC, are the derived bounds still meaningful? Any discussion here?\n\nc) the authors claim that DP-based methods are often \"unstable and difficult to debug\" (line 371-372), then are RCSL methods easy to debug and exhibit stability during training? Why RCSL methods are easy to debug? It is hard to say that decision transformer (DT) is easy to tune and debug.\n\nd) what is the benefits of RCSL on earth? Since IQL, a DP-based method, shows very good performance over all of the tested datasets. Does there exist some methods that can make RCSL work in stochastic environments? The authors mention that trajectory transformer works on MuJoCo datasets as it involves planning, will planning breaks this dilemma?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality and Significance**: this paper is quite original, which theoretically studies a very new topic in RL. The conclusions in this paper is of importance to the RL community, and can benefit the progress of offline RL\n\n**Quality**: this paper has a good quality overall. The theoretical results seem sound to some extent. The reviewer randomly checked some proofs in the appendix (but not all of them), and they seem correct, despite some minor errors in some of them. For example, in appendix Eq. (56), the reviewer thinks it ought to be $\\epsilon (\\frac{1}{\\alpha_p}+1)$ instead of $2\\epsilon (\\frac{1}{\\alpha_p}+1)$. The theoretical contribution of this paper is clear. While the empirical experiments are kind of disappointing\n\n**Clarity**: this paper is of medium clarity. The reviewer likes the way the authors tell a story like this. While the figures are of poor quality, which negatively affect the clarity of this paper. Some figures, e.g. figure 3 and 4, are sort of blur.\n\n**Summary of Strengths**:\n\na) the discussed topic is important and interesting. It is vital to figure out when and under what conditions will RCSL work for offline RL\n\nb) the theoretical contribution of this paper is of importance to better understand \"when return-conditioned supervised learning work for offline reinforcement learning\"\n\nc) the toy examples presented in the paper is quite helpful to understand why RCSL fail in stochastic environments\n\n**Weaknesses**:\n\na) some of the figures are blur.\n\nb) there are some grammatical issues, e.g., line 199 \"To see why this is the case, consider the MDP illustrated\" --> \"To see why this is the case, we consider the MDP illustrated\". Please do not use colloquial expressions in submitted papers. While this is just a random example. Please double check the paper to ensure that there are no grammatical issues.\n\nc) the discussion lacks depth. For example, in Section 5.1, the authors compare the sample complexity bound of RCSL and dynamic programming (DP). The reviewer cannot tell what is the point here, since the basic assumptions for these bounds are different. Whereas the bound for top-%BC are omitted in the main text. Meanwhile, the discussion in Section 5.1 does not reveal why RCSL fails while DP-based methods work.\n\nd) the empirical evidence is expected to be enriched. Currently, RCSL is only tested on some selected datasets, e.g., 1 D4RL MuJoCo dataset, 1 D4RL Adroit dataset, and 2 D4RL Antmaze datasets. RCSL fails/underperforms baselines on 3 out of 4 datasets. The reviewer agrees that MuJoCo/Antmaze datasets come from \"stochastic\" environments. While there is only one single test on datasets with narrow distribution (e.g., human). The reviewer cannot say that it can convince readers on the proposed claims in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "cVeFKbJqU2s", "reviewer": "Reviewer_a7eE", "summary": "This paper studies the capacities and limitations of return-conditioned supervised learning (RCSL) methods and illustrates their relationship to behavior cloning (BC) and dynamic programming (DP). The main conclusion is that RCSL methods are theoretically incapable of achieving high performance when the underlying environment is non-deterministic and the return coverage is insufficient. These findings are both intuitive and theoretically sound.", "questions": "1. In the training phase, RCSL minimizes the empirical negative log-likelihood loss described by Equation (1), while the test-time policy is defined as Equation (2). It seems that there is a mismatch between the training-time policy and the test-time one: the training-time policy is conditioned on the return of an entire trajectory $g(\\tau)$ but the test-time policy is conditioned on a condition function $f(s)$ which varies with the states in a trajectory. The authors should explain it. Besides, will this discrepancy affect the conclusion?\n2. The authors did not pre-define the variable $N$, which firstly appears in line 182 and Equation 9. (I guess it refers to the dataset size)\n3. Though the authors claim that RCSL is incapable of learning high-performance policies when the environment is stochastic, which is described by Theorem 1 and explained by Figure 1, I think the authors may construct toy MDPs to provide experimental evidence in the experimental part.\n", "limitations": "It would be better if the authors could conduct experiments to support their theoretical claims. For example, experiments investigating the return coverage and stochasticity of the dataset and the environment, respectively.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The novelty of this paper lies in the theoretical part. The authors start with the setting with infinite data and a fully expressive policy and then extend the derived theorems to a more practical setting with finite data and a limited policy class. The theoretical derivations are solid and the theorems are easy to understand. The core conclusion is that RCSL is fundamentally limited when the environment is stochastic and the return coverage is insufficient, which is intuitive and theoretically sound. However, from my perspective, I think the authors should strengthen the experimental parts. For example, to verify that RCSL is indeed limited to near-deterministic environments, the authors should conduct experiments in environments with different stochasticity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MfU4GpRjgEN", "reviewer": "Reviewer_cWMV", "summary": "This paper theoretically analyzes a new popular class of RL algorithms, referred to as Return-Conditioned Supervised Learning (RCSL). The paper theoretically shows that this class of method requires stronger assumptions than standard DP-based approaches for learning the optimal policy. The paper shows that RCSL method requires near-deterministic dynamics as well as a priori knowledge of the optimal conditioning function to perform well, and construct examples where the stated assumptions are necessary for empirical performance. ", "questions": "1. It's unclear what insight Section 6.2 adds; I believe similar findings have already been reported in prior papers, for example, https://arxiv.org/abs/2112.10751. \n\nI am willing to increase my score if the authors can address my qualm about the theoretical analysis in the above section. Overall, this is a sound paper. ", "limitations": "Yes. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper is solid in all four axis: originality, quality, clarity, and significance. It is the first paper to provide a theoretical analysis of a new class of practically relevant RL algorithms. The exposition is clear, and the theoretical results are mostly proper (though, I did not check the supplement fully). The experiments are well thought out, and complement the theoretical analysis. Given that this class of algorithm has received significant attention from the RL community, this work is timely and relevant. \n\nWeakness:\n1. It's unclear that under deterministic dynamics, RCSL can recover the optimal policy, is a novel result. Under this assumption, the return can be practically treated as a new state-dimension, and RCSL reduces to Behavior Cloning with an augmented state space.\n2. The dependence on the size of the policy class needs to be clarified when comparing RCSL and DP-based algorithms in Section 5. Even in the tabular case, just to represent the policy, it would require a lookup table that is exponentially larger than what is required to represent a standard policy for DP approaches (i.e, |S||A|). For this reason, it's not clear that the comparison is super meaningful. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IFI56J_2HUo", "reviewer": "Reviewer_sw7o", "summary": "The paper considers return-conditioned supervised learning (RCSL), a class of methods that learn a return-conditioned policy from offline data and recover the evaluation policy by conditioning on a given desired return. It then analyzes these method theoretically, bounding the regret of RCSL policies under the assumption of close-to-deterministic transitions&rewards and return-coverage of the behavior policy. Moreover, the paper provides examples of MDPs where RCLS is not able to recover the optimal policy, which demonstrate that the additional assumptions are required. Lastly, in the experimental evaluation the paper shows on a toy-example scenarios where dynamic programming has advantages over RCSL and also evaluate performance on the D4RL dataset.", "questions": "Overall I think this is a well-written and interesting paper. It would perhaps be useful to explicitly state a lower-bound on performance given that the introduction is largely motivated from that perspective. It would be nice to expand the experiment section a bit further: For the toy-example and ablation study on how performance changes depending on the behavior policy would be useful and more D4RL baselines could be run. For the latter it would be particularly interesting how performance compares on different behavior datasets for the same environment.", "limitations": "While the theory clearly states its assumptions, limitations of the analysis are not discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written, easy to follow, and studies a class of methods that has been of interest lately. The theoretical analysis clearly illustrates when we can expect RCSL to work and the example-MDPs are useful to understand when it doesn't. The comparison to dynamic programming and behavior cloning is interesting, but comparing lower-bounds would probably be more convincing. The experiment section illustrates the main points of the paper, but is relatively limited in scope.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept