RvS: What is Essential for Offline RL via Supervised Learning?
Authors: Scott Emmons
Benjamin Eysenbach
Ilya Kostrikov
Sergey Levine
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
deep reinforcement learning
offline reinforcement learning

Abstract: Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin RvS learning). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.

Published as a conference paper at ICLR 2022
RvS: WHAT IS ESSENTIAL FOR OFFLINE RL VIA SUPERVISED LEARNING?
Scott Emmons' Benjamin Eysenbach", Ilya Kostrikov',
Sergey Levine'
UC Berkeley, Carnegie Mellon University
emmons@berkeleyy edu
ABSTRACT
Recent work has shown that supervised learning alone, without temporal differ- ence (TD) learning. can be remarkably effective for offline RL. When does this hold true, and which algorithmic components necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential el- ements. In every environment suite consider, simply maximizing likelihood with two-layer feedforward MI.P is competitive with state-of-the-art results substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization architecture) and choosing which information condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin RyS learning). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest number of open problems.
INTRODUCTION
Offline and off- off-policy reinforcement learning (RL) are typically addressed using value-based meth- ods. While theoretically appealing because they include performance guarantees under certain as- sumptions (27. such methods can be difficult apply practice; they tend to require complex tricks to stabilize learning and delicate tuning of many hyperparameters. Recent work has explored an alternative approach: convert the RL problem into conditional, filtered, or weighted imitation learning problem. This typically uses simple insight: suboptimal experience for one task may be optimal for another task. By conditioning on some piece of information, such as a goal, re- ward function parameterization,
reward value, such experience can be used for simple behavior
cloning (5] 6, 8 10 17 23 26 29 32 34 38). We refer to this set of approaches as RL VIA SU- PERVISED LEARNING (RvS). These approaches commonly condition goals 6 T8 29
reward
values 5 26
38
but they can also involve reweighting or filtering 10
26
32
34
Suite D4RL Kitchen
Suite D4RL Gym
Suite D4RL AntMaze
RvS-G
RvS-R
RvS-G
RvS-R
RvS-R
CQL-r
CQL-p
CQL-p
BC
CQL-r
Filt. BC
BC
CQL-r
Filt. BC
DT
BC
TD3+BC
DT
TD3+BC
Filt. BC
Onestep
Onestep
0
20
40 Return
25 50
20 40 Return
Return
RvS learning conditioned on goals (RvS-G) and on rewards (RvS-R) compared with prior approaches and baselines. Each bar is the average over many tasks in each suite. Using just supervised learning with a feedforward MLP, RvS matches the performance of methods employing TD learning and Transformer sequence models. https:: /ttp::///gthhts com/ wwwotemmonsss.ooooooooooooooosst rvs
Figure/

---

Published as a conference paper at ICLR 2022
states 8,
2
$
state
actions
a
ag
|
18
dropout
dropout
-concat
outcomes
W
22
1821
2
32)
action
fc layer
layer
[8g)
3
0,)
outcome w
(a) replay buffer
(b) training dataset
(c) network architecture
Figure 2: (a) As input, RvS takes a precollected replay buffer of experience. An outcome w can be an arbitrary function of the trajectory, such as future states rewards. (b) RyS uses hindsight relabeling of the replay buffer to construct training dataset. The observed actions act as demonstrations for the observed outcomes. (c) Our implementation of RvS uses an MLP with two fully connected (fc) layers to predict actions. At test time, we can condition on arbitrary outcomes.
RvS methods are appealing because of their algorithmic simplicity. However, prior work has put forward conflicting hypotheses about which factors are essential for their good performance, includ- ing online data [18). advantage weighting [26]. or large Transformer sequence models [5]. The first question study is: what elements are essential for effective RyS learning? Beyond this, it also remains unclear on which tasks and datasets such methods work well. For example, prior work has argued that temporal compositionality (dubbed "subtrajectory stitching") is an important com- ponent for solving offline RI. when there are few near-optimal trajectories present in the data (e.g., the Franka Kitchen and AntMaze tasks D4RI. 12]). A priori, one might expect that dynamic programming via TD learning is needed for these tasks. So we also ask: what are the limits of RyS learning, and does it scale to settings with few near-opae cetrooltimal trajectories?
The main contribution of this paper is a study of RyS methods on a wide range of offline RL prob- lems, as well as a set of analyses about which design factors matter most for such methods. First, show that pure supervised learning (maximizing the likelihood of actions observed in the data) performs as well as conservative TD learning across a diverse set of environments. Second, simple feedforward models can match the performance of more complex sequence models from prior work across a wide range of tasks. Finally, choosing to condition on reward values versus goals can have large effect on performance, with different choices working better in different domains. These simple results contradict the narrative put forward in many prior works that argue for more complex design decisions S 26 27. To the best of our knowledge, our results match or exceed those re- ported by any prior RyS method. We believe that these findings will be useful to the RL community because they help to understand the essential ingredients and limitations of RvS methods, providing a foundation for future work simple and performant offline RL algorithms.
RELATED WORK
Recent offline RL methods use many techniques, including value functions 15 27 dynamics esti-
mation alongisde value functions 24
37 42 dynamics estimation alone [
7 39 and uncertainty
this paper, we focus on offline RL methods based on conditional be- havior cloning that avoid value functions. The most common instantiation of these methods
quantification 1 41 42
18 29 Prior work has also looked at conditioning on different information, such as many previous timesteps [5] (23, tasks inferred by inverse RL [1O), and other task information (6, 21]. While these methods can be directly applied the offline RL setting, some prior works combine these methods with iterative data collection [88 (38, which is not permitted in the typical offline RL set- ting. Kumar al. (26) study reward-conditioned policies and present experiments in the offline RL setting. However, in contrast with our work, the results from Kumar et al. (26] suggest that good performance cannot be achieved through RvS learning but only through additional advantage- weighting [32] [34]. More recent work [5] presents a conditional imitation learning method for the offline RL setting that avoids advantage weighting by introducing a higher-caparte
goal-conditioned behavior cloning [8]
and reward-conditioned behavior cloning [26] 35 38
autoregressive Transformer policy. In contrast to these prior works, show that simple conditioning with standard feedforward networks can attain state-of-ther the-th---rrttt results. While these results do not require advantage weighting high-capacity sequence models, they do require careful tuning of the policy capacity.
2

---

Published as a conference paper at ICLR 2022
3 REINFORCEMENT LEARNING VIA SUPERVISED LEARNING
In this section, we describe a formulation of Reinforcement Learning via Supervised Learning. We do not propose a new method but rather place many existing methods under a common framework. After this, we will investigate what design decisions are important to make such methods work well.
To formulate RvS methods a general way, we assume an agent interacts with a Markov decision process with states 8t, actions de, initial state distribution Ps(81). and dynamics p(8t+1 8t, st,0t): The agent chooses actions using a policy (r | 8t) parameterized by 0. We assume episodes have fixed length !/ and trajectories might be described using multiple outcomes, we use f(w r) to denote the distribution over outcomes that occur trajectory. We study two types of outcomes. The first type of outcome is state (w E S) that the agent visits in the future: f(w TT.H) Unif($:411 St+2, $H). We refer to RvS methods that learn goal-conditioned policies as RvS-G. The second type of outcome is the average return (w IR) achieved over some future number of time steps: f(w Te.1r) =
($1, (1, T', $2, 02, r2,.... to denote trajectory of experience. As some
1 (w 7 E" 7 a))). (Note that we found it important to use the max episode length as constant H in the denominator for all trajectories, treating early terminations as if they received O reward after they terminated.) We refer to RvS methods that learn reward-conditioned policies other information, such the parameters of a reward function inferred via inverse RL [10) or the parameters of a task, such as turning left or right [6]. We focus on RvS methods applied the offline RL setting, These methods take as input dataset of experience, D = {r} and find the outcome-conditoon
RvS-R. It is also possible to condition
policy xo (at | St, w)) that optimizes
For all timesteps that trajectory:
For all achieved outcomes:
trajectories:
max Ø
Σ TED
Σ 15tg/T
E (wwlmw) Dog To(a ss,w)]
While the basic formulation of this conditional policy is simple, in- stantiating RvS methods that at- tain excellent results has proven challenging [26] 38). In the re- mainder this paper, we present an empirical analysis of the de- sign choices RvS methods. We aim to understand which design choices are needed to make RyS methods perform well diverse benchmark tasks, including how two different choices for (goals and rewards) compare in practice.
Algorithm 1 RyS-Learning
1: Input: Dataset of trajectories, D = {T}
Initialize policy ne (a s,,)).
3: while not converged do Randomly sample trajectories: T ~ D.
Sample time index for each trajetory, [1, H], and sample a corresponding outcome: w ~ f(w T=.)).
Compute loss: L(0)= E(rra log To(a st,w) Update policy parameters: 0 + 0 + V)(()
8: end while return Conditional policy Ta(a 18, w,
TASKS AND DATASETS
To provide a comprehensive empirical study of RyS methods, we selected a broad range of tasks; the state- base- tasks prior work on various types of RvS or conditional imitation methods typically include only subset of the domains we consider 18 26 Our goal will be to include: (/) domains and datasets that are appropriate for different types of conditioning, including goals and rewards; (2) datasets that include different proportions of near-optimal data, ranging from near-expert datasets to ones with very few or no optimal trajectories at all; (3) datasets that run the gamut terms of task dimensionality; (4) domains that have been studied by prior value- based offline RL methods.
3

---

Published as a conference paper at ICLR 2022
GCSL is suite of goal-conditioned environments used by Ghosh et al. 18 to evaluate GCSL, a goal goal-conditioned RyS method with online data collection. We adapt these tasks for offine RL by using random policy to collect training data, which results in suboptimal trajectories. The tasks include 2D navigation with obstacles (FourRooms, Eysenbach et al. [)); two end-effector controlled Sawyer robotic arm tasks (Door and Pusher, r, Nair et al. [3T];; the Lunar Lander video game, which requires controlling thrusters to land a simulated Lunar Excursion Module (Lander); and manipulation task that requires rotating valve with a robotic claw (Claw, Ahn et al. (2).
Gym Locomotion v2 tasks consist of the Hal Salffheetah, Hopper, and Walker datasets from the D4RL offline RL benchmark [12]. We use the random, medium, medí um-expert, and medium conduum-rpplay datasets in our evaluations, which consist of (a mixture of different) policies with varying levels of optimality. This requires learning from mixed and suboptimal data, and we will see that TD learning methods perform comparatively well on the random data.
Franka Kitchen vO a 9-DoF robotic manipulation task paired with datasets of human demonstrations. This task orig- inates from Gupta et al. [19] and was formalized as an of- fline RL task in D4RL [2] Solving this task requires com- posing multi-step behaviors (e.g.. open the microwave, then flip switch) from component skills. This task includes three datasets: complete, where all trajectories solve all tasks in sequence; partial, where only subset of trajectories per- form the desired tasks in sequence; and mi xed, which con- tains various subtasks but never all in sequence, requiring gen- eralization and temporal composition.
AntMaze involves controlling an 8-DoF quadruped to navigate to a particular goal state. This benchmark task, from D4RL [12], uses a Man-Markovi demonstrator policy and was intended to test agent's ability to learn lempo- ral compesitionality by combining subtrajectories of differ- ent demonstrations. There three mazes of increasing size:
umaze, medium, and large, and there are two datasets types: di verse and play. The D4RL paper [12 claims that "diverse" data navigates from random start locations to random goal locations while "play" navigates between hand-picked start-goal pairs. Prior work
has proposed that, for tasks requiring temporal compositionality. dynamic programming with value- based methods should be particularly important [12. However, we find in AntMaze that RyS out- performs all the dynamic programming methods we consider.
When applying RvS learning methods, one must choose a type of outcome, such as rewards or goals. In Kitchen and AntMaze, the reward function is implemented in terms of goals. The kitchen reward for completing multiple subtasks, so we condition RvS-G on state in which all the subtasks have been achieved. For AntMaze, we condition RvS-G on the goal location in the maze. This choice makes the additional assumption that we know how the reward function is defined, rather than just receiving samples from the reward function. Because GCSL is inherently multitask, goal-conditioned setup, we omit reward conditioning and only report RvS-G. Conversely, because there is no clear way to define performance in Gym w.r.t. goal, we omit goal conditioning and only report RvS-R. For all tasks, we report scores in the range [0, 100). For GCSL, the score corresponds to the percentage of episodes in which the agent achieves the goal. In the other environments, we follow Fu et al. 12 and normalize the score according to 100. return random (Images from 12 18] .))
expert ramdom
ARCHITECTURE. CAPACITY, AND REGULARIZATION
We instantiate the RvS framework in Section 3 using a simple feedforward, fully-connected neural
network. We condition either the reward-to to-ro-to-ggo go 26 38
goal state [18 19 29 by merely
concatenating with the input state (see Figure D. Our aim is to identify the essential components of
such methods; eschew more complex architectures such as Transformer sequence models [5] On
4

---

Published as a conference paper at ICLR 2022
the tasks consider, we show that simple implementation achieves performance competitive with prior work that uses these more complex components. However, seemingly minor choices in the architecture do matter. These architectural choices include tuning capacity and regularization, suggesting that overfitting and underfitting are major challenges for RvS. Additionally, the choice of what condition on (goals rewards) also has important and domain-specifie repereussions.
Capacity and regularization. In Figure we compare different architecture sizes and regular- ization settings on a single task from each task suite. The best- best-performing architectures are gener- ally larger than the architectures used in standard online RL (28) and imitation learning [I] [22]. This conclusion is intuitive, since RvS policies must represent both the optimal policy and policies for other conditioning values (e.g,, other goals or suboptimal rewards). This result may help ex- plain the contradictory conclusions from prior work regarding the importance of more complex design decisions
26 The results in Figure 3 also highlight the importance of regulariza- tion: while dropout sometimes has no effect (pusher), it improves performance in some tasks (kitchen-completee and worsens performance in other tasks (hopper- mepie um-expert and antmaze mee-mmedinm um umpaa))).
What can we say about these results on network capacity and regularization? The hepper-med um-expert dataset is large, and it contains only two modes: one from medium- quality policy, and one from an expert policy. So, it is not surprising that this data doesn't need regularization. In contrast, solving many different subtasks. Prior work has found that human demonstrations are harder to generally benefits from dropout. In antmaze medi um -play, relatively few trajectories are successful, so are surprised see that the best-performing hyperparameter setting does not use regularization. In pushex. regularization neither helps nor hurts, underscoring that the (lack of) impact of regu- larization depends on the task at hand. Overall, we speculate that regularization balances tension between two competing demands on the policy. First, even if the behavior policy is simple, the con- ditional action distribution may be complex. Second, the policy must sufficiently well regularized to generalize well to new goals or conditioning variables.
is small dataset of human demonstrations
fit than artificial demonstrations 30. and this may explain why kitchen-compe
Output distributions. The form of the policy's s output distribution also influences the policy's 's ca- pacity. While unimodal Gaussians are a common choice with continuous action spaces [14 26 (7], categorical distribution over a discretized action space allows the policy to represent more com- plex, multi-modal distributions [8] 29). To study the importance of the policy output distribution, use the low-dimensional GCSL environments. where
discretize the full action space. Figure 4 (left) shows that across the GCSL suite, categorical distributions either match or outper- form Gaussian distributions. Similar to the results from the previous section, these results highlight how increasing the policy's ' model capacity can improve performance. Again, our finding stands in contrast to standard RL methods, which work well with unimodal Gaussians 220 36 We conjecture
that upweighting better data, as done by previous methods (26), may make easier for low-capacity policies fit the data. While both upweighting data and increasing model capacity allow the policy to better fit the data, increasing model capacity may be simpler approach.
pusher Dropout (p 0) Dropout (p - 0.1)
hopper-medium-expert
kitchen-complete
antmaze-meddumm-play
100
75
25
0-
128
256
512
1024
128
256
512
1024
128
256
512
1024
128
256
512
1024
Layer Width
Layer Width
Layer Width
Layer
Figure 3: Capacity and regularization: We vary capacity (via network width) and regulariza- tion (via dropout) on one eavironment from each task suite. Larger networks perform better on Sopper-mediin-ezperr
and kitchen-comee cechen-complees lete, suggesting the importance of high-capacity pol- icy networks. However. dropout also usually boosts performance in kit sitchen-To coanenncomppee, suggesting that a combination of high-capacity policies with effective regularization is important for achieving good results.
5

---

Published as a conference paper at ICLR 2022
Success
0.6
40
Environment
kitchen-comples coepteee
0.4
20
Policy Distribution
kitchen-mixed
0.2
Gaussian
0
kitchen-partial
Discretized
0.0
Your_rooms claw pusher Environment
door
6
Validation Log Likelihood
Figure 4: (Left) In the GCSL suite, a categorical distribution in discretized action space generally outperforms unimodal Gaussian distribution in continuous action space. This fits the broad pattern we observe that higher-capacity RvS models perform better. (Right) In D4RL Kitchen, validation loss is only loosely correlated with final performance.
Tuning with validation loss. Hyperparameter optimization is especially important in the offline RL setting, evaluating different hyperparameter configurations requires interacting with the en- vironment [3). Since RvS methods reduce the RI. problem to a supervised learning problem, we hypothesize that the validation loss of this supervised learning problem might be an effective metric for tuning important hyperparameters, such as policy capacity and regularization. To test this, we train models on 80% of each Franka Kitchen dataset with two different hyperparameter settings: regularized setting with dropout = 0.1 and small batch size 256; and an unregularized setting without dropout and large batch size of 16, 384. We show results in Figure 4 (right). For all three datasets, validation set eπrо does correlate with performance, but the strength of this correlation varies significantly. In general, the validation loss does not provide a reliable approach for hyperpa- rameter tuning. Fully automated tuning of hyperparameters remains an open question [33 33
recipe for practitioners. We suggest the following process for online hyperparameter tun- ing: incrementally increase network width until performance saturates, and then try adding a bit of dropout regularization (c.g., dropout p = 0.1). If validation loss indicates that under/overfitting is an issue, one can also try increasing/decreasing the batch size, respectively.
Table in the Appendix summarizes the hyperparameters that we found to work best for each task.
COMPARING RvS WITH PRIOR OFFLINE RL METHODS
Having identified key design decisions for implementing RvS learning, we now compare RvS to prior offline RL methods. We evaluate both goal-conditioned behavioral cloning (RvS-G) and reward-conditioned behavioral cloning (RvS-R) using the domains in Section 4 We then discuss what these results imply about the performance, design parameters, and limitations of RvS methods.
Baselines and prior methods. On the D4RL benchmarks, we compare RvS learning to (i) value- based methods and (ii) prior supervised learning methods and behavioral cloning baselines. For (i), we include CQL (27] as well as the more recent TD3+BC [4] and Onestep RL 4 methods. TD3+BC and Onestep RL both involve elements of behavior cloning. We use CQL-p denote the CQL numbers published in [12] and CQL-r to denote our best attempt to replicate these results using open-source code and hyperparameters from the CQL authors. For (ii), we include behavioral cloning (BC), which does not perform any conditioning: Filtered ("Filt.") BC, a baseline appearing in (S] which performs BC after filtering for the trajectories with highest cumulative reward; and Decision Transformer (DT) [5], which conditions on rewards and uses large Transformer sequence model. Both BC and Filtered BC are our own optimized implementations 255. tuned thoroughly in similar manner as RvS-R and RvS-G. In Kitchen and Gym locomotion, Filtered BC clones the top 10% trajectories based on cumulative reward. In AntMaze, Filtered BC clones all successful trajectories (those with reward of I) and ignores the failing trajectories (those with reward of O). On the GCSL benchmarks, we compare to GCSL using numbers reported by the authors [18]. This gives the GCSL baseline the advantage of online data, whereas RyS uses only offline data. We do not run RvS-G G on the Gym locomotion tasks, which are typically expressed as cowaardmaxximzzaiion tasks, not goal-reaching tasks. For additional details about the baselines, Appendix A

---

Table 1: Overall performance. RvS conditioned on goal states (RvS-G) is state-of-the-art in the AntMaze, Kitchen, and GCSL suites. RvS conditioned on rewards (RvS-R) performs worse at these tasks, highlighting the importance of the conditioning variable. In Gym, RvS-R matches the performance of Decision Transformer while only using an MLP. *For the sake of completeness, we include CQL-p, TD3+BC, and Onestep numbers that use AntMaze version vO. See Appendix A for more discussion of this and other experiment details.
Published as conference paper at ICLR 2022
Suite
Eavironment
BC
Filt
BC
TD3+BC
Onestep
CQL-t
CQL-p
Ryš-R
RyS-G
GCSL
umazze-v?
54.6
60.0
78.6
64.3
44.8
74.0
65.6
64.4
65.4
diverse-v2
45.6
46.5
71.4
60.7
23.4
84,0
51.2
70.1
60.9
medium-play-v22
0.0
42.1
10.6
0.3
0.0
61.2
1.0
4.5
58.1
D4RL AntMaze
medium-divene--2
53.7
67.3
large play-v2 large
0.0
28.0
0.2
0.0
0.0
15.8
0.0
3.5
32.4
antmze v2 average
16.7
41.4
27.3*
20.9*
11.4
50.6
19.8
25.6
53.5
wallchectal-random--V
2.0
11.0
18.6
hopger-random-v22
8.5
walker2d-aadoom-22
1.7
1.7
1.6
6.1
2.5
2.0
-0.2
randomn-22 average
halfcheetah medium-replaye
36.6
40.6
44,6
42.4
47.3
36.6
38.0
hopper-mediun-celay-.
18.1
71.0
82.7
73.5
26.0
62.5
81.8
71.6
86.1
66.6
60.6
mediumn replay average
26.9
59.7
62.4
61.7
77.1
62.0
37.4
D4RL Gym
multchectal-mcdion-e
42.6
48.3
55.6
49.1
42.6
41.6
hupper-mediune
52.9
56.9
59.3
83.3
64.6
67.6
60.2
walker2d medium
71.7
medium-V v2 average
56.9
58.1
63.3
74.8
65.5
61.4
57.8
halfchectah-modiinee mcdiini-ncimm-------22222
55.2
93.5
85.8
86.8
92.2
hopper-nn perper-medium--eee -xdue---e-----
52.5
110.9
98.0
102.1
102.0
107.6
101.7
walker2d medium expert-v2
107.5
109.0
110.1
110.9
109.5
108.1
106.0
medium-cxpeet--2 average
gym-v2 average
39.6
56.2
58.2
61.4
63.0
57.0
54.7
comphe--ommpete-10
11.8
43.8
50.2
kitchen-nixcd-v0
51.5
40.0
24.2
51.0
1.1
60.3
DARL Kitchen
kitchen kitchen-partial- partial-V
38.0
20.8
49.8
$1.4
kitchen-vi average
51.5
36.7
18.9
48.2
1.0
54,0
claw
14.8
28.0
GCSL
lour rooms
63.0
81.0
lonar
54.7
70.0
pusher
81,8
83.0
gest average
62,0
58.0

---

Published as a conference paper at ICLR 2022
Overall performance and comparison to prior methods. Sableessoww the results of this com- parison, leading to several conclusions about how our RvS implementations compare to prior value- based and supervised methods. In all suites, either RvS-G or RvS-R attains results that are compa- rable to the best prior method. The finding that conditional imitation with standard fully connected networks attains competitive results stands in contrast to prior work, which emphasizes advantage weighting and Transformer sequence models [S] (6). As we discussed in Section 5 good perfor- mance requires careful combination of high capacity and regularization, and it may simply be that this seemingly contradictory combination of components was not adequately explored in prior work.
Subtrajectory stitching. The mi xed dataset in Kitchen and all of the AntMaze tasks consist of suboptimal trajectories. Attaining expert performance requires recombining parts of these trajecto- ries. Such stitching is usually viewed as a key benefit of dynamic programming methods, so is surprising that RvS-G performs as well prior methods based on dynamic programming (TD3+BC and CQL). In these same tasks, RvS-G outperforms RvS-R. We speculate that the spatial information encoded by goal information helps RvS-G generalize. BC works well in the Kitchen environment but poorly in the AntMaze tasks, perhaps becaue the Kitchen task contains experience that is easier imitate. As a direction for future work, we propose studying if conditioning on goals can help provide compositionality in space just as the Bellman backup provides compositionality in time.
Reaching arbitrary goals. An important capability RvS-G, com- pared with more standard offline RL methods. is that it learns pol- icy that can reach many goals. To test how well RvS-G can learn to reach arbitrary goals from random offline data, compare GCSL, an RvS learning method that performs iterative online data collec- tion [18]. We the same tasks used by GCSL. Each of the tasks in the GCSL suite requires that the agent reach goal drawn uniformly random from the set of possible goals. The results shown in Fig- ure indicatee indicate that RvS-G can successfully reach many different goals entirely from randomly collected offline data. Although it may still be that iterative online collection, proposed in prior work [18 (38, may be helpful in some domains, in this case simple offline training appears to suffice.
RvS-G
GCSL-
25
50 Return
Figure GCSL results.
Random datasets. While on average RvS-R is competitive with prior methods on the Gym Loco- motion tasks, TD learning performs better the random datasets; indeed, CQL performs especially well on hal selfcheetaiin randorr (Table D. This suggests that RvS methods may more generally perform worse on random datasets in comparison to TD learning methods.
Analysis of reward conditioning. Next, we analyze what RvS-R actually learns and how it uses the conditioning variable. First, ueeeperrt task
y=
use the analyze the behavior of RvS-R for target rewards. The wa walke dataset contains two modes ("medium" and "expert", as shown in Figure 6 Conditioning the network on range of reward targets (X- axis), F shows that the policy's 's achieved return (Y-axis) corresponds only to the two modes in the dataset. The policy cannot in- terpolate between the two modes. In effect, appears that RvS is mimicking subset of the demonstrations, but doing so without explicit filtering (as is done in Filtered BC).
um-expe
Achieved Returns
50-
frequency of returns in dataset
100
Target Returns
Figure 6: A failure to interpolate.
This analysis highlights the importance of the conditioning variable. It is not obvious priori that a reward target of 100 will lead to return of 70, whereas reward target of 1 110 will lead to return of 105. Following prior work (5). we tune the reward target for every task (see Table Choosing this hyperparameter in truly offline setting is important problem for future work.
8

---

Published as a conference paper at ICLR 2022
7 DISCUSSION AND FUTURE WORK
We presented an empirical study of offline RL via supervised learning (RvS) methods, which solve RL problems via conditional imitation learning. While prior work is divided on which elements of RyS are crucial for good performance, even how well such methods actually work, our results provide three conclusions. First, if capacity and regularization are chosen correctly, RyS methods with simple fully connected architectures can match or outperform the best prior methods. Second, properly choosing the conditioning variable (c.g.. goal or reward) is critical to the performance of RvS methods. Third, RvS methods remain competitive in tasks such Franka Kitchen and AntMaze, where there is little optimal data. These conclusions suggest multiple directions for future work. Because policy capacity and regularization are critical to good performance, is there method for automatically tuning these hyperparameters? Validation error is not a reliable metric. Also, the choice of conditioning variable is important: how might we automate this choice? Addressing these questions may increase the performance and applicability of RyS learning methods.
ACKNOWLEDGMENTS
This work was supported in part by the DOE CSGF under grant number DE-SC0020347.
We thank Michael Janner for discussions about architectures, and we thank Aviral Kumar and Justin Fu for discussions about datasets. We thank Sam Toyer, Micah Carroll, and anonymous reviewers for giving feedback on initial drafts of this work.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[2] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. In Conference on Robot Learning (CoRL., 2019.
[3] Arthur Argenson and Gabriel Dulac-Amold. ference on Learning Representations, 2021. URL http::///ppeneeeeeeeeeeeeeeeeeee
Model-based offline planning.
International Con-
OMN G5xad4
[4] David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off- policy evaluation. In Thirty-Fiiee Conference on Neural Information Processing Systems, 2021. URL https://openrevit
net. forummidd-10s www.11108cnn... Ct
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence mod- cling. In Thirty-Fift Fifth Conference on Neural Information Processing Systems, 2021. URL https: //openrevi ew, net/ wtt.:uumi...aa7mm4a555m
[6] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to- end driving via conditional imitation leaming. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4693-4700. IEEE, 2018.
[7] Stefan Depeweg, José Miguel Hernández-Lohatt Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. In International Conference Learning Representattions (ICLR 2017), 2017.
[8] Yiming Ding, Carlos Florensa, Pieter Abbcel, and Mariano Phiclipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems, pp. 15324-15335, 2019.
[9] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search the replay butfer: Bridging planning and reinforcement learning. In Advances Neural Information Processing Systems, pp. 15246 15257, 2019.
[10] Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse RI.: hindsight inference for policy improvement. In Advances in Neural Information Processing Systems, 2020.

---

Published as a conference paper at ICLR 2022
[11] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learning Representations, 2018. URL https:
//operreieew wetp/oooun...
[12] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4RL.: Datasets for deep data-driven reinforcement learning. arXiv preprint urXiv:2004.072) 07219, 2020.
[13] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang, Michael Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off policy evaluation. In International Conference on Learning Representa- tions, 2021. URL /tttps://opennevvvvie revi net. ? d kwSeG*etvv-
[14] Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https::/oppenreeiittt net/ forum?iid=332u7deee.
[15] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
[16] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07746 2021.
[17] Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal condi- tioned policies. In International Conference on Learning Representations, 2018
[18] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysen- bach, and Sergey Levine. Learning to reach goals via iterated supervised lcarnigg. In International Conference on Learning Representations, 2021. URL htttps://openrevieew net peet/Forum?ic id=
6yL
[19] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, pp. 1025-1037,11 2020.
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870, PMLR, 2018.
[21] Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Prze- myslaw Mazur. Sean Micklethwaite, Nicolas Griffiths, Amar Shah, et al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pp. 251-257. IEEF, 2020.
[22] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural informa- tion processing systems, 29:4565-4573, 2016.
[23] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[24] Rahul Kidambi. Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- Based Offline Reinforcement Learning. In H. Larochelle, M. Ranzato, R. Hadsell, F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 21810-21823. Cur- ran Associates, Inc., 2020. URL httpthttps///cotoeoo ing. neur
neur httpp/ccppppppe///////apa//
[25] Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021. URL
ht https://githhne com/ http:///ikottttoo/oooaoe
[26] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioeed policies. arXiv: 13465, 2019.
arXiv preprint
[27] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin Advances
Advances in Neural Information Processing Systems, 2020.
[28] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Con- ference Learning Representations, 2016.
10

---

Published as a conference paper at ICLR 2022
[29] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar. Jonathan Tompson, Sergey Levine, and Pierre Sermanct. Learning latent plans from play. Conference on Robot Learning, pp. 1113-1132. PMLR, 2020.
[30] Ajay Mandlekar. Danfei Xu. Josiah Wong, Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline hu- man demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning, 2021. URL https://openreviiet net wttp://foommmaaisssttrttte
[31] Ashvin V Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31: 9191-9200, 2018.
[32] Gerhard Neumann, Jan Peters, et al. Fitted Q-iteration by advantage weighted regression. In Advances in Neural Information Processing Systems 2I-Proceedings of the 2008 Conference, pp. 1177-1184, 2009.
[33] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint 200000000009555 2020.
[34] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weiighte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:19100.00177, 2019.
[35] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-jus map them to actions. arXiv preprint arXiv:1912.02887 2019.
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. arXiv preprint arXiv: 1707,00077 2017.
[37] Jian Shen, Mingcheng Chen, Zhicheng Zhang, Zhengyu Yang, Weinan Zhang, and Yong Yu. Model-beol based offline policy optimization with distribution correcting regularization. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 174-189. Springer, 2021.
[38] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint 1912v: 2019.
[39] Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement lcarning. Engineering Applications of Artificial Intelligence, 104: 104360,,,, 2021.
[40] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Repre- sentations, 2020. URL
net 1orumm21d-rkxhh222F1
[41] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforeement learning, In Marina Mcila and Tong Zhang (eds.), Proceedingy of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11319-11328. PMLR, 18-24 Jul 2021. URL https: wwwli. mttm
//pzoceedings www.ong..........m...m......mmmm mlr wttpeeam........seeeeeeeeee
[42] Tianhe Yu, Garrett Thomas, I Lantao Yu, Stefano Ermon, James 7ou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based Offline Policy Optimization. Advances in Neural Information Processing Systems, 33:14129- 1114449991114,,,1111111111114444 2020.
11

---

Published as a conference paper at ICLR 2022
Table Hyperparameters. Architecture and design parameters that we found to work best each domain. We define an epoch length include all start-goal pairs in GCSL, i.e., be |D] (2)) In
D4RL, we set all epoch lengths at 2000 (50) 2450000.
Hyperparameter
Value
Environment
Hidden layers
All
Layer width Nonlinearity
1024
All
ReLU
All
Learning rate Epochs
le-3
All
10
GCSL
Kitchen
2000
Gym
Gradient steps
20000
AntMaze
Batch size
256
GCSL,, Kitchen
16384
Gym, AntMare
Dropout
GCSL, Kitchen
Gym, AntMaze
Goal state
Given
GCSL
All subtasks completed
Kitchen
y) location
AntMaze
Reward target
110
Gym medium-cxpert, AntMaze, Kitchen
90
Gym (hopper,
walker2d-medium
60
Gym hopper-medium
40
Gym random, halfcheetah- (mediuu,, medium-replay) GCSL.
Policy output
Discrete categorical Unimodal Gaussian
Kitchen, Gym, AntMaze
A EXPERIMENT DETAILS
In this section, we provide more details about our experiments each environment suite. For every task, we use 5 random training seeds and 200 evaluation rollouts for RvS.
GCSL Following the GCSL protocol [8], we collect offline data for RvS by performing random rollouts in each environment. As in GCSL, we set the environment time limit (i.e., max episode duration) to actions, and the rollout policy uses a discretized action space. We collect 50,000 timesteps of experience in FourRooms, 100,000 timesteps of experience in Door and Lander, and 500,000 timesteps of experience in Pusher and Claw. We take our GCSL baseline numbers from those reported in the GCSL paper at the corresponding number of timesteps for each environ- ment 18
Gym Locomotion We use the v2 versions of the datasets for all algorithms. Because the v2 datasets fix issues with the vO datasets, we omit the published CQL-p numbers, which use vO and are no longer comparable. We instead report our replication of CQL,, denoted CQL-r, using v2. We report the DT (27). TD3+BC [[4,, and Onestep [4] numbers from the corresponding papers. We use the reverse KL regularization variant of Onestep because this variant attains the best performance. Because DT did not run experiments with the random datasets, we run them ourselves using the default hyperparameters the
codebase released by the DT authors. We omit numbers for RvS-G because there is no clear way to define the Gym tasks in terms of goals.
Franka Kitchen We use the v0 versions of the datasets and omit TD3+BC, Onestep, and DT num- bers, which are not provided by the authors. We include the previously published CQL-p numbers from D4RL. 12 as well as CQL.-, our own replication of CQI., and BC and Filtered BC, our own baselines.
AntMaze We use the v2 versions of the datasets for RvS CQL-r, BC, Filtered BC, and DT. Be- cause the DT authors do not test in AntMaze, we ran these experiments ourselves using the default hyperparameters in the open- open-source codebase released by the DT authors. We include previously published numbers for CQL-p [12 and TD3+BC [14], as well as Onestep numbers received via email correspondence with the Onestep authors. These numbers for CQL-p. TD3+BC, and Onestep the vō versions of the datasets, which the same the v2 versions except that the v0 timeout
12

---

Published as a conference paper at ICLR 2022
0.08
Dropout
1.6
0.07
0.1
1.4
Width of Hidden Layers
0.06
1.2
1024
Validation MSE
1.0
0.05
Retum
0.8
0.04
0.6
0.03
0.4
0.02
0.2
0.0
0.01
5000
10000
15000
20000
25000
Gradient Steps
Figure Validation mean-squared-error and evaluation return versus training gradient steps in Frank Kitchen. Each line includes 5 random seeds across each of the three datasets: complete, partial, and mi med. Surprisingly, we find that although each hyperparameter setting has nearly identical validation loss, the evaluation return varies by as much as 1.4x. This indicates that dropout and network capacity are having important effects on performance beyond validation loss.
flags indicating the ends of episodes contain errors. Because initial experiments suggest that TD learning methods are fairly robust to errors in the timcout flags, we include these numbers for the sake of completeness.
Codebases Here are links to all of the codebases we use in our experiments:
RvS: /ts.tt/ss//gttabbccccc////ttmmmr
2. h0tp://teee//ttttioc///////oo/oa
3. http:://ggttttccco/////////eeees
4. BC and Filtered BC: Ccchac/pss/ggggbbooooooooostttt/
B THE IMPACT OF MODEL CAPACITY AND REGULARIZATION
As we find in Section that model capacity and regularization are important for performance in seemingly contradictory ways, we hypothesize that this can be explained by validation loss. So, for 5 random seeds in each of the Franka Kitchen datasets (completes partial, and mi xec), measure validation loss and return throughout training for three different settings of policy hyper- parameters: network width 1024 with dropout p 0.1, network width 1024 with dropout p = (no dropout), and network width 256 with dropout O (no dropout). We expect that a larger network width leads to better validation loss, explaining its better return. We expect adding regularization also has this pattern of better validation loss and therefore better return.
Figure 7 we plot the results: validation loss nearly identical across all of the hyperparameter settings, but return vary by as much as 1.4x between different hyperparameter settings. We also observe that evaluation return steadily increases even after validation loss has mostly leveled off. This is surprising and refutes our hypothesis; furthermore, it provides the insight that model capacity and dropout are having effects beyond just the validation loss. A similar phenomenon occurs in Neural Machine Translation, where cross entropy loss can decline while the quality of the translation
13

---

Published as a conference paper at ICLR 2022
Conditoning Strategies D4RL medium expert
Commanding Subtasks kichen-complete-v
Conditioning Optimied
Checkpant Welicaioon
120
Langth
Reward Goal
Reward Scalar
50
Takum
40
Awibardd
hopper
rondem
lettle snitelade cabinettyname
Erstrummnnt
Oiaal
(a) D4RL Gym
(b) k toche-ccomp ete
Figure 8: (a) Different strategies for choosing goals D4RL Gym. (b) Commanding the policy to complete different subtasks in ki kitchhe-ccomplete
(measured by BLEU score) remains constant [[], and in contrastive representation learning, where lower capacity models have worse losses on the test set but still learn better representations for downstream tasks [40]. It is important open problem for the field to understand why architecture decisions can impact performance beyond validation loss.
C COMPARISON OF CONDITIONING STRATEGIES
To further study how different goal and reward conditioning strategies compare, in Figure Sa we show the performance of four strategies: Reward Scalar, Reward Goal, Length Goal, and Opt imi zed Goal. In Reward Scalar, we condition RvS-R on return that is various frac- tions of expert performance. In Reward Goal, condition RvS-G states from randomly selected timesteps near the end the highest-reward demonstration trajectories. tion RyS-G on states from randomly selected timesteps near the end of the longest demonstration trajectories, In Optimized Goal, we do a random search over 200 Length Goals and condition RvS- on the highest performing one. (Note that Optimized Goal requires access to the environment, so it is not strictly offline RL; we are including it here an oracle experiment.) We see that across all the goal selection strategies we consider, the performance of RvS-G cannot match the performance of RvS-R, since the Gym tasks are not inherently goal-conditioned.
Length Goal, we condi-
we compare the performance of RvS-G in kitchen-complett when commanded task requires the agent to open a mi- crowave, move a kettle, flip a light switch, and slide open a cabinet. For "all," we command policy to reach state where all the subtasks are achieved. For "random," " we randomly select one of the tasks and command the policy to reach a state where that task is complete. For "microwave," "kettle," "light switch," and "slide cabinet," subtask is achieved. For "dynamic," rather than leaving the goal fixed throughout the trajectory, we initially command the agent to complete the first subtask. Then, we update the command to include the second subtask (and so on for all remaining subtasks). As expected, we see that the oracle dynamic strategy, as well conditioning on all subtasks completed, perform the best. Surprisingly, commanding the microwave to be opened outperforms commanding the other individual subtasks. We speculate that the microwave subtask might appear most often in the demonstrations.
In Figure 8b
to reach different goal states. The sitchen-complee
command the agent to reach state where that one
the first subtask is complete,
14


Reviews:
[{"review_id": "zYfynOsMbj", "reviewer": "Reviewer_oQXj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\\\nThe paper provides a broad overview and makes a categorization.\nContributions to the discussion are provided, that might influence future research.\n\nWeaknesses\\\nThe finding that regularization of the learned policy is particularly significant suffers somewhat from the lack of a recipe to perform this regularization offline.\n\nOther comments\\\nI find the sentence \"For example, any experience is optimal for learning to reach the final state in that experience.\" too ambiguous.\n\nThe Introduction talks about a \"reward parameter\", I think at this point it should be made clearer what is meant by this.\n\n\"cloning(Codevilla\" -> \"cloning (Codevilla\"\n\nIn the Related Work section, prior work is classified by whether it uses a value function, uses a dynamics model, or uses uncertainty quantification. It remains unclear that (Yu et al., 2020), (Shen et al., 2021) and (Kidambi et al., 2020) use both the dynamics model and a value function and only (Argenson & Dulac-Arnold, 2020) uses the dynamics model without a value function. At this point, it is helpful to also mention [1] and [2] as other examples of using dynamics models without using the value function.\n\n[1] S. Depeweg et al., Learning and policy search in stochastic dynamical systems with Bayesian neural networks, ICLR 2017\n\n[2] P. Swazinna et al., Overcoming model bias for robust offline deep reinforcement learning, EAAI 2021\n\n\n\"actions s_t\" -> \"actions a_t\"\n\nThe axis labels in figures 1-5 are very small. It would be ideal if it had the same size as the caption font.\n\nThe text \" RvS methods can often attain results that are comparable favorably to the best prior methods,\" should be corrected.\n\nPlease check the bibliography for accidental lower case letters, like „rl“, „q-learning“\n\n", "summary_of_the_paper": "The paper investigates different variants of behavior cloning. The methods investigated aim at achieving better policies for use in offline RL through supervised learning than the average behavior contained in the data. The methods studied are categorized and examined against three to four benchmark problems.", "summary_of_the_review": "The paper gives a good overview of different variants of behavior cloning. The contributions to the discussion may be helpful for future research. However, my feeling is that there is little reliable insight provided. Therefore, I think it is also possible that the benefit is small.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jLfFlbkCZTZ", "reviewer": "Reviewer_hoyq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== post author response: I have increased my score, but, I retain concerns about comparisons with more expressive classes of approaches such as the decision transformer.\n\n1. While the paper presents interesting insights, I am not sure how much these findings can be generalized particularly in offline RL for locomotion tasks, where, typical benchmarks consider all 4 types of data collection strategies (random, medium, medium-replay, medium-expert) -- this paper considers only the latter two variants for their analysis. The other two variants are highly challenging in the offline RL context and require leveraging on elements of pessimism/conservatism to guide policy learning. The paper requires to present studies on these tasks to have a more holistic view of applying RvS methods to offline RL.\n\n2. The comparison of this paper with the Decision Transformer is not well accounted for. The Decision Transformer work of Chen et al. (2021) can be viewed as a general framework used for synthesizing a variety of behaviors through conditioning on the rewards (or other events) during inference time. Such a perspective isn't quite considered by this work, and I am not sure how straightforward it is to programmatically generate a variety of behaviors without recourse to density modeling and sequence modeling. To present a fair comparison to works such as Decision Transformer, the paper needs to present results where a learnt policy can produce behaviors based on appropriate conditioning at inference time (and not just restricted to training time).", "summary_of_the_paper": "This paper considers design choices involved in offline RL approaches that consider a reduction to weighted/conditional behavior cloning, a class of approaches referred to as Reinforcement Learning via Supervised Learning (RvS). The paper studies various issues including expressivity of policy architecture, regularization, choice of conditioning variables (e.g. based on goals or rewards). \n\nAt a high level, the takeaways are (a) RvS approaches are successful when using neural networks with appropriate capacity and regularization, (b) with appropriate conditioning variables (goal/reward based conditioning), (c) RvS can obtain policies that exhibit compositional behavior by conditioning on appropriate events.", "summary_of_the_review": "The paper certainly takes a step forward with regards to understanding what components of RvS algorithmic frameworks matter to obtain satisfactory results - but, the paper falls short because (a) the paper doesn't present results with all types of datasets in the D4RL MuJoCo tasks (and this is standard in the offline RL literature), (b) the comparison to decision transformer isn't fully sketched out because the decision transformer framework is more general in that it offers the ability to generate a variety of behaviors at inference time by changing the conditioning variable during inference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VPtzTfzo6P", "reviewer": "Reviewer_dVtg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides empirical studies of offline RL via supervised learning. \n\n## Strengthens\n\nThe biggest findings are two aspects:\n\n* The authors find more complex design choices, such as the large sequence models and value-based weighting schemes used in some prior works, are generally not necessary, which is important in my opinion; especially, obtaining good performance using simple feedforward neural network is interesting;\n* The empirical results show that carefully designed RvS methods can attain results that match or exceed the best prior methods across a range of different offline RL benchmarks, e.g. when goal or reward is carefully chosen.\n\n## Weakness\n\nSome of the expositions are not very clear: \n\n1. The explanation of the objective (1) is not clear, for example, when $f$ uses RCBC, is $|f(\\tau_{t+1}:H)|$ simply $1$? Also, why this objective should be used remain elusive.\n\n2. Other than the goal and reward aspects, what are the other aspects that matter?\n\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper studies the importance of the design decisions for supervised learning type reinforcement learning algorithms. Through extensive experiments, find that more complex design choices, such as the large sequence models and value-based weighting schemes used in some prior works, are generally not necessary. Our results show that carefully designed RvS methods can attain results that match or exceed the best prior methods across a range of different offline RL benchmarks, including datasets with little or no optimal data.", "summary_of_the_review": "In general, I like the new perspective this paper presented. In particular, the finding \"find that more complex design choices are generally not necessary\" provides a different way of thinking offline RL. Therefore, I prefer to accept at the moment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KAupfA5yBGw", "reviewer": "Reviewer_ehJE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is good to see papers which deep dive into existing methods and study which aspects of the algorithm contribute to improved performance. \n\nMy major concern with the paper is the RvS algorithm is not described completely for me to understand the experiments. It appears that equation (1) in the paper is being used for gradient ascent for policy \\pi parameterized with \\theta. But the accumulation of rewards given by f() is in the denominator, so I do not understand how maximizing this objective leads to a policy that maximizes the rewards. The equation is unlike those given by papers cited (Kumar et al, Srivastava et al). \n\nGiven that this is supervised learning, the basic steps of training the model is also unclear. Essential hyper-parameter values such as learning rate is not provided. Given that there is no good way to determine overfitting as per validation set experiments in the paper, it is not clear how is training stopped in the first place. The number of epochs or steps of gradient update is not provided for any of the methods.\n\nGiven the above observation, it is difficult to believe the results reported in the paper. In addition, the most important insight from the paper is that model capacity and dropout play a key role in final performance. But, given these are contradictory, no further investigation is done to understand why this is the case. Instead, the paper leaves the subject with a conjecture. \n\nThe other aspects of the paper are not entirely surprising, and make sense.", "summary_of_the_paper": "The paper studies the behavior cloning based strategies of offline RL algorithms in different type of environments and reports that performance primarily depends on model size and regularization. The results contradict some of the earlier claims, and the authors conjecture that model size and regularization characteristics can explain past results. The paper also discusses additional insights such as the importance of conditioning and simple validation based evaluation fail to generalize. ", "summary_of_the_review": "The algorithm under study is not described completely. Important hyper-parameters are missing. Because of this, the final results are circumspect.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the behavior cloning based strategies of offline RL algorithms in different type of environments and reports that performance primarily depends on model size and regularization. The results contradict some of the earlier claims, and the authors conjecture that model size and regularization characteristics can explain past results.  

During the review period, the reviewers agreed that the paper has certain merits, and on the other hand, they also raised some concerns, regarding some missing technical details, whether the empirical finding could be trusted, the generalization of the findings to more scenarios, and the comparison with some highly related papers. The authors did a good job in their rebuttal, which removed many of the above concerns (although not all) and convinced the reviewers to raise their scores. As a result, we believe it is fine to accept the paper (although somehow like a weak accept).
Decision: Accept (Poster)