Implicit Offline Reinforcement Learning via Supervised Learning
Authors: Alexandre Piché
Rafael Pardinas
David Vazquez
Igor Mordatch
Igor Mordatch
Christopher Pal
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
Energy Based Model
Offline Reinforcement Learning via Supervised Learning

Abstract: Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset of varied behaviors. It is as simple as supervised learning and Behavior Cloning (BC) but takes advantage of the return information. On BC tasks, implicit models have been shown to match or outperform explicit ones. Despite the benefits of using implicit models to learn robotic skills via BC, Offline RL via Supervised Learning algorithms have been limited to explicit models. We show how implicit models leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets. Furthermore, we show how closely related our implicit methods are to other popular RL via Supervised Learning algorithms.

Under review as a conference paper at ICLR 2023
IMPLICIT OFFLINE REINFORCEMENT LEARNING VIA SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from dataset collected by policies of differ- ent expertise levels. It is as simple as supervised learning and Behavior Cloning (BC), but takes advantage of return information. On datasets collected by policies of similar expertise, implicit BC has been shown to match or outperform explicit BC. Despite the benefits of using implicit models to learn robotic skills via BC, offline RL via Supervised Learning algorithms have been limited to explicit mod- els. We show how implicit models can leverage return information and match outperform explicit algorithms to acquire robotic skills from fixed datasets. Fur- thermore, we show the close relationship between our implicit methods and other popular RL via Supervised Learning algorithms to provide unified framework. Finally, we demonstrate the effectiveness of our method on high-dimension ma- nipulation and locomotion tasks.
INTRODUCTION
Large datasets of varied interactions combined with Offline RL algorithms are a promising direction to learning robotic skills safely (Levine et al.) 2020). A practical and straightforward approach to leveraging large and varied robotics datasets is to convert the RL problem into a supervised learning problem (Emmons et al. 2021 Chen et al.) 2021). RI. via Supervised Learning (RvS) algorithms can be seen as condiioanniiionnl,, -filterd, or -weighted BC. Here, we use RvS to refer to policies where the action distribution is conditioned on a return defined by the user or generated by the agent.
Reinforcement Learning via Supervised Learning algorithms are as simple as BC ones, but since they take advantage of the return information, they can leverage sub-optimal interactions. In order to be effective for datasets collected by policies of different expertise levels, these algorithms must model multi-modal joint distributions, e.g., over the actions and the return. Previous RvS methods either discretized the variables and could use multinomial distributions (Chen et al.) (222)) or ignored the multi-modality of the variables and used simple distributions such as a Gaussian distribution (Ku- mar et al., 2019). This paper shows that implicit models can be excellent for modeling multi-modal distributions without the need discretize the return and action variables. In addition, implicit mod- els have been shown to better extrapolate and model discontinuities than explicit models allowing them to capture complex robotic behaviors (Florence et al. 202T). Despite the advantages of using implicit models, RvS algorithms have been limited to explicit models.
In this work, we bridge the gap between implicit models and RvS, and propose the first implicit RvS (IRvS) algorithm. We show that our IRvS algorithm is closely related to other RvS algorithms but has the advantages of 1) not requiring the user or a second neural network to specify a target return, and 2) not requiring continuous outputs to be discretized to capture multi-modal distributions. Fur- thermore, we demonstrate the superiority of IRvS over RvS to interpolate, model discontinuity, and handle higher-dimension action spaces in an environment with linear dynamics. Furthermore, using the same objective as IRvS, we derive a novel RvS algorithm formulation where the target return is obtained by maximizing the exponential tilt of the return. Finally, show that our method achieves state-of-tt ofetth--rrrttt on the difficult ADROIT and FrankaKitchen manipulation tasks and is competitive with Offline RL and RvS algorithms suite of robotic locomotion environments.

---

Under review as a conference paper at ICLR 2023
predicted actions
predicted actions
à
G
predicted returns
Fo(s, G)
arg ming, Eo(s, a, G) - -7 G
S
G
S
a
G unobserved returns
unobserved actions
predicted returns
states
states
(a) Explicit RvS.
(b) Our Implicit RvS (IRvS) approach.
Figure Comparison of implicit and explicit Reinforcement Learning via Supervised Learning (RvS). Implicit models can be more straightforward and use single neural network. In contrast, explicit models require either a second neural networks (not depicted here) or the user to specify a task-specific target return G.
2 PRELIMINARIES
Offiine Reinforcement Learning. We consider Markov Decision Process (MDP) (S,A, R, R) with states s E S, actions a E A. transition function P(s'|s, a)), a), and reward function r = R(s, a). We aim learn policy π > A maximizing the expected return E,>z [G(r)]. where G(r)
Et= Tr- Contrary "online" RL agents that can interact with the environment, we focus agents that must learn policy exclusively from dataset D of trajectories T = (Su. an, Tu...., SH, a TH) collected under behavior policy μ. For the rest of the paper, work with the tuple (s, a, Gi), where G, Σ Pú the cumulative return starting from state-action (s;,
H
Implicit Models and Energy- -Based Models. We define an implicit model as general func- tion approximator E R.' + where inference can be performed via optimization of ÿ arg.in miny Eo(x, y). In this work, we focus on Energy Based Models (LeCun et al. 2006) (EBMs). EBMs associate sealar energy to configuration of variables yx, The model trained predict lower energy on the observed data than on the unobserved one. We use an EBMs to model the conditional density
poly|x) exp(- exp(-Eo(x,y))
Zo
where Eg is the energy and the normalizing constant Zo = Jy exp(-Eo(ss) y Training the density model be performed using the InfoNCE loss function (Oord et al. 2018 defined
((
Σ
log x, ()/)))))))
Naw 1j=1
exp(- E. (x,, y:)) Nus
(y:
x, {
exp( Ex(x, y:)) ΣSnrr exp( Líg (xi, v!))
where ( Nax a set of negative counter-examples. Negative sampling can be performed by min- imizing y arg miny Eg (x;, y) using stochastic gradient Langevin dynamics (SGLD) (Welling
& Teh 2011 see Algorithm [ for details. Finally, prediction can be done by minimizing the energy
y = arg min Eg (x,
(3)
yey
which can also be done via SGLD.
Implicit Behavior Cloning, Behavior Cloning (Pomerleau) 1988) is a simple and popular way to acquire robotic skills from a dataset of expert demonstrations. The policy is usually modeled via
2

---

Under review as a conference paper at ICLR 2023
explicit model and learned by minimizing the squared error loss Eg, [ fo to clone the behavior policy μ. It is particularly effective when the dataset consists expert trajecto- ries. |Florence et al. (202I) introduced Implicit Behavior Cloning where they leverage EBM to learn policy from offline data. They showed that EBMs could capture multi-modal distribution, in- terpolate, and model discontinuities making them particularly effective continuous control tasks. The EBM is trained to learn the conditional density
Pe (als) = exp( ( a))
(4)
Z
The decision-making is performed by minimizing the energy
à arg min Eg (s. a).
(5)
aEA
which can be minimized using SGLD. The advantages of EBMs allowed IBC to outperform explicit methods by 100% in some environments. While IBC performs particularly well on expert data, does not leverage return information, enabling it to leverage more diverse non-expert data. This is an important limitation, as sub-optimal data might be easier obtain, have better state coverage and result in better representation than expert data.
3 IMPLICIT OFFLINE RL VIA SUPERVISED LEARNING
3.1 METHOD
Inspired by the successes of EBMs on robotic tasks, we introduce Implicit Reinforcement Learning via Supervised Learning (IRvS) to improve the IBC algorithm by modeling the dependencies be- tween the state, action, and return variables. Modeling these dependencies would allow the robotics community implicit methods varied datasets instead of being limited to datasets of expert demonstrations. Specifically, we model the following state conditional distribution
(a Gls) exp(-Ee((s) a, G))
Zo
We now have access to differentiable joint density model po G s) from which we sample action and return pairs using SGLD. Doing so results in actions likely under the training dataset irrespective of their associated returns, which is equivalent the IBC algorithm. Instead, want to bias sampling towards actions with high returns.
To sample action with high return, we instead sample from the exponential tilr (Asmussen & Glynn 2007) in G of the state conditional distribution. Specifically, we use the exponential tilt in G defined
Po (a, G sc; ss; = (a G St) exp(n --(())),,
where k(n)) is the cumulant generating function log Eie" and controls the trade-off between the likeliness of an action-return pair under the model and the magnitude of the return. We note that the exponential tilt density is still differenntable and can be efficiently sampled via SGLD. Specif- ically, the policy (and its associated return) be obtained by maximizing the log density of the exponential tilt
à G arg min Eo a, G) 1G aEA.GEG
where we can safely ignore the normalizing constant s(n). Informally, the above scheme generates high, but plausible returns under the model. For the rest of the paper, we omit k(n) to allege the notation.
3.2 DIDACTIC EXAMPLE
We build simple didactic example to better understand how the aaper-parameter n changes the behavior of the IRvS algorithm. The environment has linear dynamics and goals located at each corner of the room. Each goal yields different reward (-1, 1/2, 1), and the dataset is collected
3

---

Under review as a conference paper at ICLR 2023
i) pal(a, G|s)dG
ii) Spela, G|salda
iii) pe(a, G[se))
1.0
1.0
0.5
De(Gisa)
0.8
return (G)
So
0.6
0.0
0.4-
-0.5
0.2
0.0
1.0
-0.5
0.5
return (G)
action (a)
(a) IRvS 1) 0. In Figure observe that the policy is the as the goal agnostic policy that collected the data. Figure ii), we observe that IRvS can successfully model possible non-discounted returns. In
Figure iii), we observe that IRvS can successfully capture the joint distribution over actions and returns and their dependencies, where arctan
SPola, G\s)exP(3G)dG
ii) palla, G/s))exp33G)d
iii) Pa(a, G|5p)ex)(3G)
1.0
6
0.5
return (G)
50
4
D224
-0.
0
11.0
1.0
-0.5
0.0
0.5
1.0
return (G)
action (a)
(b) IRvS = 5.0. In Figure i), we observe that IRvS policy is now leading towards the largest goal for most of the state space. In Figure ii), observe that the exponential tilt of the return distribution biased towards the highest reward. Figure iii), we observe that the mass concentrated over return and actions near T/4 which are the optimal actions and returns.
Figure 2: Effect 77' This experiment demonstrates how IRvS can leverage sub-optimal demon- strations to achieve near-optimal behaviors by increasing the parameter v) The change caused by increasing 17 -1 from 0 to 3 can be observed the policy, the return prediction, and their joint distribution.
of behavior
under behavior policy that visits each goal with equal probability. The dataset thus consists of 25% of optimal demonstrations and 75% sub-optimal ones. The 2d actions (Az. Ay) are represented as arctan Ay Ax
In Figure 2a we study the behavior of IRvS when 77 0 and is equivalent to IBC. Since the resulting policy is reward-agnostic, we expect the policy learned by IRvS to visit each goal with equal probability which can indeed be observed in the first sub-figure. The IRvS algorithm also successfully models the distribution of non-discounted returns which are possible reach from the starting state. We note that IRvS model the continuous return distribution, while RyS would need the returns to be discretized to capture the multi-modality of the distribution, see Figure 3 Finally, observe that the joint distribution returns and actions is successfully captured in the third figure.
In Figure 2b study the impact of increasing n to 3. In the first sub-figure, observe that the policy is now heading towards the largest reward for most of the state space. In the second sub-figure, the distribution over return has now mostly shifted towards larger returns which
4

---

Under review as a conference paper at ICLR 2023
Empirical distribution
b) Tilting distribution
c) Exp. tilted distribution
piG|stexp(G k(1.0))
exp(G K(1))
p|G|s)
return (G)
return
return (G)
Figure 3: For RyS, a distribution biased towards higher returns. As we increase . the mass shifts towards higher returns. A limitation of these RvS methods is that the continuous distribution of return has to be discretized as shown here. While our proposed implicit method does not require the return to be discretized.
the exponential tilt of learned distribution over the return obtain
likely under the dataset, i.e., +1. In the third sub-figure, we observe that the most likely action is x/4 which is pointing towards the best goal.
4 RELATED WORK
4.1 IMPLICIT MODELS AND REINFORCEMENT LEARNING
We note that other works have used EBMs in RL. Interestingly, most previous works treat the Q- value as the energy and do not model how likely the Q-value is (or the return in our case). Sallans & Hinton (2004) parametrize the Restricted Boltzmann Machine (Freund & Haussler 1991] Welling et al. 2004) negative energy function as the Q value. And more recently, Haarnoja et al. (2017) learn Q function via the (soft-) Bellman equation, then treat the Q function as the negative energy function, which is sampled via stochastic neural network. Recently, EBM techniques have been used learn conservative Q-values (Kostrikov et al. 202T)
4.2 CONNECTION TO RETURN CONDITIONAL POLICIES
Offline RL via supervised learning is often done by learning return conditional policy (rriaastaaal et al. 2019 Kumar et al.) 2019: Emmons et al. 2021 Chen et al.) 2021). Specifically, the policy p(a s, G) is conditioned
the return G. which can easily learned via supervised learning and does not require Temporal- Difffernnce learning (Sutton) 1988). At test time, the policy requires a Temporal-Difference
return-to-go be conditioned on. It is often treated task-dependent hyper-parameter (Chen et al. 2021 Emmons et al. 202T) or biased estimate of the return (a scalar instead of a distri- bution) estimated by second neural network (Kumar et al. 2019).
Inspired by the exponential tilt formulation used for IRvS, we propose instead to model the state conditional distribution over returns p(G s) using supervised learning by discretizing the returns, see Figure 3, Similarly to IRvS, the return to go can be obtained by maximizing the exponential tilt
G arg max p(G s,)) exp(1) -G x(n))
resulting in high return which also likely under the learned model. The three distributions are depicted in Figure 3 Thus, we argue that if the action is obtained by maximizing the learned likelihood of the model, e.g., outputting the mean a Gaussian distribution.
à arg max (a St, G.).
(10)
Thus, RvS and IRyS obtain the actions by maximizing the same objective, and their main differences in the modeling choices. First, using an implicit model is simpler than explicit since it
5

---

Under review as a conference paper at ICLR 2023
IRy5
RvS
(a) agent must reach the left-most (b) Near discontinuities, IRvS reaches both goals, while RvS fails to reach either goal and converges to the purple point.
goal.
i) Extrapolation
ii) Dimensions
iii) Discontinuities
1.0
% Success
0.5
IRvS RvS
0.0
0.1 0.25 0.5 0.75 1.0
4
8 16
24
32 1e-4 1e-3 1e-2
le-1
D
5
(c) Difference in performance between implicit RvS and (explicit) RvS.
Figure Comparison between Implicit and Explicit RvS method. Figure a) shows how the agent changes focus from one goal to the other depending which one is placed more to the left. Figure b) depicts how implicit vs explicit Rvš behave when the goal to reach is unclear. Figure e) measure the performance of implicit and explicit RvS method along extrapolation capabilities, environment dimensionality, and policy discontinuities.
requires only a single neural network, (a s)s), compared to two networks for an explicit model, i.., pe St, G.) and pe(Gr (Gr 1s),, see Figure Second, implicit models can model rich continuous distributions making them a natural tool for control tasks, while explicit models must discretize continuous variables or assume a parametric distribution such as the Gaussian to model the multi- modal action and return distributions. summary, by modeling the dependency between the state, action, and return with an implicit model, we bridge an important gap between the popular IBC and RvS methods.
Concurrently, [ee et al. (2022) introduced binary "expert" variable, which leads to similar way to select the return-to-go on, which the policy is conditioned on. They then demonstrated impressive performance using Transformer return-to-ga go, they discretize the variable and model it with a categorical distribution. We note that introducing binary "expert" variable is closely related to the concept of "optimality" in control-as- inference [evine 2018).
the Atari suite. In order to model the distribution over the
EXPERIMENTAL RESULTS
5.1 ADVANTAGES IMPLICIT RYS OVER EXPLICIT RvS
In order to understand the advantages of implicit RvS over explicit RyS, we proposed simple navigation task with linear dynamics, where the agent has to reach the left-most goal. i.e. the goal with the smallest first dimension. The optimal policy while being relatively simple is discontinious as the left-most goal changes. For example, in Figure slightly moving the bottom goal left results in completely different optimal action.

---

Under review as a conference paper at ICLR 2023
Experimental settings. For all experiments, we keep the number of demonstrations constant 100,000. The agent's 's initial position s is sampled uniformly from [-1, The 2 goals during training are independently sampled uniformly from 1-0.1,0.1J2. The state is composed of the agent's position and two goals, resulting in a state of dimensions 3 x D. The optimal action is a' 9- s which is also of dimension D, where g" is the left-most goal. To better understand the advantages of implicit models over explicit ones, we measure i) the performance as we increase the dimensionality of the environment, ii) the ability to model discontinuous policies, and iii) the generalization capabilities of the model as we change the goal sampling distribution.
Extrapolation. Implicit models have been shown to extrapolate more effectively than explicit ones. This environment demonstrates this is also the case for implicit versus explicit RvS. To study ex- trapolation, the dataset is composed of demonstrations where the goals are placed near the origin, i.e., 9 is sampled on [-0.] 0.1, 0.1] At the same time, we can increase the goal sampling radius (c) at test time to measure the extrapolation capabilities of the model. In Figure |4c a) we observe that implicit RvS extrapolates more effectively than explicit RvS to changes in goal radius much larger than observed during training.
High-dimensions. As we increase the environment's s dimensionality but keep the dataset's size con- stant, the training demonstrations are, on average, further away from what is observed at test time. Thus increasing the dimensions of the environment increases the difficulty of the task. In Figure 4c] b), we can observe that implicit RvS performance degrades more gracefully than explicit RvS. Sim- ilar behavior has also been observed in Florence al. (2021).
Discontinuities.
Looking at Figure 4a we can see that as the bottom goal slide to the right, the optimal action switches from going to the bottom goal the top one: making the optimal policy highly discontinuous. This represents a challenge for explicit models as they can hardly fit discon- tinuous functions and instead fit intermediate values. We can measure the models ability to handle discontinuous functions by controlling the relative position of the second goal. Specifically, instead of sampling two goals independently (as done above), we can set the second goal's first dimension equal to the first goal's first dimension with a small amount of uniform noise sampled from [-ō, 8). In Figure 4b) we observe qualitatively the behavior of the two models for 8=1e-4. We observe that the (explicit) RvS policy converges to the purple point which is between the two goals and rarely reaches one of them, while IRvS policy reaches the correct goal half the time. Finally, in Figure 4c quantify the differences between the policies by measuring their success rate for different 5. We see that IRvS outperforms RyS for all 8.
5.2 CONTINUOUS CONTROL DATASET
We compare both IRvS and our version of RvS (as defined by Equation ( and Equation (T)) to
implicit and explicit state-of-the-ane algorithms on suite of offline RL tasks provided by d4rl (Fu al.,|21 (020). Specifically, we compare our methods to conservative Q-learning (Kumar al.) 2020). to the implicit algorithms IBC and IBC w/ RWR proposed by FForence et al. (2021). and to the
explicit algorithms BC from Fu al. (2020), and RvS hand-uune (the target returns were selected
by hand on task-level) from Emmons et al. (2021)
5.2.1 EXPERIMENT SETTINGS.
We fine-tuned the hyper-parametert -1 for our algorithm on 3 seeds, see Appendix A.3 Then we run 3 seeds on the whole suite where we evaluate our algorithms for 20 episodes, for lask- results see Table [ As in Emmons et al. (222T) we define G as the average future reward. Additionally, we normalize each state coordinate have mean 0 and standard deviation 1, and the actions and the return G to lie in [-
The ADROIT benchmark (Kumar et al., 2013) Rajeswaran et al. 2017) consists of environments (relocate, pen, door, hammer) and 3 datasets (human, cloned, expert) for total of 12 tasks. The agent controls a 24-DoF anthropomorphic manipulator and must solve challenging and dynamic dexterous manipulation. The first environment relocation requires the agent to move a ball to a randomized target. Similarly, the pen environment requires the agent to reposition a pen to match the orientation of randomized target. The door environment requires the agent undo the latch and swing the door open. Finally, the hammer environment requires the agent to pick up a hammer and drive nail into board. The human dataset is comprised of 25 human demonstrations, The
ADROIT.
7

---

Under review as a conference paper at ICLR 2023
(Ours)
(Ours)
CQL
IBC
(Oun)
(Ous))
IBC
7*
Best 1)
Hand-mned
Bast !
adrvtt
cloned
doer
6.3
13.0
-0.1
0.4
0.1
0.1
0.1
-0.1
hammer
3.9
3,9
0.8
2.1
0.3
2.8
3.1
0.3
pen
81.1
81.4
64.5
expert
101.5
hammer
127.2
127.2
125.6
86.7
118.0
127.0
127.0
123.4
128.1
138.1
$5.1
107.0
123.0
127.5
127.5
116.2
relocate
102.5
107.5
101.3
95.0
106.3
106.5
106.7
102.3
human
0.5
11.0
1.5
pen
84.3
34.4
37.5
56.7
62.9
76.6
71.4
relocatc
0.3
01.3
0.0
0.2
0.1
0.0
0.1
0.1
kitchen
complete
kinchen
65.4
78.8
33.8
43.8
1.5
69.1
45.8
46.9
76.2
mixed
kischen
67.9
67.9
47.5
51.0
52.1
46.5
47.9
29.6
portial
kitchen
52.9
42.5
mujoco
medium
41.5
hopper
61.7
61.7
29.0
64.6
60.2
$4.1
59.4
59,4
53.3
walker2d
64.9
69.5
6.6
N2.9
71.7
64,3
70.4
70.4
39.7
medium-expee expert
halfchectah
93.5
93.5
35.8
85.8
92.2
92.5
92.2
92.2
51.4
101.6
101.6
111.9
102.0
101.7
102.1
94,9
91.1
walker2d
107.2
107.2
6.4
109.5
106.0
105.8
107.8
108.5
97.8
medium replay
hopper
27.2
11.8
97.8
73.5
17.2
61.7
63.0
walker2d
$2.9
$2.9
11.3
86.1
60.6
26.6
31.8
51.8
16.0
Table 1: Normalized score per task.. In general, IRvS performs well on -igh-nimensional manip- ulation tasks where the dataset has been collected by policies on varying expertise levels. While CQL. and explicit methods perform well on the locomotion tasks. All algorithms perform well on the adroit expert dataset.
expert dataset is comprised of 5000 trajectories sampled from an expert policy. The cloned dataset is comprised of 2500 trajectories from the expert policy and 2500 trajectories from behavior cloning policy trained on the human demonstrations.
Mujoco. The offline mujoco benchmark (Todorov et al. 2012) consists of 3 locomotion envi- ronments (balfcheetah, walker2d, and hopper) and datasets (medium, medium-replay medium- expert) for total of 9 tasks. The hopper environment consists of one-legged robot system with 11-dimensional state space and degrees of freedom (DoF), halfcheetah is a cheetah robot with I--immensional state pace and 6-DoF, finally, walker2d has 17-dimensional state space and 6-DoF. The medium datasets consist of 10th samples, the medium-expert of 2 x 10th samples, and the medium replay of 10 samples.
FrankaKitchen. The FrankaKitchen (Gupta et al. 2019) (kitchen) environment places a 9-DoF robot in a realistic kitchen environment where it can intract freely with different objects, e.g. open- ing the microwave and various cabinets, moving the kettle, and turning on the lights and burners. The task consists of reaching the desired configuration of the objects in the scene. The kitchen bench- mark consists of 3 different tasks (complete, partial, and mixed). The complete dataset includes demonstrations of all target sub-tasks being completed order. The partial dataset includes con- sists of multiple other tasks being performed including the 4 target sub-tasks being completed in sequence. Finally, the mixed dataset consists of various sub-tasks, but the 4 target sub-tasks are never completed in sequence together.
5.2.2 RESULTS
ADROIT. On 11 of the 12 datasets, IRvS obtains or matches the best performance. We also note that, excluding IRvS, IRvS y = obtains the best performance on 6 of the 12 datasets. With the exception of Conservative Q-learning (CQL.) (Kumar et al., 2020) obtaining the best score on hammer-human, implicit methods outperform explicit ones most tasks. The gap between IRvS and RyS particularly noticeable for the human and cloned datasets hinting that implicit methods perform better for high-dimensional control tasks combined with datasets collected by policies of different proficiency.
Mujoco. CQL obtains the best score 8 of the 9 datasets while IRvS n 1 obtains the best score on halfcheetah medium-expert. These results suggest that TD learning and trajectory stitching crucial the mujoco benchmark obtain the best performance. Interestingly, IRvS and RvS
8

---

Under review as a conference paper at ICLR 2023
walker2d
halfcheetah
hopper
Normalized Return
100
80
60
5
10
5 1
10
0
10
n
1 n-
Figure 5: Effectiveness 17 to incorporate return information. We report the performance of IRvS as a function of 17 on the medium-expee datasets, For walker2d, we observe that increasing 7-1 slightly improves the performance, but does not impact performance too much past a certain point. For halfcheetah, we see that increasing 17_ dramatically improves the performance. For hopper, we observe that increasing 77 improves the performance up to a point, then the return falls below the behavior cloning performance (1) 0). These observations confirm that n allows us to trade-off between the likelihood of a behavior and its potential return. denote rj (the across all tasks) by red vertical lines.
best overall
hand-tuned (Emmons et al. 2021) obtain almost identical performance on the medium and medium- expert datasets, while RvS obtains much better scores on the medium-replay datasets. It is worth noting that our implementation of RvS 7j most of the locomotion tasks, where RvS hand-runed condition the following hand-selected return: 110 for the medium-expe expert dataset, 90 for hopper,
10 achieves roughly the same score as RvS hand-ture
ceeeeedd-mddiu--rplleee and walker2d- medium, 60 for hopper-medium, and 40 for halfcheetah-mediine medium-replay.
IRvS reaches the best score on all datasets. Additionally, excluding IRvS, reaches the best score on 2 of the 3 tasks, while IBC (Florence et al. (2021) obtains the best score on kitchen complete which is well suited for behavior cloning. Surprisingly, RvS 1- 10 outperforms by wide margin RvS hand-tuned. Overall, implicit methods outperform explicit methods for all kitchen tasks.
FrankaKitchen.
IRvS n
Sensitivity Analysis. In Figure we observe that for every task the best performance is obtained with 7 > 0. Thus confirming the effectiveness of to incorporate return information by 0). Interestingly, increasing 17 beyond a point does impact the performance for walker2d. However, increasing 77 beyond point decreases the performance on hopper, thus implying that some tasks cannot solved by naively conditioning on the maximum observed return. Finally, while the best performance is reached using different 7/-1, we found that using 7) (red vertical line) for tasks results good overall performance. Additional figures on the sensitivity of n can be observed in Appendix A.3
outperforming IBC (1)
CONCLUSION
This work bridges essential gap between implicit models and RyS methods. By bridging this gap, practitioners will be able to enjoy the advantages of implicit models to learn robotic skills from an offline dataset of varied expertise. Overall, IRvS is conceptually simpler than previous explicit RvS methods as it requires a single network and does not require quantizing the return. In addition to being simpler, we have shown that IRvS can better capture discontinuities, extrapolate, and handle higher dimensional problems than explicit RvS. Additionally, we use insight from IRvS to derive a simpler RvS algorithm where the target return is controlled via a temperature 7j instead of being specified by hand or treated as scalar. Furthermore, we have demonstrated performance matching or exceeding ofte-off-hhe-rsulle on challenging offline RL benchmarks, particularly on the high- dimensional manipulation tasks where IRvS greatly outperforms the baselines. Finally, have performed sensitivity analysis where we have shown that our newly introduced hyper-paramen -1 can be set to 1 on all tasks without substantially degrading performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Søren Asmussen and Peter W Glynn. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.0133 0134444,.0.1,,,... 2021.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint vrxiv:2112.1075// 2021.
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. arXiv preprint arXiv:2109.00 1et.:21090011110 00137, 2021.
Yoav Freund and David Haussler. Unsupervised learning of distributions binary vectors using two layer networks. Advances in neural information processing systems, 4, 1991.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910. 1956, 2019.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110006166 2021.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 1rXiv:1912.13465. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Vikash Kumar, Zhe Xu, and Emanuel Todorov. Fast, strong and compliant pneumatic actuation for dexterous -oddnn-driven hands. In 2013 IEEE international conference on robotics and automa- fion, pp. 1512 1519. IEEE, 2013.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. tutorial energy-based learning. Predicting structured data, 1(0), 2006.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv:2205. 15241, 2022.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprini arXiv: 1805 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.01643; 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 2018
Dean A Pomerlcau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
10

---

Under review as a conference paper at ICLR 2023
Brian Sallans and Geoffrey E Hinton. Reinforcement learning with factored states and actions. The Journal of Machine Learning Research, 5:1063-1088, 2004.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaškowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. arXiv preprint arxivv1912.02877 2019.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference Machine Learning, 2011.
Max Welling, Michal Rosen-Zvi, and Geoffrey E Hinton. Exponential family harmoniums with an application to information retrieval. Advances in neural information processing systems, 17, 2004.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 STOCHASTIC GRADIENT LANGEVIN DYNAMICS
Algorithm Stochastic Gradient Langevin Dynamics
Require: Initial state S
Require: Initial action a (aminn amms
Require: Initial return G ~ 1(
Require: Learning rate a
1: # Density Estimation
2: for E ,,....,, 100) do
3: 4 V Eo G)
4: + (l-ppppp2 5.9 + -05, 0.5). E 1(0
5: (a, G) t (a, G) au
6: end for
7: Return: as,a, G)
A.2 HYPER-PARAMETERS
For IRvS, all the hyper- parameerrs with the exception of the learning rate are the same as the ones reported by Florence et al. (202T). For our implementation of RvS, we closely followed the hyper-
parameters reported by Emmons et al. (202T.
plyper-parameter
Chosen Value
Swept Value
FBM variant
Langevin
batch size
512
512
learning rate
Te-3
le-4, 5e-4, 1e-3
learning rate decay
0.99
learning rate decay steps
100
network size (width depth)
5T2x8
activation
ReLU
swish, leaky ReLU, ReLU none, layer regular, spectral norm
regularization
layer type
spectral norm
train counter-examples
action boundary buffer
0,05
gradient penalty
final step only
gradient margin
1.0
Langevin iterations
100
Langevin learning rate init.
0.5
Langevin learning rate final
le-5
Langevin polynomial decay power
2
Langevin delta action clip
0.5
Langevin noise scale
0.5
Langevin 2nd iteration learning rate
Te-5
Didactic example gradient steps
2000
Navigation task gradient steps
10000
Continuous control gradient steps
100000
Table 2: Hyper parameter for IRvS
CQL Results, All the CQL results are from Fu et al. (2020), except for the mujoco results which from Emmons et al. (2021), since the results were from the previous simulator version.
IBC and filter-IBC. We have reproduced the results from Florence et al. 2021 such that IBC, IRvS, and IBC w/ RWR share the same code-base.
RvS. We report the results RyS results from Emmons et al. (202I) as RyS. we note that they con- dition return of 10 for the medium-expert dataset, 90 for hopper, walker2dd-mediu--repl.. and
12

---

Under review as a conference paper at ICLR 2023
Hlyper-paaamtter
Chosen Value
Swept Value
batch size
16384
16384
learning rafe
le-3
le-4, 5e-4, le-3
network size (width depth)
2048x2
activation
ReLU
swish, leaky ReLU, ReLU
Navigation task gradient steps
10000
Continuous control gradient steps
10000
Scales
Ic-5
le-4, 1c-5, lc-6
Number of atoms
101
21, 51, 101
Table Hyper parameter for RvS
walker2d- malker2d-medium, 60 for hopper-medium, and 40 for halfcheetah meficheetah-meiimm, medium-replay. Ad- ditionally, we reported the RvS 77' 10, where the action selection and return prediction are performed as described in Equation (T)) and Equation (9) respectively.
A.3
SENSITIVY ANALYSIS
walker2d
halfcheetah
hopper
70
42.0
65
60
medium
return
60
41.5
58
55
41.0
56
50
40.5
54
90
100
104
80
medium-exper exppet
return
102
100
80
60
98
50
70
33
50
20
32
medium-replay
return
40
31
15
30
30
10
29
5
20
28
0
5 n-1
10
10
0
10
-1
r
Figure 6: Mujoco locomotion tasks. Hyper-parameter selection performed on these 3 seeds, and final results in Table are reported on new seeds.
13

---

Under review as a conference paper at ICLR 2023
relocate
pen
door
hammer
0.1
12.5
1.2
80
10.0
return
cloned
0.8
-0.1
2.5
0.4
0.0
108
130
102
107
120
return
100
expert
106
115
105
120
98
104
110
15
80
0.2
10
numan
70
0.5
10
7-1
7-1
7-1
Figure 7: Adroit manipulation tasks. Hyper-pane paremm-rreeeer selection was performed on these 3 seeds, and final results in Table I are reported on 3 new seeds.
kitchen partial
kitchen complete
kitchen mixed
Normalized Return
75
60
60
50
50
50
25
40
0
10
5
-1
n
n-1
Figure 8: Kitchen manipulation tasks. Hyper-parameter selection was performed on these 3 seeds, and final results in Table are reported on 3 new seeds.
14


Reviews:
[{"review_id": "t_l78s7w01", "reviewer": "Reviewer_ExRr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to incorporate the return information in implicit behaviour cloning. Empirical study confirms the efficacy of the proposed algorithms in some domains", "summary_of_the_review": "see above", "strength_and_weaknesses": "Strength:\nThe paper is well written and easy to follow. The idea is simple but proves to be effective in some domains. \n\nWeakness:\n1) The improvements in adroit and kitchen are impressive. However, the reason for the failure in Mujoco is not clear. Since the main contribution of this work is empirical, I feel the authors should dig into Mujoco more deeply to understand this failure.\n2) I think some important baselines are missing, e.g., the authors should include comparison with offline RL methods with return-conditioned policy with explicit model, mentioned in Section 4.2. \n3) Table 1 results from only 3 seeds without metrics on variation. So it is not as informative as it should be.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RzUo6wL1zVy", "reviewer": "Reviewer_3Dga", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using implicit models for offline reinforcement learning. More specifically, this paper proposes using an implicit model to model the distribution of actions and returns conditioned on observations, and to bias sampling towards actions with high returns. The paper evaluates using implicit models against state of the art offline reinforcement learning methods on the ADROIT, Mujoco, and FrankaKitchen settings.", "summary_of_the_review": "I think this is an interesting direction, but due to the limitations in the performance of the method and drawbacks of the method itself, I think the paper is not ready for acceptance. ", "strength_and_weaknesses": "Strengths:\n- The paper is well-written and clear to follow.\n- The proposed method is interesting and seems to be pretty simple and elegant. \n- The toy examples and plots are nice in providing some intuition about the approach.\n\nWeaknesses:\n- It's not clear why IRvS is better than the naive way of incorporating return information by combining IBC with RWR. Furthermore, IRvS only does slightly better than IBC w/ RWR, and the performance is similar on most tasks. IRvS with the best eta does perform better, but requires tuning for eta. Much more thorough analysis is needed here, as this is the main contribution of the paper.\n- Training EBMs with Langevin Dynamics can be difficult and requires many hyperparameters. \n- Sometimes reward information is unavailable or difficult to get, particularly in real robotics, and the method may struggle in these cases.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EVGIZExERPY", "reviewer": "Reviewer_aCoh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an Implicit Reinforcement Learning via Supervised Learning (RvS) methods by leveraging the implicit model---the composition of $\\arg\\min$ with a general function approximator $f_\\theta$ to represent the policy ($\\hat{a} = \\arg\\min_a f_\\theta (s, a)$) [1]---instead of the traditional explicit model ($a = f_\\theta (s)$) in RvS methods to solve offline RL problems. Moreover, the authors provide empirical results to show the superiority of the implicit model. Experiments demonstrate the performance improvement of the implicit model in RvS methods for offline RL.\n\n[1] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.", "summary_of_the_review": "I think the proposed method is a simple combination of IBC [1] and RvS. The claimed major novelty of this paper is the same as that proposed by IBC.\n\n[1] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.", "strength_and_weaknesses": "Strength: \n1. The authors leverage the exponential tilt density [1] to learn policies that can head toward the largest rewards from offline datasets collected by policies with different expertise levels.\n\nWeaknesses:\n1. The authors claim that they bridge an important gap between IBC [2] and RvS by modeling the dependencies between the state, action, and return with an implicit model on Page 6. However, noticing that IBC proposes to use the implicit model to model the dependencies between the state and action, I think the contribution of this paper is to introduce the return from RvS to the implicit model. Thus, the proposed method looks like a combination of IBC and RvS.\n\n2. The authors conduct experiments in Section 5.1 to show the advantages of the implicit model. However, such advantages are similar to IBC, which could hurt the novelty of this paper. The authors may want to highlight the novelty of the proposed method against IBC.\n\n3. The discussions of the empirical results in Sections 5.1 and 5.2.2 are missing. The authors may want to explain: 1) why the RvS method fails to reach either goal and converges to the purple point in Figure 4(b); 2) why the explicit methods perform better than implicit methods on the locomotion tasks.\n\n4. The pseudo-code of the proposed method is missing.\n\n[1] Søren Asmussen and Peter W Glynn. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.\n\n[2] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject