OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
Authors: Anurag Ajay
Aviral Kumar
Pulkit Agrawal
Sergey Levine
Ofir Nachum
Venue: ICLR 2021 Poster (2021)
Keywords: Offline Reinforcement Learning
Primitive Discovery
Unsupervised Learning

Abstract: Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent’s ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed:  an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning.  Primitives extracted in this way serve two purposes:  they delineate the behaviors that are supported by the data from those that are not, making them useful for avoiding distributional shift in offline RL; and they provide a degree of temporal abstraction, which reduces the effective horizon yielding better learning in theory, and improved offline RL in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning as well as exploration and transfer in online RL on a variety of benchmark domains. Visualizations and code are available at https://sites.google.com/view/opal-iclr

Published as a conference paper at ICLR 2021
OPAL: OFFLINE PRIMITIVE DISCOVERY FOR ACCEL- ERATING OFFLINE REINFORCEMENT LEARNING
Anurag Ajay"! Aviral Kumar?, Pulkit Agrawal', Sergey Levine Ofir Nachum?
MIT. "Google Research, "UC Berkeley
ABSTRACT
Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely lim- ited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning. Primitives extracted in this way serve two purposes: they delineate the behaviors that are supported by the data from those that are not, making them use- ful for avoiding distributional shift in offline RL; and they provide a degree of tem- poral abstraction, which reduces the effective horizon yielding better learning theory, and improved offline RL. in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning well as exploration and transfer online RL on a variety of benchmark domains. Visualizations and code are available at https: //site//ssssseeeeeeeeeesseeeeett google wwwooll.coo//aeeoooooooooooooo wwwoo//iww//paal///eeeee
INTRODUCTION
Reinforcement Learning (RL) systems have achieved impressive performance in a variety of on- line settings such as games (Silver et al., 2016; Tesauro, 1995; Brown & Sannhhlm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can in the environment and sample as many transitions and rewards as needed. However, in many practical applications the agent's ability to continuously act in the environment may be severely limited due to practical concerns (Dulac- Arnold et al., 2019). For example, a robot learning through trial and error in the real world requires costly human supervision, safety checks, and resets (Atkeson et al., 2015), rendering many stan- dard online RL algorithms inapplicable (Matsushima et al., 2020). However, in such settings we might instead have access to large amounts of previously logged data, which could be logged from policy even from other related tasks. For example, in self-driving
baseline hand-engineerr applications, one may have access to large amounts of human driving behavior; in robotic applica- tions, one might have data of either humans robots performing similar tasks. While these offline datasets are often undirected (generic human driving data on various routes in various cities may not be directly relevant navigation of specific route within specific city) and unlabelled (generic human driving data is often not labelled with the human's 's intended route or destination). this data is still useful in that it inform the algorithm about what is possible do in the real world, without the need for active exploration.
In this paper, we study how, in this offline setting, an effective strategy to leveraging unlabeled and undirected past data is to utilize unsupervised learning to extract potentially useful and temporally extended primitive skills to learn what types of behaviors are possible. For example, consider a dataset of an agent performing undirected navigation in a maze environment (Figure 1). While the dataset does not provide demonstrations of exclusively one specific point-to-point navigation task,
Work done during an intersship Google Brain

---

Published as a conference paper at ICLR 2021
+
R
LP
2P
(a) medium (diverse) (b) medium (CQL+OPAL)
(c) large (diverse)
(d) large (CQL+OPAL)
Figure I: Visualization of (a subset of) diverse datasets for (a) antmaze medium and (c) antmaze
large, along with trajectories sampled from CQL+OPAL trained on diverse datasets of (b) antmaze medium and (d) antmaze large.
nevertheless presents clear indications of which temporally extended behaviors are useful and natural in this environment (e.g.. moving forward, left, right, and backward), and our unsupervised learning objective aims to distill these behaviors into temporally extended primitives. Once these locomotive primitive behaviors are extracted, extended action space for learning a task policy with offline RI, which only needs to focus on task relevant navigation, thereby making task learning easier. For example, once specific point-to-point navigation is commanded, the agent can leverage the learned primitives for locomotion and only focus on the task of navigation, as opposed to learning locomotion and navigation from scratch.
use them as compact constrained temporally-
We refer to our proposed unsupervised learning method as Offline Primitives for Accelerating of- fline reinforcement Learning (OPAL), and apply this basic paradigm to offline RL, where the agent is given single offline dataset to use for both the initial unsupervised learning phase and then a subsequent task-directed offiin policy optimization phase. Despite the fact that no additional data is used, we find that our proposed unsupervised learning technique can dramatically improve offline policy optimization compared to performing offline policy optimization on the raw dataset directly. To the best of our knowledge, ours is the first work to theoretically justify and experimentally verify the benefits of primitive learning in offline RL settings, showing that hierarchies can provide tempo- ral abstraction that allows to reduce the effect of compounding errors issue in offline RL. These theoretical and empirical results are notably in contrast to previous related work in online hierarchi- cal RL (Nachum et al., 2019b), which found that improved exploration is the main benefit afforded by hierarchically learned primitives. We instead show significant benefits in the offline RL. setting, where exploration irrelevant.
Beyond offline RL, and although this isn't the main focus of the work, we also show the applicability of our method for accelerating RL by incorporating OPAL as a preprocessing step to standard online RL, few-shot imitation learning, and multi-task transfer learning. In all settings, we demonstrate that the use of OPAL can improve the speed and quality of downstream task learning.
2 RELATED WORK
Offline RL. Offline RL presents the problem of learning a policy from a fixed prior dataset of transitions and rewards. Recent works in offline RL (Kumar et al., 2019; Levine al., 2020; Wu et al., 2019; Ghasemipour et al., 2020; Jaques et al., 2019; Fujimoto et al., 2018) constrain the policy to be close the data distribution to avoid the use of out-of-distribution actions (Kumar et al., 2019; Levine et al., 2020). To constrain the policy, some methods use distributional penalties, as measured by KL divergence (Levine et al., 2020; Jaques al., 2019), MMD (Kumar et al., 2019), or Wasserstein distance (Wu et al., 2019). Other methods first sample actions from the behavior policy and then either clip the maximum deviation from those actions (Fujimoto et al., 2018) or just use those actions (Ghasemipour et al., 2020) during the value backup to stay within the support of the offline data. In contrast to these works, OPAL uses an offline dataset for unsupervised learning of a continuous space of primitives. The use of these primitives for downstream tasks implicitly constrains a learned primitieeddirectig policy to stay close to the offline data distribution. As we demonstrate in our experiments, the use of OPAL in conjunction with an off-the-shhe offline RL algorithm in this way can yield significant improvement compared to applying offline RL to the dataset directly.
2

---

Published as a conference paper at ICLR 2021
Online skill discovery. There are number of recent works (Eysenbach et al., 2018; Nachum al., 20;8;; Sharma et al., 2019) which use unsupervised objectives to discover skills and use the discov- ered skills for planning (Sharma et al., 2019), few-shot imitation learning, online RL (Eysenbach et al., 2018; Nachum et al., 2018a). However, these works focus on online settings and assume access to the environment. In contrast, OPAL. focuses settings where large dataset of diverse behaviors is provided but access to the environment is restricted. It leverages these static offline datasets to discover primitive skills with better state coverage and avoids the exploration issue of learning primitives from scratch.
Hierarchical policy learning, Hierarchical policy learning involves learning a hierarchy of policies where a low-level policy acts as primitive skills and a high-level policy directs the low-level policy to solve task. While some works (Bacon et al., 2017; Stolle & Precup, 2002; Peng et al., 2019) learn discrete set of lower-level policies, each behaving as primitive skill, other works (Vezhn- evets et al., 2017; Nachum et al., 201 80; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills representing the lower-level policy. These methods have mostly been applied in on- line settings. However, there have been some recent variants of the above works (Lynch et al., 2020; Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) which extract skills from prior dataset and using it for either performing tasks directly (Lynch et al., 2020) or learning downstream tasks (Shankar & Gupta, 2020; Krishnan et al., 2017; Merel al., 2018) with online RI. While OPAL is related to these works, we mainly focus on leveraging the learned primitives for asymptot- ically improving the performance of offline RL; i.c., both the primitive learning and the downstream task must be solved using single static dataset. Furthermore, we provide performance bounds for OPAL and enumerate the specific properties an offline dataset should possess to guarantee improved downstream task learning, while such theoretical guarantees are largely absent from existing work.
PRELIMINARIES
We consider the standard Markov decision process (MDP) setting (Puterman, 1994), specified by
tuple M A,, p4,,,, where S represents the state space, AA represents the action space,
P(s'|s, (s, a) represents the transition probability, (u()) represents the initial state distribution, r(s,a) a) E (-Rmax: Rmma)) represents the reward function, and 7 (0, 1) represents the discount factor. A policy π in this MDP corresponds to function S + ((A), where A(A) is the simplex over A.
It induces a discounted future state distribution d=, defined by d' (1 - 7) Eee 7P((s,
s|TT, where P(s; = S|ı) is the probability of reaching the state s at time by running on M.
For a positive integer k. we use (s) = (1 - (1-a)) Er= **
s/T) to denote the every-
k-step state distribution of л. The return of policy in MDP M is defined as JRI(T,M)
B (awx(al)) a)). We represent the reward- and discount-agnostie environment as tuple 8 AS
F.
I-
a, We aim to use a large, unlabeled, and undirected experience dataset D : (-f := (s,,a;); d) i= associated with E to extract primitives and improve offline RL for downstream task learning. To account for the fact that the dataset D may be generated by a mixture of diverse policies start- ing at diverse initial states, we assume D is generated by first sampling a behavior policy π II along with an initial state 8 ~ K, where II, k represent some (unknown) distributions over poli- cies and states, respectively, and then running on E for time steps starting at so s. We define the probability of a sub-trajectory T := (S0, ag..... Sc-1, 9c-1) in D under a pol- --- P(selst ax-1) 11=0 π (aa)st)) and the conditional probability as icy as (T)) <(so) I (T]s) = 1[s so] Ii= P(s,/st-1, Ii T (ay]s)). In this work, we will show how to apply unsupervised learning techniques to D to extract continuous space of primitives (ala,, 2), where z E Z, the latent space inferred by unsupervised learning. We intend to use the learned ra(a,s, 2) to asymptotically improve the performance of offline RL for downstream task learning. For offline RL, we assume the existence of a dataset D" == { := (st, ae re) 1 h corresponding to the same sub- uub-trajectories in D labelled with MDP rewards. Additionally, we can use the extracted primitives for other applications such as fho-shot imitation learning, online RL, and online multi-tasé -ask transfer learning. We review the additional assumptions for these applications in Appendix A.
N
3

---

Published as a conference paper at ICLR 2021
Forward
Autoancoding loss
Fine-tuning BC
OPAL
Priitive Polley sa(o)s,
Encodar (ol=lr)
Primitve Polly sa(a a)
Encoder (o(s/r)
Laba led Data
Prinitive Policy as(a/s.2)
fr T- (r.
every slep
Untroelied a Undincttdd Latta
Di (r (se 3)mmd)
Dii Us5 2o L Krell
Task Pollcy Vip (sle)
Task Polisy (sle)
onca per stops
Offiine unsupervise primitive
(2) Ofline training taok policy
policy exocuitio
OPAL
Figure 2: Overview of offline RL with OPAL. OPAL is trained unlabelled data D using autoen- coding objective. For offline RI, the encoder first labels the reward-labelled data DD" with latents, and divides it into Pi and D The task policy Ty is trained on Di using offline RL while the primitive policy To is finetuned on D using behavioral cloning (BC).
OFFLINE RL WITH OPAL
In this section, we elaborate on OPAL, our proposed method for extracting primitives from and then leveraging these primitives to learn downstream tasks with offline RL. We begin by describ- ing our unsupervised objective, which distills ID into a continuous space of latent-conditionnd and temporally-extende
primitive policies no (a)ss, z). For learning downstream tasks with offline RI, we first label D with appropriate latents using the OPAL encoder (o and then learn a policy (z|s) which is trained to sample an appropriate primitive every steps to optimize specific task, using any off-the-shelf offline RL algorithm. A graphical overview of offline RL with OPAL is shown in Figure 2. While mainly focus on offline RL, briefly discuss how learned primitives for few-shot imitation learning, online RL, and multi-task online transfer learning in section and provide more details in Appendix A.
the
4.1 EXTRACTING TEMPORALLY
PRIMITIVES FROM DATA
We would like to extract continuous space of lemporally-extended primitives Te (als,, from D which can later use as an action space for learning downstream tasks with offline RL. This would reduce our effective task horizon, thereby making the downstream learning easier, as well as allow the downstream policy to stay close to the offline data distribution, thereby bringing stability to the downstream learning. We propose the following objective for learning TT. incorporating an auto-encoding loss function with a KI. constraint to encourage better generalization:
min 1(0,,,0 By BrwD, 0.0..
Σ log no (a (s,, 2)
s.t. E.-D (as (2| (Ilr))) S EKL
(2)
where È indicates empirical expectation. The learned components of this objective may be inter- preted encoder, decoder, and prior:
go (=|T) encodes the trajectory T state-action pairs into a distribution latent space and gives out parameters of that distribution. In our case, we represent ys with a bidirectional GRU which takes in 7 and gives out parameters
Encoder:
Gaussian distribution (μgnc gfn)).
ne (a) is the latent-conditioned policy. It maximizes the ditional log-likelihon likelihood of actions in 7 given the state and the latent vector. In our implementation, parameterize it as leed-forward neural network which takes in current state and latent vector Gaussian distribution for the action (Pa. Oa).
Decoder (aka Primitive Policy):
and gives out parameters
Prior/Primitive Predictor: (2(s0) tries to predict the encoded distribution of the tuj-trajcctory T from its initial state. Our implementation uses a feed-forward neural network which takes in the initial state and gives parameters of Gaussian distribution (ng" ag").
4

---

Published as a conference paper at ICLR 2021
KL-constraint (Equation 2). As additional component of the algorithm, enforce consistency in the latent variables predicted by the encoder 4s (2|T) and the prior Pa (z))s))) Since our goal is to obtain primitive 2 that captures temporal sequence of actions for given uubtrrjectoory T = (8o, An, ... ,,,-1,, ac-1) ) (as defined in Section 3), we utilize a regularization that enforces the distribution, (=(=))) to be close to just predicting the primitive the latent variable given the start (i.e. Pw (=]ss))) This conditioning on the initial state regularizes the
state of this sub-trajectory, distribution go (2(7) to not overfit to the the complete sub-trajectory т as the same z should also be predictable only given so. The above form of KL constraint is inspired from past works (Lynch et al,, 2020: Kumar et al., 2020a). In particular Lynch et al. (2020) add KL-constraint (Equation 2, "Plan prior matching" in Lynch et al. (2020)) that constrains the distribution over latent variables computed only given the initial state and the goal state to the distribution over latent variables computed using the entire trajectory. Our form in Equation 2 is similar to this prior except that we do not operate in a goal-conditioned RL setting and hence only condition P. on the initial state 50-
In practice, rather than solving the constrained optimization directly, we implement the KL con- straint as a penalty, weighted by an appropriately chosen coefficient B. Thus, one may interpret our unsupervised objective as using a sequential 3-VAE (Higgins et al., 2016). However, mentioned above, our prior is conditioned on Sy and learned as part of the optimization because the set of prim- itives active in D depends on so. If B 1, OPAL is equivalent to a conditional VAE maximizing log probability of T conditioned on its initial state sp; see Appendix D for more details. Despite the similarities between our proposed objective and VAEs, our presentation of OPAL as a constrained auto-encoding objective is deliberate. As we will show in Section 4.3, our theoretical guaranteess depend on a well-optimized auto-encoding loss to provide benefits of using learned primitives ma for downstream tasks. In contrast, a VAE loss, which simply maximizes the likelihood of observed data, may not necessarily provide a benefit for downstream tasks. For example, if the data can be generated by single stationary policy, VAE-optimal policy To can simply ignore the latent z, thus producing a degenerate space of primitives. contrast, when the KL constraint in our objective is weak (i.c., CKL > 0 or / < 1), the auto-encoding loss is encouraged to find unique for distinct T to optimize reconstruction loss.
4.2 OFFLINE RL. WITH PRIMITIVES FOR DOWNSTREAM TASKS
After distilling learned primitives from D in terms of an encoder (=(r))) a latent primitive policy (or decoder) we (a s, 22. and prior (z]so), OPAL then applies these learned models improve offline RL for downstream tasks.
As shown in Figure 2, our goal is use dataset with reward labeled sub-trajectories D (T+ (st, at, ) learn behavior policy æ that maximizes cumulative reward. With OPAL, use the learned primitives To (alss, 2) as low-level controllers, and then learn high-level controller (z|s). To do so, we relabel the dataset D in terms of temporally extended transitions using the learned encoder (=()) Specifically, we create a dataset D = {($) 24, Ei= ""r/ | where 2; ~ (-(T)) Given Di any off-the-shelf offline RI. algorithm can be used to learn TØ (in our experiments we use (Kumar et al., 2020b)). As a way to ensure that the c-step transitions
T, = (s), at, r)) remain consistent with the labelled latent action 2,, we finetune Te Te(a (s, 2) on Di {(({{, {)) m with simple latent-conditioned behavioral cloning loss:
min E (T,
Σ log No (arlst, 2)
(3)
4.3 SUBOPTIMALITY AND PERFORMANCE BOUNDS FOR OPAL
Now, will analyze OPAL and derive performance bounds for it in the context of offline RL, formally examining the benefit of the temporal abstraction afforded by OPAL as well as studying what properties D should possess so that OPAL can improve downstream task performance.
As explained above, when applying OPAL to offline RL, we first learn the primitives (a(als, 2) using D, and then learn a high-level task policy Ty (2]s)) in the space of the primitives. Let T (2|s)
be the optimal task policy. Thus the low-level To and high-level T&. together comprise a hierarchical policy, which we denote as ne. To quantify the performance of policies obtained from OPAL.,
5

---

Published as a conference paper at ICLR 2021
define the notion of suboptimality of the learned primitives Te(als, in an MDP M with an associated optimal policy π as
SubOpt(O) == [JRL(=" MM) JRL [To,
(4)
To relate SubOpt(0) with some notion of divergence between x¹ and na we introduce the fol- lowing performance difference lemma.
Lemma 4.0.1. If πi and are two policies in M, then
2 [RL M) (T2, M)I (1- -)) AmaxXBadd (rr(mm(rsssslll/111))))) (5)
where (T))) denotes the TV divergence over c-length sub-trajectories T sampled from vs. T2 (see section 3). Furthermore,
2 SubOpt(O) 1 (0 -7) Beax Bandy" (T (=) (T(+))]-
(6)
The proof the above lemma and all the following results are provided in Appendix B.I.
Through above lemma, we showed that the suboptimality of the learned primitives can be bounded by the total variation divergence between the optimal policy x² in M and the optimal policy acting through the learned primitives T0, We now continue to bound the divergence between x' and To terms of how representative DD is of л" and how optimal the primitives my are with respect to the auto-encoding objective (cquation 1). We begin with definition of how often arbitrary policy appears in II, the distribution generating D:
S Theorem 4.1. Let 0, ó, w be the outputs of solving equation 1, such that [0, w) cc. Then, with high probability 1 8. for any म that is Ç-common in II, there exists a distribution H over such that for mg (r|s) Ez-H Too (T(2,
Definition 1. We say a policy
M Ç-common II if E,.lll (Dr((aa(aa)(((((TTT(TT))
SS
E Exsx Drv DTV((T(T))lll (T(S))]
((())) S c+
Ec
He
2
where He = B Eizo log π (aelse)) (i.e. a constant and property of DD and S positive constant incurred due to sampling error in J(O, (.w) and depends on concentration prop- erties of T Te(a,s (), 2) and (2(T).
Corollary 4.1.1. If the optimal policy n² of M is C-common in II, and high probability 1 5,
d=
&. then, with
20
S+ (st
25 SubOpt(O) (1 - 71 7) Rmmx
SJ
E
Hhe
(i.e. Ç approaches 0) and better primitives are learned (i.e. decreases). While it might be tempting to increase e (i.e. the length of sub-trad cub-trajectories) reduce the suboptimality. a larger will inevitably make practically harder to control the autoencoding loss e.. thereby leading to increase in overall suboptimality and inducing a trade-off in determining the best value of c. our experiments we treat e as hyperparameter and set it to 10, although more sophisticated ways determine can be interesting avenue for future work.
As we can see, (ubOpttO) will reduce D gets closer
Till now, have argued that there exists some near-optimal task policy т&. if 0 is sufficiently learned and n" is sufficiently well-represented in D. Now, we will show how primitive learning can improve downstream learning, by considering the benefits of using OPAL with offline RL. Building on the policy performance analysis from Kumar al. (2020b), we now present theoretical results bounding the performance of the policy obtained when offline RL is performed with OPAL.

---

Published as a conference paper at ICLR 2021
Environment
BC
BEAR
EMAQ
CQL
CQL+OPAL (ours)
antmaaze medium (diverse)
0.0
8.0
0.0
53.7 = 6.1
81.1 3.1
antmaze large (diverse)
0.0
0.0
0.0
14.9 + 3.2
70.3 1 2.9
kitchen mixed
47.5
47.2
70.8 4 2.3
52.4 + 2.5
69.3
2.7
kitchen partial
33.8
13.1
74.6
+ 0.6
50.1
+ 1.0
80.2 + 2.4
Table I: Average success rate (%) (over 4 seeds) of offline RL methods: BC, BEAR (Kumar et al.,
2019), EMAQ (Ghasemipour et al., 2020), CQL (Kumar et al., 2020b) and CQL+OPAL (ours).
Theorem 4.2. Let Tg- Tg- (z|s) be the policy obtained by CQL and let τ.- ((=s) refer the policy when Tg. (z|s) is used together with Te(als, , 2). Let = { ~ II] refer to the policy generating D' in MDP M and HH (zls) TB,0==8 (z(Z)T) Then, (T 8. M) 2 ( M)
with high probability 1 where
IV (2((DQQ (T T (s) + +1)
== O (1-
E
and 2 Ma
(9)
(To 33 (s)]
(10)
1- xi (e)
where DOQL is measure of the divergence between policies; see the appendix for a formal statement.
The precise bound along with a proof described in Appendix B.1. Intuitively, this bound suggests that the worst-case deterioration over the learned policy depends on the divergence between the learned latent-space policy DeQL. and the actual primitive distribution, which is controlled via any conservative offline RL algorithm (Kumar et al. (2020b) in experiments) and the size of the latent space [z). Crucially, note that comparing Equation to the performance bound for CQL (Equation 6 in Kumar et al. (2020b)) reveals several benefits pertaining to (1) temporal abstraction a reduction in the factor of horizon by virtue of yr, and (2) reduction in the amount worst-case error propagation due a reduced action space |2] vs. |A]. Thus, as evident from the above bound, the total error induced due to a combination of distributional shift and sampling is significantly reduced when OPAL is used compared to the standard RI counterpart of this bound which is affected by the size of the entire action space for each and every timestep in the horizon. This formalizes our intuition that OPAL helps to partly mitigate distributional shift and sampling error. One downside of using latent space policy is that we incur unsupervised learning eтror while learning primitives. However, empirically, this unsupervised learning error gets dominated by other error terms pertaining to offline RL. That is, it is much easier to control unsupervised loss than errors arising in offline RL.
EVALUATION
In this section, we will empirically show that OPAL improves learning of downstream tasks with offline RL, and then briefly show the with few-shot imitation learning, online RL, and online multi-task transfer learning. Unless otherwise stated, we use e = 10 and dim(2 Z) = 8. See Ap- pendix C for further implementation and experimental details. Visualizations and code are available at https:////. //siisisssssss//.////....ees/set google e. com wttpe//oooo/wee//aaaaaailll
5.1 OFFLINE RL WITH OPAL
Description: We use environments and datasets provided in D4RL (Fu al., 2020). Since the aim of our method is specifically to perform offline RL in settings where the offline data comprises varied and undirected multi-task behavior, we focus on Antmaze medium (diverse dataset), Antmaze large (diverse dataset), and Franka kitchen (mixed and partial datasets). The Antmaze datasets involve simulated ant robot performing undirected navigation in a maze. The task to use this undirected dataset to solve specific point-to-point navigation problem. traversing the maze from to the opposite corner, with only sparse 0-1 completion reward for reaching the goal. The kitchen datasets involves franka robot manipulating multiple objects (microwave, kettle, etc.) either in
comner

---

Published as a conference paper at ICLR 2021
2P
26
(a) medium (CQL) (b) medium (CQL+OPAL)
(c) large (CQL)
(d) large (CQL+
Figure 3: State visitation heatmaps for antmaze medium policies learned using (1) CQL and (2)
CQL+OPAL, and antmaze large policies learned using (3) CQL and (4) CQL+OPAL.
Environment
BC
BC+OPAL (ours)
BC+SVAE
antmaze medium (diverse)
30.1 + 3.2
81.5 2.7
72.8 2.3
antmaze large (diverse)
9.2 = 2.5
63.5 + 2.3
49.4 + 2.2
Table 2: Average success rate (%) (over 4 seeds) of few-shot IL methods: BC, BC+OPAL, and BC+SVAE (Wang et al., 2017).
undirected manner (mixed dataset) or in a partially task directed manner (partial dataset). The task to use the datasets to arrange objects in a desired configuration, with only sparse 0-1 completion reward for every object that attains the target configuration.
Baseline: We use Behavior cloning (BC), BEAR (Kumar et al., 2019). EMAQ (Ghasemipour et al., 2020), and CQL (Kumar et al., 2020b) as baselines. We compare to CQL+OPAL, which first uses OPAL to distill primitives from the offline dataset before applying CQL to learn a primitive-directin high-level policy.
Results: As shown in Table 1, CQL+OPAL outperforms nearly all the baselines on antmaze (see Figure and Figure 3 for visualization) and kitchen tasks, with the exception of EMAQ having simi- lar performance on kitchen mixed. To ensure fair comparison with EMAQ, we use an autoregressive primitive policy. With the exception of EMAQ kitchen mixed, we are not aware of any existing offline RL algorithms that achieves similarly good performance on these tasks; moreover, we are not aware of any existing online RL algorithms which solve these tasks (see Table for some compar- isons), highlighting the benefit of using offline datasets to circumvent exploration challenges. There are two potential reasons for OPAL's success. First, temporally-extent the reward propagation learning problem easier. Second, the primitives may provide better latent action space than the atomic actions of the environment. To understand the relative importance of these factors, we experimented with an ablation of CQL+OPAL that uses 1 to remove temporal abstraction. In this case, we find the method's performance to be similar to standard CQL. This implies that the temporal abstraction provided by OPAL is one of the main contributing factors to its good performance. This observation also agrees with our theoretical analysis. See Appendix for detailed discussion.
primitives could make
5.2 FEW-SHOT IMITATION LEARNING WITH OPAL
Description: Previously, we assumed that we have access to task reward function, but only undi- rected data that performs other tasks. Now, we will study the opposite case, where we are not provided with a reward function for the new task either, but instead receive small number of task- specific demonstrations that illustrate optimal behavior. Simply imitating these few demonstrations insufficient to obtain good policy, and our experiments evaluate whether OPAL incorporate the prior data to enable few-shot adaptation in this setting. We use the Antmaze envi- ronments (diverse datasets) to evaluate method and use expert policy for these environments to sample n = 10 successful trajectories.
effectively
Behavior cloning (BC) and the model from Wang et al. (2017), which prescribes using a sequential VAE (SVAE) over state trajectories in conjunction with imitation learning. As shown in Table 2, BC+OPAL clearly outperforms other baselines, show-
Baseline and Results:
For baselines,
8

---

Published as a conference paper at ICLR 2021
Environment
HIRO
SAC+BC
(ACTOPAL(Ours)
DDQN+DDCO
antmaze medium sparse (diverse)
0.0
0.0
81.6 + 3.7
0.0
antmaze large sparse (diverse)
0.0
0.0
0.0
0.0
antmaze medium dense (diverse)
0.0
0.0
81.3 3.3
0.0
antmaze large dense (diverse)
12
0.0
81.5 3.9
0.0
Table 3: Average success rate (%) (over 4 seeds) of online RL methods: HIRO (Nachum et al.,
201 8a), SAC+BC, SAC+OPAL, and DDQN+DDCO (Krishnan et al., 2017). These methods were for 2.5e6 steps for antmaze medium environments and 17.5e6 steps for antmaze large environ- ments.
Models
MTT0
MT500
PPO
15.2 + 4.8
5.1 = 2.2
PPO+OPAL(OurS)
70.1 + 4.3
45.3 + 3.1
SAC
39.5
28.8
Table 4: Due to improved exploration, PPO+OPAL outperforms PPO and SAC on MT10 and MT50 in terms of average success rate (%) (over 4 seeds).
ing the importance of temporal abstraction and ascertaining the quality of learned primitives. See Appendix A for detailed discussion.
5.3 ONLINE RL AND MULTI-TASK TRANSFER WITH OPAL
Description: For online RL and multi-task transfer learning, we learn a task policy in space of primitives To (als, z) while keeping it fixed. For multi-task transfer, the task policy also takes in the task and we use = and = 8. Since the primitives need to transfer to a different state distribution for multi-task transfer, only learns the action sub-trajectory distribution and doesn't 't take in the state feedback. See Appendix A for a detailed description models. For online RL, we the Antmaze environments (diverse datasets) with sparse and dense rewards for evaluating method. For online multi-task transfer learning, we learn primitives with expert data from pick-and- place task and then 2020)), containing 10 and 50 robotic manipulation tasks which needs to be solved simultaneously.
learn multi-task policy for MT1O and MT50 (from metaworld (Yu al.,
Baseline and Results: For online RL, we use HIRO (Nachum al., 2018b), state-of-the-ar hicrarchical RL method, SAC (Haarnoja et al., 2018) with Behavior cloning (BC) pre-training on D. and Discovery of Continuous Options (DDCO) (Krishnan et al., 2017) which uses D learn discrete set of primitives and then learns a task policy in space of those primitives with online RL (Double DQN (DDQN) (Van Hasselt al., 2015)). For online multi-task transfer learning, we use PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) as baselines. As shown in Table 3 and Table 4, OPAL uses temporal abstraction to improve exploration and thus accelerate online RL and multi-task transfer learning. See Appendix A for detailed discussion.
6 DISCUSSION
We proposed Offline Primitives for Accelerating offline RL (OPAL) as a preproccesing step for ex- tracting recurring primitive behaviors from undirected and unlabelled dataset of diverse behaviors. We derived theoretical statements which describe under what conditions OPAL can improve learning of downstream offline RL tasks and showed how these improvements manifest in practice, leading significant improvements in complex manipulation tasks. We further showed empirical demonstra- tions of OPAL's application few-shot imitation learning, online RL, and online multi-task transfer learning. In this work, we focused on simple auto-encoding models for representing OPAL, and an interesting avenue for future work is scaling up this basic paradigm to image-based tasks.
ACKNOWLEDGEMENT
We would like to thank Ben Eysenbach and Kamyar Ghasemipour for valuable discussions dif- ferent points over the course of this work. This work was supported by Google, DARPA Machine Common Sense grant and MIT-IBM grant.

---

Published as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 22-31. JMLR. org, 2017.
Christopher G Atkeson, Benzun P Wisely Babu, Nandan Banerjee, Dmitry Berenson, Christoper Bove, Xiongyi Cui, Mathew DeDonato, Ruixiang Du, Siyuan Feng, Perry Franklin, et al. No falls, resets: Reliable humanoid behavior in the darpa roboties challenge. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 623 630. IEEE, 2015.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):
885-890, 2019. ISSN 0036-8075. doi: 10. www.iiinne..a224400...0 URL sciencemag - org/ content /65/ 6456/8 /885.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: 1910. 11215, 2019.
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: /2901, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian lbarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXhv:1802.06070, 2018.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven rein- forcement learning. In arXiv, 2020. URL httttts/////xiiiv org/ http/ddf/220000 07219.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900) 2018.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- max q-learming operator for simple yet effective offline and online rl. arXiv preprint arXiv:2007. 1091, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290. 2018.
K. Hausman, T. Springenberg. Z. Wang, N. Heess, and M. Riedmiller. Learning embedding space for transferable robot skills. In International Conference on Learning Representations (ICLR), 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reinforcement learning. In Advances in Neural Information Processing Systems, pp. 10519-10531, 2019.
Natasha Jaques, Asma Ghandcharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-polics batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 190iv:/907.000 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations. arXiv preprint arXiv: 171000542 2017.
10

---

Published as a conference paper at ICLR 2021
Ashish Kumar, Saurabh Gupta, and Jitendra Malik. Learning navigation subroutines from egocentric videos. In Conference on Robot Learning, pp. 617-626. PMLR, 2020a.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy q- learning via bootstrapping error reduction. Neural Information Prucessing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. N0::20006.0477, 2020b.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132, 2020.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofrr Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.03647, 2020.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv: 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01255, 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018b.
Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. Multi-agent manipula- tion via locomotion using hierarchical sim2real. arXiv preprint arXiv:1908.05220 2019a.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint 190. 10618, 2019b.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust the curse of chaos, arXiv preprint arXiv:1902.0/220 2019.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learn- ing composable hierarchical control with multiplicative compositional policies. In Advances in Neural Information Processing Systems, pp. 3686-3697, 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth
AAAI Conference on Artificial Intelligence, 2010.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406-3413 IEEE, 2016.
Martin I. Puterman. Markov decision processes: Discrete stochastic dynamic programming. 1994.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference.
arXiv preprint 2006 2020.
11

---

Published as a conference paper at ICLR 2021
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 19X7019077.1157,, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):444488) 2016.
Martin Stolle and Doina Precup. Learning options in reinforcement learning. volume 2371, pp- 212-223, 08 2002. doi:
1_16221116..
G. Tesauro. Temporal difference learning and td-gammon. Int. Comput. Games Assoc., 18:88, 1995.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, arXiv preprint arXiv:1509.0646// 2015.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv preprint arXiv: arXiv: 7703001 161, 2017.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, 2017.
pp.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.1136/ 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100, 2020.
12

---

Published as a conference paper at ICLR 2021
Appendices
A OTHER APPLICATIONS OF OPAL
A.1 FEW-SHOT IMITATION LEARNING WITH OPAL
Additional Assumption In addition to D, we assume access to a small number of expert demon- strations DexP = (T. == (s;,a))[=) T )==1 where n < N.
How to use with OPAL? In imitation learning, the aim is to recover expert policy given small number of stochastically sampled expert demonstrations Dexp = {T; (st, ())==>)))= Ji=1 As in the offline RL setting, we use the primitives To [a [a],, 8, 2) learned by OPAL as a low-level controller and learn high-level policy (=(s)). We first partition the expert demonstrations into sub-trajectories
T--1m
par {Tr, == (5k+t) a=++)==0 for k O, T e)= of length We then use the learned encoder g0 (2(r)) to label these uub-trajectories with latent actions 2,,k (2 and thus create dataset D {( Zi,A) for k = o,..., c)>- We use D to learn the high-level policy (2|s) using behavioral cloning. As in the offline RL setting, we also finetune To to(a|s, 2) with latent-conditioned behavioral cloning to ensure consistency of the labelled latents.
Evaluation Description: We receive small number of task-specific demonstrations that illustrate optimal behavior. Simply imitating these few demonstrations is insufficient to obtain a good policy, and our experiments evaluate whether OPAL can effectively incorporate the prior data to enable few- shot adaptation in this setting. We use the Antmaze environments (diverse datasets) to evaluate our method and use an expert policy for these environments to sample 10 successful trajectories.
Baseline: We evaluate two baselines. First, we test a simple behavioral cloning (BC) baseline, which trains using max-likelihood loss on the expert data. In order to make the comparison fair to OPAL (which uses the offline dataset D in addition to the expert dataset), we pretrain the BC agent on the undirected dataset using the same max- lix-liield loss. As second baseline and to test the quality of -ptteextracted primitives, we experiment with an alternative unsupervised objective from Wang et al. (2017), which prescribes using sequential VAE (SVAE) state trajectories in conjunction with imitation learning.
As shown Table 2, BC+OPAL clearly outperforms other baselines, showing the impor- lance of temporal abstraction and ascertaining the quality of learned primitives. SVAE's slightly worse performance suggests that decoding the state trajectory directly is more difficult than simply predicting the actions, as OPAL does, and that this degrades downstream task learning.
Results:
A.2 ONLINE RL WITH OPAL
Additional Assumptions: We assume online access to M; i.e., access to Monte Carlo samples of episodes from M given arbitrary policy æ.
How to use with OPAL? To apply OPAL to standard online RL, we fix the learned primi- tives (a]ss, 2) and learn a high-level policy (2(s))) in an online fashion using the latents as lemporally-extended actions. Specifically, when interacting the environment, (2|s)) chooses an appropriate primitive every e steps, and this primitive acts on the environment directly for e timesteps. Any off tffftth--shef online RL algorithm can be used to learn v.. In our experiments, we use SAC (Haarnoja et al., 2018). To ensure that ny. (2]s) stays close to the data distribution and avoid generalization issues associated with the fixed TT (alas =). we add an additional KL penalty to the reward of the form DKL ( (
Evaluation Description: We Antmaze medium (diverse dataset) and Antmaze large (diverse dataset) from the D4RL task suite (Fu et al., 2020) to evaluate our method. We evaluate using both a dense ietance-based reward -1|g - antxy |l and sparse success- success-based reward 1 [llg antxyll < 0.5] (the typical default for this task), where antxy is the 2d position of the ant in the maze.
Baseline: To solve these tasks through online RL, need both (i) hierarchy (i.e. learning a policy on top of primitives) which improves exploration (Nachum et al., 2019b) and (ii) unlabelled (i.e. no task reward) offline dataset which allows to bootstrap the primitives. This informed our choice
13

---

Published as a conference paper at ICLR 2021
of these three baselines. First, to test the role of D in exploration, use HIRO (Nachum al., 2018b), a state-of-the-ar hierarchical RL method, as a baseline. Second, to test the role of temporal abstraction, we pre-train flat policy on D using behavioral cloning (BC) and then finetune the pol- icy on downstream tasks with SAC. Third, to test the quality of extracted primitives for online RI,, extract discrete set of primitives with Deep Discovery of Continuous Options (DDCO) (Krish- nan et al., 2017) and use Double DQN (DDQN) (Van Hasselt et al., 2015) to learn a task policy in the space of learned discrete primitives.
As shown in Table 3, SAC+OPAL outperforms all the baselines, showing (1) the impor- tance of D exploration, (2) the role of temporal abstraction, and (3) the good quality of learned primitives. Except for HIRO on Antmaze large with dense rewards, all other baselines fail to make any progress all. In contrast, SAC+OPAL only fails to make progress on Antmaze large with sparse rewards.
Results:
A.3 ONLINE MULTI-TASK TRANSFER LEARNING WITH OPAL
Additional Assumption: We assume the existence of M additional MDPs {M; (S., A,P,, 7)) where the action space A, and the discount factor 7 are same as those of M.
How to use it with OPAL. In the multi-task setting, we aim to learn near-optimal behavior policies on M MDPs {M, (S,, A. Pa, } As in the previous applications of OPAL, we learn set of high-level policies Ty (z|ss, which will direct pretrained primitives (a z) maximize cumulative rewards. Since the state space in the M MDPs is potentially distinet from that in the offline dataset D. we cannot transfer the state distribution and can only hope to transfer the action sub-trajectory distribution. Therefore, during the unsupervised training phase for learning To, we make the encoder and the decoder blind to the states in sub-trajectory. Specifically, the encoder be- comes (ngnc, ( 20 st, (d++)/=0) and is represented by a bidirectional GRU. The decoder becomes To Te((
11----1Haa) which decodes the entire action sub-trajectory from the latent vector and is represented by a GRU. With these agnestii primitives in hand, we then learn policy T& (2|s,, i) using any off-the-shelf online RL method. In our experiments, we use Proximal Policy Optimization (PPO) (Schulman et al., 2017).
Evaluation Description: We use the Metaworld task suite (Yu et al., 2020) to evaluate our method.
The dataset DD for learning primitives consists of trajectories generated by an expert policy for goal-conditioned pick-and-place task. The pick-and-place task is suitable for unsupervised primitive learning because contains all the basic operations (eg: move, grasp, place) required for performing more complex manipulation tasks in the Metaworld. Once we have learned the primitives, we learn a policy Te (a|s, i) for the MT10 and MT50 benchmarks, where MT10 and MT50 contain 10 and 50 robotic manipulation tasks, respectively, which we need to be solved simultaneously. In these experiments we use 5 and dim Z) 8.
Baseline: We use SAC (Haarnoja et al., 2018) and PPO (Schulman et al., 2017) as baselines.
Results: As shown in Table 4, PPO+OPAL clearly outperforms both PPO and SAC, showing the importance of temporal abstraction in online multi-task transfer.
B PROOF OF THEOREMS
B. BOUNDING THE SUBOPTIMALITY OF THE LEARNED PRIMITIVES
We will begin by proving the following lemma which bounds the sampling error incurred by [d, w).
Lemma B.0.1. With high probability 1 8,
J(0, 0,w0) - -a..trrrmm
log To (arls) 2) s
//55
S1
(11)
where S.y is constant dependent on concentration properties of no (a) 8, 2) and go(2))).
14

---

Published as a conference paper at ICLR 2021
Proof. To be concise, let denote the sampling error in [ w,
A 10,,0,00 (e
log no (arls;,,
(12)
Applying Chebyshev's 's inequality to A,, we get that, with high probability 1 1 1-8,
Vatralllro Ei=o log vo (arlst;)))
SJJ
A S
(13)
Therefore, combining all the equations, we have
Σ log mo(a 2) s ((0,
SJ
(14)
We present a general performance difference lemma that will help our proof of Lemma 4.0.1.
Lemma B.0.2. If æı and л2 are two policies M, then
2 [J,((T,,M) - (T2, (1 7)2 Rmax DDv
(15)
Proof. Following the derivations in Achiam et al. (2017) and Nachum et al. (2018a), express the performance of policy a in M in terms of linear opterators:
(T, =1 (1-7)- (1 PI,,
(16)
where R is vector representation of the rewards of M, II, linear operator mapping state distributions to state-action distributions according to x, and is vector representation of the initial state distribution M. Accordingly, we express the performance difference mı T2 as,
(RL (T2, M)) = [R -11--111 P) (1 --111) [1224)
(17)
< Rmax [I - P) 11 - (1 -l1pp) 112
(18)
By the triangle inequality, we may bound equation 18 by
Rmax (ICT PI (I (Z--7/0 -1lpp) -1 + (1-Mi= (19)
We begin by approaching the first term inside the parentheses of equation 19. That first term may be expressed as
- (1-7I1P)- 111---------------1-1011---1111-
(20)
|y (I
(21)
I 7 ((11 - 112) -11 111
(22)
27
(23)
[1 7)
Now continue the second term inside the parentheses of equation 19. This second term may be expressed as
(1 (1-211+P) 1222 < (1I) 112)ul
(24)
2
(25)
15

---

Published as a conference paper at ICLR 2021
To combine equations 23 and 25, note that d =
(1- 7) n.
(26)
Thus, we have
2
() M))
(1 (1- Rmax Band (Pro(o(((((ssss)sllaaaaaa
(27)
as desired.
Lemma 4.0.1. If æı and ma are two policies M. then
2
(T,,
1 - (1- 7) Bmax (
(28)
where ( denotes the TV divergence over c-length sub-trajectories T sampled from vs. 22 (see section 3). Furthermore,
2 (1 7-) 77) R Roax
SubOpt(O) 5
(29)
Proof. We focus proving equation 28, as the subsequent derivation of equation 29 is straightfor- ward by definition of SubOpt.
To derive equation 28, may simply consider and acting in an "every-c-steps" version of M, where the action space now т and reward are accumulated over c steps using y-discounting.
Note that in this abstracted version M, the max reward is 1-2 I- Rmax and the MDP discount is 7. Plugging this into the result of Lemma B.0.2 immediately yields the desired claim.
Theorem 4.1. Let 0, ó, w be the outputs of solving equation 1, such that J(0, w, ec- Then, with high probability 1 8, for any that (-common in II, there exists a distribution H over z such # that for mg (T|s) = Ez- (To (T)2)))
SJ σ
E.w (Dwv (r(3s) c
Ec +
V
He
(30)
2
where He = =E--IL.w EE log (ar]s:)] (i.e. constant and property of DD and S is pos- itive constant incurred due to sampling error in (0, @, (,w) and depends on concentration properties of To(a)s, and (2)T))
Proof. We start with application of the triangle incquality:
(rv(T(r]s)] H (r(s)) < Drv(F(r(s))((((ls) + Drv(r(r|s)[I=?) (ris)).
(r(ss))
(31)
Taking expectation with respect II, 8 x on both the sides, we get
E,-x (Drv (r)) < E. Drv (rv(((r)ll<(T)) B Drov))))
(32)
< S (Drv(((>))
(33)
+ Eawll
(34)
<< + V (oonl log Eg [T]]]]
(35)
< ( + V (oo log no ( (T|2))
(36)
16

---

Published as a conference paper at ICLR 2021
The last two inequality used Jensen's inequality. Let H2) (0ool),rrll00(17)) Cancelling the dynamics and using equation 14 we get,
Esnx DTo (T( (r))
55+
R Bww.
(log ग (acls)) log (ogmo(aalls 2))
2
ST 1()
<(+
He + V
2
SJ
=
He + + #1
/8
2
d=
Corollary 4.1.1. If the optimal policy 't" of M is {-common in II and high probability
S E. then, with
25 SubOpt(O) E - 70 - 7) Rmax
5J 8
c+
c +
+ Ho
(37)
Proof. Expanding lemma 4.0.1 using the above assumption, we have SubOpt(O) |JRL " ,M) (RI( MM]
(38)
- - (1-)))(1- -)) Rmar anag (ri(s(())) (Tls))
Ba
(39)
d=
R Rmax
Eann [Dv (rls)l (r|s))
(40)
(1 - -0)g)((1 7)
K
26
EmaaeEy Drv (T (r) (r]s))
(41)
(1- (1-70)(1- 7)
Now, we can use theorem 4.1 to prove the corollary.
B.2 PERFORMANCE BOUNDS FOR OPAL
Theorem 4.2. Let Tg- (2|8) be the policy obtained by CQL and Tg- ((al)) refer to the policy when us- (z|8) is used together with na(als, (018,)) 2). Let TT = {T; III} refer to the policy generating
Dr in MDP M and TH (2|s) = T8, ~ (=(r)) Then, [[ p. M) 2 J J[T M)-K
with high probability 1 - 8 where
== 0
E
and Tu
[DI (T (T) + +1)
(42)
(1 7- 7)
and S (s) DDQu (s)]
(43)
Proof. We assume that the variational posterior q obtained after learning OPAL from D same (or nearly same) the true posterior p(=/T). de be used to define . (=] as #े = ~ nT; = ~ (z(r). This induces an MDP Mμ = (S, 2, P2, where
the inferred latent space for choosing primitives, P. and r= are the latent dynamics and reward function such that Site ~ P- (= =) St+i+1 ~ P(st4sta atti), atts ~ ( 2,) Vi E Σi=1 2'r((rre 7'r(stti, 414s), and of is the new discount factor ef-
(0, 1..... 1} and r. (st, fectively reducing the task horizon by a factor of c. x[ass, 2) is the primitive induced by Is and ns. Since do captures the true posterior, m(a 8, 2) the optimal primitive you learn and its
17

---

Published as a conference paper at ICLR 2021
autoencoding loss, under true expectation, is E Er | Σi= log [(ce 2)
Therefore, T ~ B8 = a = ~ x)),,)). Ty used to collect the data Dr which induces an empirical MDP. We refer to the empirical MDP induced by D" as M (S. A, P. where ()) = El _/===_== and p($) = Sen=p1/s0=2 We use g to get D from N
E(sa)>p ]===,,==ā]
Dr which induces another empirical MDP Mr. Using these definitions, we will try to bound (0: J(TT, M)).
Let's break [J [TT, into
[T 0, M) (TB, M)] [J , M) [ Ms))l
(44)
+ [( : Mu ,
(45)
+ [ ,H)) - -J(T,,
(46)
Since qs captures the true variational posterior, TB = 2 Tb..T x(- 2) and therefore,
[ MM) - (TT, M)) O. For bounding, , Mr - J MH)), we use theorem 3.6 from Kumar et al. (2020b) and apply it to Mи to get
[((To. ,
(47)
Cr,
7 Rm
2
1-g0 (1 (1- - 7)
snn Bnd (s)
[D(s)
Deou (Toi , (s) +1
(48)
E snnd Fun (s)
(T m [s)] K2
(49)
Now, we will try to bound [T a. M) J Mr) The only difference between the two is that the primitive Ta(a, =) is used in M and the primitive (als, is used in Mr. Therefore, write the above bound (= function at a particular state Using Lemma 4.0.1, we get
J (()))))) ((+l-s), M)]. Let's first bound their value
[T 0, M)
Ma)
(50)
2R mRmax E,
S (1 - (1 Bndd - Drr
(51)
2Rmax F.
V D (0
S (1 - (1 y) andd
(52)
Now, we will try to bound DKL (e(e)) To (T(ss))
We have
Tee (2|s)
P(s St
x (ag
Eswaye (=())s))
log
(53)
TV- (z|s)
P( st- a--1)
To (aaa 2)
= -E (1(1)))))
log π (ar 2) - log To (oe((8))))
(54)
TT- (2|s) n H (2|s)
E2 (1s)))
log (al log To (c(lss)))
(55)
t=U
Tw- (z)
log T (a,1s,, 2) - log To (a 2)
(56)
/ (2|s)
Tw (z|s)
SJ
c/ +
(57)
T3 (2|s)
18

---

Published as a conference paper at ICLR 2021
The last equation comes from above definition of c and equation 14. We will now try to bound
Ten (=]s) m (=|s)
using Pcou((Tp T3 (s). Using definition of DCQL "B / (s), we have
TTp (2)8) TH (z|s)
x=) = Σ Tw (2|8)
(58)
2
2
Tw- TT- (2|s) (z|s)
T (2|s)
DeQL(TTT mB (s)+ 1= Σ Tg (z|s)
M
(z|s)
(59)
3 (z|s)
[v=(2|e) (z)a)
where _ = arg max,
To be concise, let
SJ Eo +
4
(60)
8
Combining above equations, we have
D DeQL 33 (s)
DKL (2]5) (u (2(s))((())) S Dej
(61)
x (z|s)
Using this bound the returns, we get
(T M, (TT (()))))
(62)
2Rmax Ey oe Band
(Poa T (2)s)
Doeu(TT TT (s) +
(s)
< (1 (1 -)
V
A.
(63)
K1
(64)
We get K1 K2. We apply O to get the notation in the theorem.
C EXPERIMENT DETAILS
C.I OPAL EXPERIMENT DETAILS
Encoder The encoder qo (2|T) takes in state-action trajectory 7 of length c. It first passes the indi- vidual states through fully connected network with 2 hidden layers of size H and ReLU activation. Then it concatenates the proccessed states with actions and passes it through bidirectional GRU with hidden dimension of И and 4 GRU layers. projects the output GRU to mean and log standard deviation of the latent vector through linear layers.
Prior The prior (z(s) takes in the current state s and passes it through fully connected network with 2 hidden layers of size H and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of the latent vector through linear layers.
Primitive Policy The primitive (i.c. decoder) (a) has same architecture the Prior but it takes in state and latent vector and produces mean and log standard deviation of the action. For kitchen environments,
autoregressive primitive policy with architecture used by EMAQ (Ghasemipour et al., 2020).
We use H 200 for antmaze environments and H 256 for kitchen environments. In both cases, OPAL was trained for 100 epochs with a fixed learning rate of le - 3, B = 0.1 (Lynch et al., 2020), Adam optimizer (Kingma & Ba, 2014) and batch size of 50.
C.2 TASK POLICY ARCHITECTURE
In all environments, for task policy, we used fully connected network with 3 hidden layers of size 256 and ReLU activation. It then projects the output the hidden layers to mean and log standard deviation of the latent vector through linear layers.
19

---

Published as a conference paper at ICLR 2021
Environment
dim (2) =1
dim 2) =
dim 2)
antmaze medium (diverse)
68.7 - 2.3
81.1 3.1
81.3 = 1.8
Table 5: Average success rate (%) (over 4 seeds) of CQL+OPAL for different values of d2m We fix = 10.
C.3 SAC HYPERPARAMETERS
We used SAC (Haarnoja al., 2018) for online RL experiments in learning a task policy either in action space A or latent space Z. For the discrete primitives extracted from DDCO (Krishnan al., 2017), we used Double DQN Van Hasselt et (2015). We used the standard hyperparameters for SAC and Double DQN provided in rlkit code base (https: : //gttuub.bb./// commvvtccyyz rlkit) with both policy learning rate and q value learning rate as 3e 4.
C.4 CQL HYPERPARAMETERS
We used CQL (Kumar et al., 2020b) for offiin RI. experiments in learning task policy either in action space A or latent space 2. We used the standard hyperparameters, as mentioned Kumar al. (2020b)., with minor differences. We used policy learning rate of 3e 5, q value learning rate of 3e - 4, and primitive learning rate of 3e - 4. For antmaze tasks, we used CQL(H) variant with and learned a. For kitchen tasks we used CQL(p) variant with fixed 10. In both cases, we ensured a never dropped below 0.001.
D CONNECTION BETWEEN OPAL AND VAE OBJECTIVES
We are given an undirected, unlabelled and diverse dataset D of sub-trajectories of length c. We would like to fit sequential VAE model to D which maximizes
- max E [log pT(T[so)]
(65)
where is initial state of the sub-trajectory. Let's consider
log(e(r)so) log / po( log
/
po(T, ((00) de
(66)
(0(2/T)
(using Jensen's inequality)
f
logpe(T(2,00) - log (2(7))
(o(=)=))loppo(TT 2|s0) - log (z/T))]d = (z)qu(=1r)
(= (2(2s))
(67)
Using the above equation, we have the following lower-bound for our objective function
logpo(T|z, so) log (0(2/T)
max max D (o
2 max E.n.D
(68)
0,0
po
max E, 0.4
|log ( so)] DKI. (Kr.(qa(2l==)lpooeesss)
(69)
We separate the parameters of decoder from prior and hence write po(2(30) (=(20))). We can
expand log po (T(2,, su) = Σi=l log ((s a + Ei= log Te(a (s,, 2). Since P is fixed can be removed from the objective function. Therefore,
write the objective function as
Tc-1
Σ log (e(a z) - (exx
max 0,4
(70)
where 3 = 1. This is similar to the autoencoding loss function we described in section 4.
E ABLATION STUDIES
As shown in Table we experimented with different choices of dim(Z) on antmaze-man medium (diverse). Using the hyperparameters from Nachum al. (2018a), fixed 10. While
20

---

Published as a conference paper at ICLR 2021
dim(2) [3) 8, 16 gave similar performances, dim(2) 4 performed slightly worse Therefore, we selected dim(2) [ = for our final model as it was simpler.
due to temporal abstraction and not better action space learned through latent space, we tried с = (dim(2) 8) and found the performance to be similar that of CQL (i.e. 55.3 + 3.8) thereby empirically supporting the theoretical benefits of temporal abstraction.
Temporal abstraction actually helps: To empirically verify that the gain in performance
and 10 to work well with other environments well. However, acknowledge that the performance of CQL+OPAL can be further improved by carefully choosing better hyperparameters for each environment by using other offine hyperparameter selection methods for offline RL, which is a subject of future work.
We found dim(2)
F ALTERNATIVE METHODS FOR EXTRACTING PRIMITIVES FROM OFFLINE DATA
We describe alternative methods for extracting primitive policy from offline data. These methods are offline variants of CARML (Jabri et al., 2019) and DADS (Sharma et al., 2019). We tried these techniques in early phase of our project and used the environment antmaze-medium (diverse) to evaluate these methods.
Let's consider an offline undirected, unlabelled and diverse dataset D [ af) Let = (sx){=0 represent the state trajectory. To extract primitives, we first cluster the trajectories by maximizing the mutual information between the state trajectory T and latent variable (indicating cluster index) with respect to the parameters of joint distribution (T,
pw (=) (T|2). For ) (i.e. discrete latent variables sampled from Categorical
now, we consider (2) Cat PPP)
distribution) and represent as one-hot vector of dimension The choice of (2) consistent
with the choices made in Jabri et al. (2019) and Sharma al. (2019). Since is discrete, we can use
Bayes rule to calculate (r..((2 as
(=(2)p((T)2)
(2
(71)
E (2(2)
Our objective function becomes
log (T(
max (T:22) = max E D. (#/T) 6,50
(72)
0,.0
P(T)
Offline CARML and offline DADS differ only in how they model ps (T(2))
Offline CARML We model po(T(z) II Po(sl2)) and hence, log log2)(T(2)
logpo(stlz).
Offline DADS We model po p(so) I Po(su(sc-1,2) and hence, log po (T(22) p(so) + > log (x)):t1)))) Here, we only model pa pr 2) and not p(so).
Since, log is additive in nature, p(so) will be ignored while calculating gradient.
To optimize equation 72, we use Algorithm from Jabri et al. (2019). Once we have clustered the state trajectories J with labels by maximizing (TT:),, we can use behavioral cloning (BC) to learn T))
Finally, we use (2(T) to label the leward-labelled data D = {(s), af, r with latents, and transform it into Dí {s)) zi, Σi=] >'ri, "ri,.... ( J. The task policy ni is trained on Dí using Conservative Q Learning (CQL.) (Kumar al., 2020b). Since the primitive policy my is trained after (=IT) is fully trained, it doesn't It need any additional finetuning.
F.1 RESULTS
Using the hyperparameters from Nachum et al. (2018a), we used c = 10. We experimented with different values of & 10, 20 and found that k 10, 20 works the best (see Table 6 for more
21

---

Published as a conference paper at ICLR 2021
Models
= 5
10
k: 20
CQL+Offline DADS
31.4 + 5.7
59.1 + 3.1
59.6 2.9
CQL+Offlnee CARML
13.3 + 4.7
15.1 + 2.6
14.9 + 3.8
Table 6: Average success rate on antmaze medium (diverse) (%) (over 4 seeds) of CQL combined with offline DADS and offline CARML for different values of k.
Environment success
CQL
CQL+OPAL
CQL+ DADS
CQL+ CARMI.
rate cumulative dense reward cumulative dense
53.7 + 6.1
81.1 = 3.1
59.1 + 3.1
15.1 2.6
-12138.6 + 720.3
-7795.7 = 535.4
-1184.77 = 610.1
-13387.3 = 710
reward (last steps)
-45.1 + 9.2
-7.8 + 4.6
-33.1 = 8.1
-51.6 = 5.6
Table 7: Average success rate (%), cumulative dense reward, and cumulative dense reward (last steps) (over 4 seeds) of CQL combined with different offline skill discovery methods on antmaze medium (diverse). For CQL+ + (Offline) DADS and CQL + (Offtine) CARML, we use k = 10. Note that CQL+OPAL outperforms both other methods for unsupervised skill discovery on all of these different evaluation metrics.
details). We went with A: = 10 as our final model since it's simpler. Offline CARMI effectively uses only skills as the other 4 skills had (=) = Offline DADS uses all the skills. The results are described in Table 7. In addition to calculating the average success rate, we also calculate the average cumulative dense rewards for entire trajectory and the average cumulative dense rewards for the last 5 time steps. Here, the dense reward is negative l2 distance to the goal. The resulting trajectory clusters (using a subset of the dataset) from discrete skills are also visualized in Figure where different colors represent different clusters.
Since offline CARML treats the states in the trajectory conditionally independent of each other given 2, the clustering mainly focuses the spatial location. Therefore, offline CARML isn't able to separate out different control modes starting around the same spatial locations which explains its poor performance when combined with CQL. As we can see from Figure 5, offline CARML is able to make progress towards the goal, but gets stuck along the way due to poor separation of control modes. On the other hand, offline DADS treats the state transitions the trajectory conditionally in- dependent of each other given 2 and thus clusters trajectories with similar state transitions together. This allows more effectively separate out the control modes. Therefore, CQL+offline DADS slightly improves upon CQL but is still limited by discrete number of skills. Furthermore, increasing the number of skills from 10 to 20 gives similar performance. Moreover, these methods, it's Bayes rule to calculate po.. (z/r). Therefore,
tractable use continuous skill space since decided to switch to learning a 3-VAE (Higgins et al., 2016) style generative model with continuous skill space i.e. OPAL.
(a) Offline DADS
(b) Offine CARML
Figure 4: Visualization of (a subset of) dataset trajectories colored according to their assigned cluster
using (a) Offline DADS and (b) Offline CARML. We use k 10.
22

---

Published as a conference paper at ICLR 2021
(a) CQL
(b) CQL+OPAL
(c) CQL+Offline DADS (d) CQL+Offine CARML
Figure 5: State visitation heatmaps for antmaze medium policies learned using (I) CQL, (2)
CQL+OPAL, (2) CQL+Offine DADS and (4) CQL+Offline CARML. Note that while offline CARML and offline DADS get stuck at various corners of the maze, OPAL is able to find its path through the maze to the goal location on the top right.
F.2 TRAINING DETAILS
For clustering by optimizing equation 72, both offline CARML and offline DADS only considers the global x-y pose of the ant and ignores other dimensions of the state space. These methods fail to work when considering the full state space.
Offline CARML (s|2) takes the latent one-hot vector and passes it through a fully connected network with 2 hidden layers of size H = 200 and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of the reduced state (only global x-y pose considered) through linear layers.
Offiine DADS (s 2) has the same architecture as (s(() but also takes in the reduced state from the previous timestep.
The primitive policy Th (a|s, =) takes the current state and latent one-hot vector and passes it through fully connected network with 2 hidden layers of size H = 200 and ReLU activation. then projects the output of the hidden layers to mean and log standard deviation of action through linear layers.
Primitive Policy
We perform the clustering for 25 epochs with a fixed learning rate le - 3, Adam opti- mizer (Kingma & Ba, 2014) and batch size of 50 using the Algorithm from Jabri al. (2019).
Task Policy For task policy (s), we used a fully connected network with 3 hidden layers of size 256 and ReLU activation. It then projects the output of the hidden layers to the logits (corresponding to the components of discrete latent space) through linear layers.
CQL Hyperparameters We used the standard hyperparameters for CQL(H) with discrete action space, as mentioned in Kumar et al. (2020b).
23


Reviews:
[{"review_id": "bDV_GCn84xx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the RL setting, this paper tackles the case where an agent may have access to large amounts of offline experience data. The objective of the work is to find an effective way to leverage this data for finding temporally extended primitive behaviors. The paper provides results that show how performing offline primitive learning can be leveraged for improving few-shot imitation learning as well as exploration and transfer on a variety of benchmark domains.\n\nThe paper tackles an important question in reinforcement learning: learning temporally extended primitive behaviors from off-policy data is a very relevant question. However, I found the motivation for the approach quite vague as well as different elements that require clarification (see below) and because of this, I can't recommend acceptance.\n\nMotivation for the approach: \n- if the downstream policy has to stay close to the offline data distribution, it seems to me that if the offline data distribution is obtained with a bad policy, this can not lead to interesting decision-making.\n- can you explain why equation 2 \"motivates better generalization\"?\n\nOther concerns:\n- Why can gamma not have a value of 0 (first paragraph preliminaries)\n- What does it mean \"To capture multi-modality in a dataset\" ? (second paragraph preliminaries)\n- Figure 1:  What does the (81.1%) and (70.3%) refer to?\n- Table 4: why is SAC given without standard deviation?\n\nSome text improvements:\n- \"we focus on focus on\"\n\nOther comment:\n- The number of seeds (4) should ideally be increased.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4-vXDXw39l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning\n### Summary\nThe authors present OPAL, an offline reinforcement learning approach that distills useful common behaviors from offline transition data. They leverage a variational GRU to encode trajectories into a latent space of primitive policies. These learned primitives can later on be transferred to other tasks by learning a high-level controller over the extracted primitive latent space. Their experimental results show that OPAL compares favorably against other SOTA methods in offline, online and few-shot reinforcement learning settings.\n\nOverall, the paper is clearly written, and the approach is potentially practical for real-world applications, due to its good performance on both offline and few-shot adaptation settings. \n\n### Strength\n- The proposed method can leverage large unlabeled data without pre-defined reward functions.\n- The property of OPAL is mathematically analyzed.\n- OPAL can potentially benefit from the advancement of offline RL algorithms by simply replacing CQL.\n- Can be adapted to a new task through few-shot imitation learning.\n\n### Weakness\n- Training details are missing for the OPAL encoder.\n- Not exactly a weakness, but it would be nice to see if the method works in real-world settings (real-world on/offline data). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0gVhohYtfQV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\n\nTo best leverage diverse datasets of logged experience, the authors\npropose to extract a space of primitive behaviors in a continuous space,\nand to use these for downstream learning. The primitive behaviors are\nlearned through a VAE loss, and CQL is applied to learn a policy over\nthe primitives. The authors claim that this approach to offline RL\navoids known distribution shift and allows for temporal abstraction. The\nmethod is also applied to few-shot imitation learning, exploration and\ntransfer to online RL.\n\nDecision\n--------\n\nDespite this paper being an interesting read, I feel that my concerns\nabout the experiments lead me to not be confident in the proposed\napproach. As such, my preliminary rating for the paper is \"Okay but not\ngood enough\". The baselines chosen for the experiments do not seem\nrepresentative of the problem being addressed. This also leads into the\nmotivation, where OPAL is motivated from the offline RL perspective but\ndoes not explicitly mitigate the issues in offline RL. Only the\ntheoretical results investigate the effect of a fixed replay-buffer, but\neven these claims are framed in terms of what $\\mathcal{D}$ should be to\nensure downstream RL.\n\nOriginality\n-----------\n\nThe proposed approach seems limited in its novelty, combining approaches\nfrom skill-discovery, hierarchical RL and somewhat from offline RL. The\ntheory section is interesting, however the technical arguments seem\nsimilar to that of Nachum et al. (2018).\n\nQuality and Clarity\n-------------------\n\nThe overall quality of the paper is very high. The writing is clear, and\nthe theoretical arguments are put into an easily understandable context.\n\nStrengths\n---------\n\n-   OPAL leverages techniques from many areas of machine learning:\n    unsupervised learning, hierarchical RL and offline RL. This is an\n    interesting combination, and deserves to be investigated. It seems\n    like the workhorse of OPAL is the unsupervised learning component\n    however, and perhaps this should be emphasized and investigated\n    independently.\n-   The theoretical section is very clear and well contextualized. The\n    claims in the paper seem correct, and they do provide much needed\n    validation for hierarchical RL. The assumption of an optimal\n    high-level controller seems strong, but the analysis is nonetheless\n    interesting.\n\nWeaknesses\n----------\n\n-   Despite Figure 2, OPAL is difficult to disentangle with many moving\n    parts. It doesn't help that there is no discussion of how\n    hyperparameters were chosen for each component, or an ablation study\n    to investigate different hierarchical RL approaches, fine-tuning,\n    offline RL algorithms, or VAE losses.\n-   The proposed method is not specifically designed to leverage\n    anything particular in offline RL. As you show, it can be applied to\n    online RL. However, why is this motivated from the offline\n    perspective? OPAL itself does not seem to mitigate distributional\n    shift.\n-   It would be helpful to directly compare to other skill-discovery\n    methods that have been applied to online RL but can be adapted to\n    offline RL. For example, what prevents the work by Campos et\n    al. (2020) and the baselines therein to be applied in conjunction\n    with an offline RL algorithm? The baselines for the experiments do\n    not seem representative of the problem you are addressing. You use\n    standard offline RL algorithms, yet claim that the temporal\n    abstraction of skill discovery is crucial to the results. As such,\n    you should compare OPAL to other skill-discovery algorithms that can\n    be combined with offline RL.\n\nDetailed Comments\n-----------------\n\n-   Section 4.1: \" Prior: ρω(z|s0) tries to predict the encoded\n    distribution of the sub-trajectory..\" Prior doesn't seem like the\n    right word here, since it is being learned.\n-   Section 4.2: In what way is the task-specific policy $\\pi_\\psi$\n    different from the learned prior in Section 4.1? Both are\n    distributions over latent $z$ conditioned on a state. While the\n    prior is designed to only be conditioned on the initial state, this\n    is still quite similar to the high-level policy $\\pi_\\psi$.\n-   Section 4.2: \"\\[to\\] ensure that the c-step transitions remain\n    consistent … with the labelled latent action $z_i$\" Why is this\n    necessary, should CQL ensure consistency at the action-level, while\n    the latent actions are consistent by design?\n-   Corollary 4.1.1: How would $\\epsilon$ actually approach\n    $\\mathcal{H}_c$ in the algorithm? The condition is never explicitly\n    enforced because you formulate it as a constrained optimization\n    problem and you are never able to change $\\mathcal{H}_c$ since it is\n    a constant.\n-   Section 5.1, baselines: shouldn't baselines be compared to different\n    unsupervised skill discovery algorithms, paired with offline RL\n    algorithms? BEAR and EMAQ are offline RL algorithms without any\n    temporal abstraction. And as you say in the results section with an\n    ablation study, this temporal abstraction is crucial.\n-   Section 5.1, results: An ablation study is discussed but I cannot\n    find the corresponding results table/figure in the paper.\n-   Section 5.2, Table 3: With so many 0.0's, this leads me to believe\n    that the baselines are quite weak, or the problem is too hard for\n    the baselines. How were the baselines chosen. For example, why was\n    DDCO paired with DDQN instead of SAC?\n\nMinor Comments\n--------------\n\n-   Section 5.2 mis-capitalized We \"in this setting We use the Antmaze\n    environ-\"\n-   Section 5.3, Table 4: why are there no standard errors for SAC?\n\nPost Rebuttal\n--------------\nAfter discussion with the authors, I have decided to increase my score to a 6. The authors have addressed many of my concerns with respect to motivation, theoretical analysis and empirical evidence. As it stands, I still think OPAL is hampered by the many \"moving parts\" involved. The theoretical analysis and empirical evidence suggests a very effective approach however, and should lead to further work combining variational sequence encoding techniques for primitive extraction in RL. My decision would be to \"accept the paper if there is room\".\n\nReferences\n--------------\nCampos, Víctor, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i-Nieto, and Jordi Torres. 2020. “Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills.” *arXiv:2002.03647*.\n<http://arxiv.org/abs/2002.03647v4>.\n\nNachum, Ofir, Shixiang Gu, Honglak Lee, and Sergey Levine. 2018. “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning.” *arXiv:1810.01257*. <http://arxiv.org/abs/1810.01257v2>.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting mix of new theoretical and empirical results showing how learning temporally extended primitive behaviors can help improve offline (batch) RL.

Although 2/3 reviewers initially raised concerns regarding the motivation of the approach and some of the choices that were made, the authors did an excellent job at addressing these concerns in detail, and there is now a consensus towards acceptance.

I consider that this work is a meaningful contribution towards better offline RL, which is definitely a very important use case in practice. The authors have given convincing explanations to motivate their approach, and made several improvements to the paper. As a result, I am recommending it for acceptance, as a poster.
Decision: Accept (Poster)