ERATING OFFLINE REINFORCEMENT LEARNING
Year: 2021

Abstract:
Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent’s ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning. Primitives extracted in this way serve two purposes: they delineate the behaviors that are supported by the data from those that are not, making them useful for avoiding distributional shift in offline RL; and they provide a degree of temporal abstraction, which reduces the effective horizon yielding better learning in theory, and improved offline RL in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning as well as exploration and transfer in online RL on a variety of benchmark domains. Visualizations and code are available at https://sites.google.com/view/opal-iclr

heading: 1 INTRODUCTION
text: Reinforcement Learning (RL) systems have achieved impressive performance in a variety of online settings such as games (Silver et al., 2016; Tesauro, 1995; Brown & Sandholm, 2019) and robotics (Levine et al., 2016; Dasari et al., 2019; Peters et al., 2010; Parmas et al., 2019; Pinto & Gupta, 2016; Nachum et al., 2019a), where the agent can act in the environment and sample as many transitions and rewards as needed. However, in many practical applications the agent’s ability to continuously act in the environment may be severely limited due to practical concerns (DulacArnold et al., 2019). For example, a robot learning through trial and error in the real world requires costly human supervision, safety checks, and resets (Atkeson et al., 2015), rendering many standard online RL algorithms inapplicable (Matsushima et al., 2020). However, in such settings we might instead have access to large amounts of previously logged data, which could be logged from a baseline hand-engineered policy or even from other related tasks. For example, in self-driving applications, one may have access to large amounts of human driving behavior; in robotic applications, one might have data of either humans or robots performing similar tasks. While these offline datasets are often undirected (generic human driving data on various routes in various cities may not be directly relevant to navigation of a specific route within a specific city) and unlabelled (generic human driving data is often not labelled with the human’s intended route or destination), this data is still useful in that it can inform the algorithm about what is possible to do in the real world, without the need for active exploration.
In this paper, we study how, in this offline setting, an effective strategy to leveraging unlabeled and undirected past data is to utilize unsupervised learning to extract potentially useful and temporally extended primitive skills to learn what types of behaviors are possible. For example, consider a dataset of an agent performing undirected navigation in a maze environment (Figure 1). While the dataset does not provide demonstrations of exclusively one specific point-to-point navigation task,
∗Work done during an internship at Google Brain
it nevertheless presents clear indications of which temporally extended behaviors are useful and natural in this environment (e.g., moving forward, left, right, and backward), and our unsupervised learning objective aims to distill these behaviors into temporally extended primitives. Once these locomotive primitive behaviors are extracted, we can use them as a compact constrained temporallyextended action space for learning a task policy with offline RL, which only needs to focus on task relevant navigation, thereby making task learning easier. For example, once a specific point-to-point navigation is commanded, the agent can leverage the learned primitives for locomotion and only focus on the task of navigation, as opposed to learning locomotion and navigation from scratch.
We refer to our proposed unsupervised learning method as Offline Primitives for Accelerating offline reinforcement Learning (OPAL), and apply this basic paradigm to offline RL, where the agent is given a single offline dataset to use for both the initial unsupervised learning phase and then a subsequent task-directed offline policy optimization phase. Despite the fact that no additional data is used, we find that our proposed unsupervised learning technique can dramatically improve offline policy optimization compared to performing offline policy optimization on the raw dataset directly. To the best of our knowledge, ours is the first work to theoretically justify and experimentally verify the benefits of primitive learning in offline RL settings, showing that hierarchies can provide temporal abstraction that allows us to reduce the effect of compounding errors issue in offline RL. These theoretical and empirical results are notably in contrast to previous related work in online hierarchical RL (Nachum et al., 2019b), which found that improved exploration is the main benefit afforded by hierarchically learned primitives. We instead show significant benefits in the offline RL setting, where exploration is irrelevant.
Beyond offline RL, and although this isn’t the main focus of the work, we also show the applicability of our method for accelerating RL by incorporating OPAL as a preprocessing step to standard online RL, few-shot imitation learning, and multi-task transfer learning. In all settings, we demonstrate that the use of OPAL can improve the speed and quality of downstream task learning.

heading: 2 RELATED WORK
text: Offline RL. Offline RL presents the problem of learning a policy from a fixed prior dataset of transitions and rewards. Recent works in offline RL (Kumar et al., 2019; Levine et al., 2020; Wu et al., 2019; Ghasemipour et al., 2020; Jaques et al., 2019; Fujimoto et al., 2018) constrain the policy to be close to the data distribution to avoid the use of out-of-distribution actions (Kumar et al., 2019; Levine et al., 2020). To constrain the policy, some methods use distributional penalties, as measured by KL divergence (Levine et al., 2020; Jaques et al., 2019), MMD (Kumar et al., 2019), or Wasserstein distance (Wu et al., 2019). Other methods first sample actions from the behavior policy and then either clip the maximum deviation from those actions (Fujimoto et al., 2018) or just use those actions (Ghasemipour et al., 2020) during the value backup to stay within the support of the offline data. In contrast to these works, OPAL uses an offline dataset for unsupervised learning of a continuous space of primitives. The use of these primitives for downstream tasks implicitly constrains a learned primitive-directing policy to stay close to the offline data distribution. As we demonstrate in our experiments, the use of OPAL in conjunction with an off-the-shelf offline RL algorithm in this way can yield significant improvement compared to applying offline RL to the dataset directly.
Online skill discovery. There are a number of recent works (Eysenbach et al., 2018; Nachum et al., 2018a; Sharma et al., 2019) which use unsupervised objectives to discover skills and use the discovered skills for planning (Sharma et al., 2019), few-shot imitation learning, or online RL (Eysenbach et al., 2018; Nachum et al., 2018a). However, these works focus on online settings and assume access to the environment. In contrast, OPAL focuses on settings where a large dataset of diverse behaviors is provided but access to the environment is restricted. It leverages these static offline datasets to discover primitive skills with better state coverage and avoids the exploration issue of learning primitives from scratch.
Hierarchical policy learning. Hierarchical policy learning involves learning a hierarchy of policies where a low-level policy acts as primitive skills and a high-level policy directs the low-level policy to solve a task. While some works (Bacon et al., 2017; Stolle & Precup, 2002; Peng et al., 2019) learn a discrete set of lower-level policies, each behaving as a primitive skill, other works (Vezhnevets et al., 2017; Nachum et al., 2018b; 2019a; Hausman et al., 2018) learn a continuous space of primitive skills representing the lower-level policy. These methods have mostly been applied in online settings. However, there have been some recent variants of the above works (Lynch et al., 2020; Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) which extract skills from a prior dataset and using it for either performing tasks directly (Lynch et al., 2020) or learning downstream tasks (Shankar & Gupta, 2020; Krishnan et al., 2017; Merel et al., 2018) with online RL. While OPAL is related to these works, we mainly focus on leveraging the learned primitives for asymptotically improving the performance of offline RL; i.e., both the primitive learning and the downstream task must be solved using a single static dataset. Furthermore, we provide performance bounds for OPAL and enumerate the specific properties an offline dataset should possess to guarantee improved downstream task learning, while such theoretical guarantees are largely absent from existing work.

heading: 3 PRELIMINARIES
text: We consider the standard Markov decision process (MDP) setting (Puterman, 1994), specified by a tupleM = (S,A,P, µ, r, γ) where S represents the state space, A represents the action space, P(s′|s, a) represents the transition probability, µ(s) represents the initial state distribution, r(s, a) ∈ (−Rmax,Rmax) represents the reward function, and γ ∈ (0, 1) represents the discount factor. A policy π in this MDP corresponds to a function S → ∆(A), where ∆(A) is the simplex over A. It induces a discounted future state distribution dπ , defined by dπ(s) = (1 − γ)∑∞t=0 γtP(st = s|π), where P(st = s|π) is the probability of reaching the state s at time t by running π on M. For a positive integer k, we use dπk (s) = (1 − γk) ∑∞ t=0 γ
tkP(stk = s|π) to denote the everyk-step state distribution of π. The return of policy π in MDP M is defined as JRL(π,M) =
1 1−γEs∼dπ,a∼π(a|s)[r(s, a)]. We represent the reward- and discount-agnostic environment as a tuple E = (S,A,P, µ). We aim to use a large, unlabeled, and undirected experience dataset D := {τ ri := (st, at)c−1t=0}Ni=1 associated with E to extract primitives and improve offline RL for downstream task learning. To account for the fact that the dataset D may be generated by a mixture of diverse policies starting at diverse initial states, we assume D is generated by first sampling a behavior policy π ∼ Π along with an initial state s ∼ κ, where Π, κ represent some (unknown) distributions over policies and states, respectively, and then running π on E for c time steps starting at s0 = s. We define the probability of a sub-trajectory τ := (s0, a0, . . . , sc−1, ac−1) in D under a policy π as π(τ) = κ(s0) ∏c−1 t=1 P(st|st−1, at−1) ∏c−1 t=0 π(at|st), and the conditional probability as
π(τ |s) = 1[s = s0] ∏c−1 t=1 P(st|st−1, at−1) ∏c−1 t=0 π(at|st). In this work, we will show how to apply unsupervised learning techniques to D to extract a continuous space of primitives πθ(a|s, z), where z ∈ Z , the latent space inferred by unsupervised learning. We intend to use the learned πθ(a|s, z) to asymptotically improve the performance of offline RL for downstream task learning. For offline RL, we assume the existence of a dataset Dr := {τ ri := (st, at, rt)c−1t=0}Ni=1, corresponding to the same sub-trajectories in D labelled with MDP rewards. Additionally, we can use the extracted primitives for other applications such as few-shot imitation learning, online RL, and online multi-task transfer learning. We review the additional assumptions for these applications in Appendix A.

heading: 4 OFFLINE RL WITH OPAL
text: In this section, we elaborate on OPAL, our proposed method for extracting primitives from D and then leveraging these primitives to learn downstream tasks with offline RL. We begin by describing our unsupervised objective, which distills D into a continuous space of latent-conditioned and temporally-extended primitive policies πθ(a|s, z). For learning downstream tasks with offline RL, we first label Dr with appropriate latents using the OPAL encoder qφ(z|τ) and then learn a policy πψ(z|s) which is trained to sample an appropriate primitive every c steps to optimize a specific task, using any off-the-shelf offline RL algorithm. A graphical overview of offline RL with OPAL is shown in Figure 2. While we mainly focus on offline RL, we briefly discuss how to use the learned primitives for few-shot imitation learning, online RL, and multi-task online transfer learning in section 5 and provide more details in Appendix A.

heading: 4.1 EXTRACTING TEMPORALLY-EXTENDED PRIMITIVES FROM DATA
text: We would like to extract a continuous space of temporally-extended primitives πθ(a|s, z) from D which we can later use as an action space for learning downstream tasks with offline RL. This would reduce our effective task horizon, thereby making the downstream learning easier, as well as allow the downstream policy to stay close to the offline data distribution, thereby bringing stability to the downstream learning. We propose the following objective for learning πθ, incorporating an auto-encoding loss function with a KL constraint to encourage better generalization:
min θ,φ,ω J(θ, φ, ω) = Êτ∼D,z∼qφ(z|τ)
[ − c−1∑
t=0
log πθ(at|st, z) ]
(1)
s.t. Êτ∼D[DKL(qφ(z|τ)||ρω(z|s0))] ≤ KL (2) where Ê indicates empirical expectation. The learned components of this objective may be interpreted as encoder, decoder, and prior:
Encoder: qφ(z|τ) encodes the trajectory τ of state-action pairs into a distribution in latent space and gives out parameters of that distribution. In our case, we represent qφ with a bidirectional GRU which takes in τ and gives out parameters of a Gaussian distribution (µencz , σ enc z ).
Decoder (aka Primitive Policy): πθ(a|s, z) is the latent-conditioned policy. It maximizes the conditional log-likelihood of actions in τ given the state and the latent vector. In our implementation, we parameterize it as a feed-forward neural network which takes in current state and latent vector and gives out parameters of a Gaussian distribution for the action (µa, σa).
Prior/Primitive Predictor: ρω(z|s0) tries to predict the encoded distribution of the sub-trajectory τ from its initial state. Our implementation uses a feed-forward neural network which takes in the initial state and gives out parameters of a Gaussian distribution (µprz , σ pr z ).
KL-constraint (Equation 2). As an additional component of the algorithm, we enforce consistency in the latent variables predicted by the encoder qφ(z|τ) and the prior ρω(z|s0). Since our goal is to obtain a primitive z that captures a temporal sequence of actions for a given sub-trajectory τ = (s0, a0, · · · , sc−1, ac−1) (as defined in Section 3), we utilize a regularization that enforces the distribution, qφ(z|τ) to be close to just predicting the primitive or the latent variable z given the start state of this sub-trajectory, s0 (i.e. ρω(z|s0)). This conditioning on the initial state regularizes the distribution qφ(z|τ) to not overfit to the the complete sub-trajectory τ as the same z should also be predictable only given s0. The above form of KL constraint is inspired from past works (Lynch et al., 2020; Kumar et al., 2020a). In particular Lynch et al. (2020) add a KL-constraint (Equation 2, “Plan prior matching” in Lynch et al. (2020)) that constrains the distribution over latent variables computed only given the initial state and the goal state to the distribution over latent variables computed using the entire trajectory. Our form in Equation 2 is similar to this prior except that we do not operate in a goal-conditioned RL setting and hence only condition ρω on the initial state s0.
In practice, rather than solving the constrained optimization directly, we implement the KL constraint as a penalty, weighted by an appropriately chosen coefficient β. Thus, one may interpret our unsupervised objective as using a sequential β-VAE (Higgins et al., 2016). However, as mentioned above, our prior is conditioned on s0 and learned as part of the optimization because the set of primitives active in D depends on s0. If β = 1, OPAL is equivalent to a conditional VAE maximizing log probability of τ conditioned on its initial state s0; see Appendix D for more details. Despite the similarities between our proposed objective and VAEs, our presentation of OPAL as a constrained auto-encoding objective is deliberate. As we will show in Section 4.3, our theoretical guarantees depend on a well-optimized auto-encoding loss to provide benefits of using learned primitives πθ for downstream tasks. In contrast, a VAE loss, which simply maximizes the likelihood of observed data, may not necessarily provide a benefit for downstream tasks. For example, if the data can be generated by a single stationary policy, a VAE-optimal policy πθ can simply ignore the latent z, thus producing a degenerate space of primitives. In contrast, when the KL constraint in our objective is weak (i.e., KL 0 or β < 1), the auto-encoding loss is encouraged to find a unique z for distinct τ to optimize reconstruction loss.

heading: 4.2 OFFLINE RL WITH PRIMITIVES FOR DOWNSTREAM TASKS
text: After distilling learned primitives from D in terms of an encoder qφ(z|τ), a latent primitive policy (or decoder) πθ(a|s, z), and a prior ρω(z|s0), OPAL then applies these learned models to improve offline RL for downstream tasks.
As shown in Figure 2, our goal is to use a dataset with reward labeled sub-trajectories Dr = {τi := (sit, a i t, r i t) c−1 t=0}Ni=1 to learn a behavior policy π that maximizes cumulative reward. With OPAL, we use the learned primitives πθ(a|s, z) as low-level controllers, and then learn a high-level controller πψ(z|s). To do so, we relabel the dataset Dr in terms of temporally extended transitions using the learned encoder qφ(z|τ). Specifically, we create a dataset Drhi = {(si0, zi, ∑c−1 t=0 γ trit, s i c)}Ni=1, where zi ∼ qφ(·|τi). Given Drhi, any off-the-shelf offline RL algorithm can be used to learn πψ(z|s) (in our experiments we use CQL (Kumar et al., 2020b)). As a way to ensure that the c-step transitions τi := (s i t, a i t, r i t) c−1 t=0 remain consistent with the labelled latent action zi, we finetune πθ(a|s, z) on Drlo = {((sit, ait)c−1t=0 , zi)}Ni=1 with a simple latent-conditioned behavioral cloning loss:
min θ Ê(τ,z)∼Drlo
[ − c−1∑
t=0
log πθ(at|st, z) ] . (3)

heading: 4.3 SUBOPTIMALITY AND PERFORMANCE BOUNDS FOR OPAL
text: Now, we will analyze OPAL and derive performance bounds for it in the context of offline RL, formally examining the benefit of the temporal abstraction afforded by OPAL as well as studying what properties D should possess so that OPAL can improve downstream task performance. As explained above, when applying OPAL to offline RL, we first learn the primitives πθ(a|s, z) using D, and then learn a high-level task policy πψ(z|s) in the space of the primitives. Let πψ∗(z|s) be the optimal task policy. Thus the low-level πθ and high-level πψ∗ together comprise a hierarchical policy, which we denote as πθ,ψ∗ . To quantify the performance of policies obtained from OPAL,
we define the notion of suboptimality of the learned primitives πθ(a|s, z) in an MDP M with an associated optimal policy π∗ as
SubOpt(θ) := |JRL(π∗,M)− JRL(πθ,ψ∗ ,M)|. (4) To relate SubOpt(θ) with some notion of divergence between π∗ and πθ,ψ∗ , we introduce the following performance difference lemma.
Lemma 4.0.1. If π1 and π2 are two policies inM, then
|JRL(π1,M)− JRL(π2,M)| ≤ 2
(1− γc)(1− γ)RmaxEs∼d π1 c [DTV(π1(τ |s)||π2(τ |s))], (5)
where DTV(π1(τ |s)||π2(τ |s)) denotes the TV divergence over c-length sub-trajectories τ sampled from π1 vs. π2 (see section 3). Furthermore,
SubOpt(θ) ≤ 2 (1− γc)(1− γ)RmaxEs∼dπ ∗ c [DTV(π ∗(τ |s)||πθ,ψ∗(τ |s))]. (6)
The proof of the above lemma and all the following results are provided in Appendix B.1.
Through above lemma, we showed that the suboptimality of the learned primitives can be bounded by the total variation divergence between the optimal policy π∗ inM and the optimal policy acting through the learned primitives πθ,ψ∗ . We now continue to bound the divergence between π∗ and πθ,ψ∗ in terms of how representative D is of π∗ and how optimal the primitives πθ are with respect to the auto-encoding objective (equation 1). We begin with a definition of how often an arbitrary policy appears in Π, the distribution generating D: Definition 1. We say a policy π inM is ζ-common in Π if Eπ∼Π,s∼κ[DTV(π(τ |s)||π(τ |s))] ≤ ζ. Theorem 4.1. Let θ, φ, ω be the outputs of solving equation 1, such that J(θ, φ, ω) = c. Then, with high probability 1 − δ, for any π that is ζ-common in Π, there exists a distribution H over z such that for πHθ (τ |s) := Ez∼H [πθ(τ |z, s)],
Es∼κ[DTV(π(τ |s)||πHθ (τ |s))] ≤ ζ + √√√√1 2 ( c + √ SJ δ +Hc )
(7)
where Hc = Eπ∼Π,τ∼π,s0∼κ[ ∑c−1 t=0 log π(at|st)] (i.e. a constant and property of D) and SJ is a positive constant incurred due to sampling error in J(θ, φ, ω) and depends on concentration properties of πθ(a|s, z) and qφ(z|τ). Corollary 4.1.1. If the optimal policy π∗ of M is ζ-common in Π, and ∥∥∥d π∗ c
κ ∥∥∥ ∞ ≤ ξ, then, with
high probability 1− δ,
SubOpt(θ) ≤ 2ξ (1− γc)(1− γ)Rmax
 ζ + √√√√1 2 ( c + √ SJ δ +Hc )  . (8)
As we can see, SubOpt(θ) will reduce as D gets closer to π∗ (i.e. ζ approaches 0) and better primitives are learned (i.e. c decreases). While it might be tempting to increase c (i.e. the length of sub-trajectories) to reduce the suboptimality, a larger c will inevitably make it practically harder to control the autoencoding loss c, thereby leading to an increase in overall suboptimality and inducing a trade-off in determining the best value of c. In our experiments we treat c as a hyperparameter and set it to c = 10, although more sophisticated ways to determine c can be an interesting avenue for future work.
Till now, we have argued that there exists some near-optimal task policy πψ∗ if θ is sufficiently learned and π∗ is sufficiently well-represented in D. Now, we will show how primitive learning can improve downstream learning, by considering the benefits of using OPAL with offline RL. Building on the policy performance analysis from Kumar et al. (2020b), we now present theoretical results bounding the performance of the policy obtained when offline RL is performed with OPAL.
Theorem 4.2. Let πψ∗(z|s) be the policy obtained by CQL and let πψ∗,θ(a|s) refer to the policy when πψ∗(z|s) is used together with πθ(a|s, z). Let πβ ≡ {π ; π ∼ Π} refer to the policy generating Dr in MDPM and z ∼ πHβ (z|s) ≡ τ ∼ πβ,s0=s, z ∼ qφ(z|τ). Then, J(πψ∗,θ,M) ≥ J(πβ ,M)−κ with high probability 1− δ where
κ = O (
1
(1− γc)(1− γ)Es∼d πψ∗,θ M̂H (s)
[√ |Z|(DCQL(πψ∗ , πHβ )(s) + 1) ]) (9)
− α 1− γcEs∼d πψ∗ MH (s)
[ DCQL(πψ∗,θ, π H β )(s) ] , (10)
where DCQL is a measure of the divergence between two policies; see the appendix for a formal statement.
The precise bound along with a proof is described in Appendix B.1. Intuitively, this bound suggests that the worst-case deterioration over the learned policy depends on the divergence between the learned latent-space policy DCQL and the actual primitive distribution, which is controlled via any conservative offline RL algorithm (Kumar et al. (2020b) in our experiments) and the size of the latent space |Z|. Crucially, note that comparing Equation 9 to the performance bound for CQL (Equation 6 in Kumar et al. (2020b)) reveals several benefits pertaining to (1) temporal abstraction – a reduction in the factor of horizon by virtue of γc, and (2) reduction in the amount of worst-case error propagation due to a reduced action space |Z| vs. |A|. Thus, as evident from the above bound, the total error induced due to a combination of distributional shift and sampling is significantly reduced when OPAL is used as compared to the standard RL counterpart of this bound which is affected by the size of the entire action space for each and every timestep in the horizon. This formalizes our intuition that OPAL helps to partly mitigate distributional shift and sampling error. One downside of using a latent space policy is that we incur unsupervised learning error while learning primitives. However, empirically, this unsupervised learning error gets dominated by other error terms pertaining to offline RL. That is, it is much easier to control unsupervised loss than errors arising in offline RL.

heading: 5 EVALUATION
text: In this section, we will empirically show that OPAL improves learning of downstream tasks with offline RL, and then briefly show the same with few-shot imitation learning, online RL, and online multi-task transfer learning. Unless otherwise stated, we use c = 10 and dim(Z) = 8. See Appendix C for further implementation and experimental details. Visualizations and code are available at https://sites.google.com/view/opal-iclr

heading: 5.1 OFFLINE RL WITH OPAL
text: Description: We use environments and datasets provided in D4RL (Fu et al., 2020). Since the aim of our method is specifically to perform offline RL in settings where the offline data comprises varied and undirected multi-task behavior, we focus on Antmaze medium (diverse dataset), Antmaze large (diverse dataset), and Franka kitchen (mixed and partial datasets). The Antmaze datasets involve a simulated ant robot performing undirected navigation in a maze. The task is to use this undirected dataset to solve a specific point-to-point navigation problem, traversing the maze from one corner to the opposite corner, with only sparse 0-1 completion reward for reaching the goal. The kitchen datasets involves a franka robot manipulating multiple objects (microwave, kettle, etc.) either in an
undirected manner (mixed dataset) or in a partially task directed manner (partial dataset). The task is to use the datasets to arrange objects in a desired configuration, with only sparse 0-1 completion reward for every object that attains the target configuration.
Baseline: We use Behavior cloning (BC), BEAR (Kumar et al., 2019), EMAQ (Ghasemipour et al., 2020), and CQL (Kumar et al., 2020b) as baselines. We compare it to CQL+OPAL, which first uses OPAL to distill primitives from the offline dataset before applying CQL to learn a primitive-directing high-level policy.
Results: As shown in Table 1, CQL+OPAL outperforms nearly all the baselines on antmaze (see Figure 1 and Figure 3 for visualization) and kitchen tasks, with the exception of EMAQ having similar performance on kitchen mixed. To ensure fair comparison with EMAQ, we use an autoregressive primitive policy. With the exception of EMAQ on kitchen mixed, we are not aware of any existing offline RL algorithms that achieves similarly good performance on these tasks; moreover, we are not aware of any existing online RL algorithms which solve these tasks (see Table 3 for some comparisons), highlighting the benefit of using offline datasets to circumvent exploration challenges. There are two potential reasons for OPAL’s success. First, temporally-extended primitives could make the reward propagation learning problem easier. Second, the primitives may provide a better latent action space than the atomic actions of the environment. To understand the relative importance of these factors, we experimented with an ablation of CQL+OPAL that uses c = 1 to remove temporal abstraction. In this case, we find the method’s performance to be similar to standard CQL. This implies that the temporal abstraction provided by OPAL is one of the main contributing factors to its good performance. This observation also agrees with our theoretical analysis. See Appendix E for detailed discussion.

heading: 5.2 FEW-SHOT IMITATION LEARNING WITH OPAL
text: Description: Previously, we assumed that we have access to a task reward function, but only undirected data that performs other tasks. Now, we will study the opposite case, where we are not provided with a reward function for the new task either, but instead receive a small number of taskspecific demonstrations that illustrate optimal behavior. Simply imitating these few demonstrations is insufficient to obtain a good policy, and our experiments evaluate whether OPAL can effectively incorporate the prior data to enable few-shot adaptation in this setting. We use the Antmaze environments (diverse datasets) to evaluate our method and use an expert policy for these environments to sample n = 10 successful trajectories.
Baseline and Results: For baselines, we use Behavior cloning (BC) and the model from Wang et al. (2017), which prescribes using a sequential VAE (SVAE) over state trajectories in conjunction with imitation learning. As shown in Table 2, BC+OPAL clearly outperforms other baselines, show-
ing the importance of temporal abstraction and ascertaining the quality of learned primitives. See Appendix A for detailed discussion.

heading: 5.3 ONLINE RL AND MULTI-TASK TRANSFER WITH OPAL
text: Description: For online RL and multi-task transfer learning, we learn a task policy in space of primitives πθ(a|s, z) while keeping it fixed. For multi-task transfer, the task policy also takes in the task id and we use c = 5 and Z = 8. Since the primitives need to transfer to a different state distribution for multi-task transfer, it only learns the action sub-trajectory distribution and doesn’t take in the state feedback. See Appendix A for a detailed description of models. For online RL, we use the Antmaze environments (diverse datasets) with sparse and dense rewards for evaluating our method. For online multi-task transfer learning, we learn primitives with expert data from pick-andplace task and then use it to learn multi-task policy for MT10 and MT50 (from metaworld (Yu et al., 2020)), containing 10 and 50 robotic manipulation tasks which needs to be solved simultaneously.
Baseline and Results: For online RL, we use HIRO (Nachum et al., 2018b), a state-of-the-art hierarchical RL method, SAC (Haarnoja et al., 2018) with Behavior cloning (BC) pre-training on D, and Discovery of Continuous Options (DDCO) (Krishnan et al., 2017) which uses D to learn a discrete set of primitives and then learns a task policy in space of those primitives with online RL (Double DQN (DDQN) (Van Hasselt et al., 2015)). For online multi-task transfer learning, we use PPO (Schulman et al., 2017) and SAC (Haarnoja et al., 2018) as baselines. As shown in Table 3 and Table 4, OPAL uses temporal abstraction to improve exploration and thus accelerate online RL and multi-task transfer learning. See Appendix A for detailed discussion.

heading: 6 DISCUSSION
text: We proposed Offline Primitives for Accelerating offline RL (OPAL) as a preproccesing step for extracting recurring primitive behaviors from undirected and unlabelled dataset of diverse behaviors. We derived theoretical statements which describe under what conditions OPAL can improve learning of downstream offline RL tasks and showed how these improvements manifest in practice, leading to significant improvements in complex manipulation tasks. We further showed empirical demonstrations of OPAL’s application to few-shot imitation learning, online RL, and online multi-task transfer learning. In this work, we focused on simple auto-encoding models for representing OPAL, and an interesting avenue for future work is scaling up this basic paradigm to image-based tasks.

heading: 7 ACKNOWLEDGEMENTS
text: We would like to thank Ben Eysenbach and Kamyar Ghasemipour for valuable discussions at different points over the course of this work. This work was supported by Google, DARPA Machine Common Sense grant and MIT-IBM grant.

heading: Appendices

heading: A OTHER APPLICATIONS OF OPAL

heading: A.1 FEW-SHOT IMITATION LEARNING WITH OPAL
text: Additional Assumption: In addition to D, we assume access to a small number of expert demonstrations Dexp = {τi := (st, at)T−1t=0 }ni=1 where n N . How to use with OPAL? In imitation learning, the aim is to recover an expert policy given a small number of stochastically sampled expert demonstrations Dexp = {τi := (st, at)T−1t=0 }ni=1. As in the offline RL setting, we use the primitives πθ(a|s, z) learned by OPAL as a low-level controller and learn a high-level policy πψ(z|s). We first partition the expert demonstrations into sub-trajectories Dexppar = {τi,k := (sk+t, ak+t)c−1t=0 for k = 0, . . . , T − c}ni=1 of length c. We then use the learned encoder qφ(z|τ) to label these sub-trajectories with latent actions zi,k ∼ qφ(z|τi,k) and thus create a dataset Dexphi = {(sik+t, zi,k) for k = 0, . . . , T − c}ni=1. We use D exp hi to learn the high-level policy πψ(z|s) using behavioral cloning. As in the offline RL setting, we also finetune πθ(a|s, z) with latent-conditioned behavioral cloning to ensure consistency of the labelled latents.
Evaluation Description: We receive a small number of task-specific demonstrations that illustrate optimal behavior. Simply imitating these few demonstrations is insufficient to obtain a good policy, and our experiments evaluate whether OPAL can effectively incorporate the prior data to enable fewshot adaptation in this setting. We use the Antmaze environments (diverse datasets) to evaluate our method and use an expert policy for these environments to sample n = 10 successful trajectories.
Baseline: We evaluate two baselines. First, we test a simple behavioral cloning (BC) baseline, which trains using a max-likelihood loss on the expert data. In order to make the comparison fair to OPAL (which uses the offline dataset D in addition to the expert dataset), we pretrain the BC agent on the undirected dataset using the same max-likelihood loss. As a second baseline and to test the quality of OPAL-extracted primitives, we experiment with an alternative unsupervised objective from Wang et al. (2017), which prescribes using a sequential VAE (SVAE) over state trajectories in conjunction with imitation learning.
Results: As shown in Table 2, BC+OPAL clearly outperforms other baselines, showing the importance of temporal abstraction and ascertaining the quality of learned primitives. SVAE’s slightly worse performance suggests that decoding the state trajectory directly is more difficult than simply predicting the actions, as OPAL does, and that this degrades downstream task learning.

heading: A.2 ONLINE RL WITH OPAL
text: Additional Assumptions: We assume online access toM; i.e., access to Monte Carlo samples of episodes fromM given an arbitrary policy π. How to use with OPAL? To apply OPAL to standard online RL, we fix the learned primitives πθ(a|s, z) and learn a high-level policy πψ(z|s) in an online fashion using the latents z as temporally-extended actions. Specifically, when interacting the environment, πψ(z|s) chooses an appropriate primitive every c steps, and this primitive acts on the environment directly for c timesteps. Any off-the-shelf online RL algorithm can be used to learn ψ. In our experiments, we use SAC (Haarnoja et al., 2018). To ensure that πψ(z|s) stays close to the data distribution and avoid generalization issues associated with the fixed πθ(a|s, z), we add an additional KL penalty to the reward of the form DKL(πψ(z|s)||ρω(z|s0)). Evaluation Description: We use Antmaze medium (diverse dataset) and Antmaze large (diverse dataset) from the D4RL task suite (Fu et al., 2020) to evaluate our method. We evaluate using both a dense distance-based reward−‖g−antxy‖ and a sparse success-based reward 1[‖g−antxy‖ ≤ 0.5] (the typical default for this task), where antxy is the 2d position of the ant in the maze.
Baseline: To solve these tasks through online RL, we need both (i) hierarchy (i.e. learning a policy on top of primitives) which improves exploration (Nachum et al., 2019b) and (ii) unlabelled (i.e. no task reward) offline dataset which allows us to bootstrap the primitives. This informed our choice
of these three baselines. First, to test the role of D in exploration, we use HIRO (Nachum et al., 2018b), a state-of-the-art hierarchical RL method, as a baseline. Second, to test the role of temporal abstraction, we pre-train a flat policy on D using behavioral cloning (BC) and then finetune the policy on downstream tasks with SAC. Third, to test the quality of extracted primitives for online RL, we extract a discrete set of primitives with Deep Discovery of Continuous Options (DDCO) (Krishnan et al., 2017) and use Double DQN (DDQN) (Van Hasselt et al., 2015) to learn a task policy in the space of learned discrete primitives.
Results: As shown in Table 3, SAC+OPAL outperforms all the baselines, showing (1) the importance of D in exploration, (2) the role of temporal abstraction, and (3) the good quality of learned primitives. Except for HIRO on Antmaze large with dense rewards, all other baselines fail to make any progress at all. In contrast, SAC+OPAL only fails to make progress on Antmaze large with sparse rewards.

heading: A.3 ONLINE MULTI-TASK TRANSFER LEARNING WITH OPAL
text: Additional Assumption: We assume the existence of M additional MDPs {Mi = (Si,A,Pi, ri, γ)}Mi=1 where the action space A, and the discount factor γ are same as those of M. How to use it with OPAL? In the multi-task setting, we aim to learn near-optimal behavior policies on M MDPs {Mi = (Si,A,Pi, ri, γ)}Mi=1. As in the previous applications of OPAL, we learn a set of high-level policies πψ(z|s, i) which will direct pretrained primitives πθ(a|s, z) to maximize cumulative rewards. Since the state space in the M MDPs is potentially distinct from that in the offline dataset D, we cannot transfer the state distribution and can only hope to transfer the action sub-trajectory distribution. Therefore, during the unsupervised training phase for learning πθ, we make the encoder and the decoder blind to the states in sub-trajectory. Specifically, the encoder becomes (µencz , σ enc z ) = qφ(zt|st, {at+i}c−1i=0 ) and is represented by a bidirectional GRU. The decoder becomes πθ({at, . . . , at+c−1}|zt) which decodes the entire action sub-trajectory from the latent vector and is represented by a GRU. With these state-agnostic primitives in hand, we then learn a policy πψ(z|s, i) using any off-the-shelf online RL method. In our experiments, we use Proximal Policy Optimization (PPO) (Schulman et al., 2017).
Evaluation Description: We use the Metaworld task suite (Yu et al., 2020) to evaluate our method. The dataset D for learning primitives consists of trajectories generated by an expert policy for a goal-conditioned pick-and-place task. The pick-and-place task is suitable for unsupervised primitive learning because it contains all the basic operations (eg: move, grasp, place) required for performing more complex manipulation tasks in the Metaworld. Once we have learned the primitives, we learn a policy πψ(z|s, i) for the MT10 and MT50 benchmarks, where MT10 and MT50 contain 10 and 50 robotic manipulation tasks, respectively, which we need to be solved simultaneously. In these experiments we use c = 5 and dim(Z) = 8. Baseline: We use SAC (Haarnoja et al., 2018) and PPO (Schulman et al., 2017) as baselines.
Results: As shown in Table 4, PPO+OPAL clearly outperforms both PPO and SAC, showing the importance of temporal abstraction in online multi-task transfer.

heading: B PROOF OF THEOREMS

heading: B.1 BOUNDING THE SUBOPTIMALITY OF THE LEARNED PRIMITIVES
text: We will begin by proving the following lemma which bounds the sampling error incurred by J(φ, θ, ω).
Lemma B.0.1. With high probability 1− δ, ∣∣∣∣∣J(θ, φ, ω)− Eπ∼Π,τ∼π,z∼qφ(z|τ) [ − c−1∑
t=0
log πθ(at|st, z) ]∣∣∣∣∣ ≤ √ SJ δ
(11)
where SJ is a constant dependent on concentration properties of πθ(a|s, z) and qφ(z|τ).
Proof. To be concise, let us denote the sampling error in J(φ, θ, ω) by
∆J = ∣∣∣∣∣J(θ, φ, ω)− Eπ∼Π,τ∼π,z∼qφ(z|τ) [ − c−1∑
t=0
log πθ(at|st, z) ]∣∣∣∣∣ (12)
Applying Chebyshev’s inequality to ∆J , we get that, with high probability 1− δ,
∆J ≤
√ Varπ∼Π,τ∼π,z∼qφ(z|τ)(− ∑c−1 t=0 log πθ(at|st, z))
δ = √ SJ δ
(13)
Therefore, combining all the equations, we have
Eπ∼Π,τ∼π,z∼qφ(z|τ)
[ − c−1∑
t=0
log πθ(at|st, z) ] ≤ J(θ, φ, ω) + √ SJ δ
(14)
We present a general performance difference lemma that will help in our proof of Lemma 4.0.1. Lemma B.0.2. If π1 and π2 are two policies inM, then
|JRL(π1,M)− JRL(π2,M)| ≤ 2
(1− γ)2 RmaxEs∼dπ1 [DTV(π1(a|s)||π2(a|s))]. (15)
Proof. Following the derivations in Achiam et al. (2017) and Nachum et al. (2018a), we express the performance of a policy π inM in terms of linear opterators:
JRL(π,M) = (1− γ)−1R>(I − γΠπP)−1Ππµ, (16) where R is a vector representation of the rewards of M, Ππ is a linear operator mapping state distributions to state-action distributions according to π, and µ is a vector representation of the initial state distribution ofM. Accordingly, we express the performance difference of π1, π2 as,
|JRL(π1,M)− JRL(π2,M)| = |R>((I − γΠ1P)−1Π1µ− (I − γΠ2P)−1Π2µ)| (17) ≤ Rmax|(I − γΠ1P)−1Π1µ− (I − γΠ2P)−1Π2µ|. (18)
By the triangle inequality, we may bound equation 18 by
Rmax ( |(I − γΠ1P)−1Π1µ− (I − γΠ2P)−1Π1µ| +
|(I − γΠ2P)−1Π1µ− (I − γΠ2P)−1Π2µ| ) . (19)
We begin by approaching the first term inside the parentheses of equation 19. That first term may be expressed as
|(I − γΠ2P)−1(I − γΠ2P − (I − γΠ1P))(I − γΠ1P)−1Π1µ| (20)
= |γ(I − γΠ2P)−1(Π1 −Π2)P(I − γΠ1P)−1Π1µ| (21) ≤ γ
1− γ |(Π1 −Π2)P(I − γΠ1P) −1Π1µ| (22)
= 2γ
(1− γ)2Es∼(1−γ)P(I−γΠ1P)−1Π1µ[DTV(π1(a|s)‖π2(a|s))]. (23)
Now we continue to the second term inside the parentheses of equation 19. This second term may be expressed as
|(I − γΠ2P)−1(Π1 −Π2)µ| ≤ 1
1− γ |(Π1 −Π2)µ| (24)
= 2
1− γEs∼µ[DTV(π1(a|s)‖π2(a|s))]. (25)
To combine equations 23 and 25, we note that
dπ1 = γ · (1− γ)P(I − γΠ1P)−1Π1µ+ (1− γ) · µ. (26)
Thus, we have
|JRL(π1,M)− JRL(π2,M)| ≤ 2
(1− γ)2 RmaxEs∼dπ1 [DTV(π1(a|s)‖π2(a|s))], (27)
as desired.
Lemma 4.0.1. If π1 and π2 are two policies inM, then
|JRL(π1,M)− JRL(π2,M)| ≤ 2
(1− γc)(1− γ)RmaxEs∼d π1 c [DTV(π1(τ |s)||π2(τ |s))], (28)
where DTV(π1(τ |s)||π2(τ |s)) denotes the TV divergence over c-length sub-trajectories τ sampled from π1 vs. π2 (see section 3). Furthermore,
SubOpt(θ) ≤ 2 (1− γc)(1− γ)RmaxEs∼dπ ∗ c [DTV(π ∗(τ |s)||πθ,ψ∗(τ |s))]. (29)
Proof. We focus on proving equation 28, as the subsequent derivation of equation 29 is straightforward by definition of SubOpt.
To derive equation 28, we may simply consider π1 and π2 acting in an “every-c-steps” version of M, where the action space is now τ and reward are accumulated over c steps using γ-discounting. Note that in this abstracted version ofM, the max reward is 1−γc1−γ Rmax and the MDP discount is γc. Plugging this into the result of Lemma B.0.2 immediately yields the desired claim.
Theorem 4.1. Let θ, φ, ω be the outputs of solving equation 1, such that J(θ, φ, ω) = c. Then, with high probability 1 − δ, for any π that is ζ-common in Π, there exists a distribution H over z such that for πHθ (τ |s) := Ez∼H [πθ(τ |z, s)],
Es∼κ[DTV(π(τ |s)||πHθ (τ |s))] ≤ ζ + √√√√1 2 ( c + √ SJ δ +Hc ) , (30)
whereHc = Eπ∼Π,s∼κ,τ∼π[ ∑c−1 t=0 log π(at|st)] (i.e. a constant and property ofD) and SJ is a positive constant incurred due to sampling error in J(θ, φ, ω) and depends on concentration properties of πθ(a|s, z) and qφ(z|τ).
Proof. We start with application of the triangle inequality:
DTV(π(τ |s)||πHθ (τ |s)) ≤ DTV(π(τ |s)||π(τ |s)) + DTV(π(τ |s)||πHθ (τ |s)). (31)
Taking expectation with respect to π ∼ Π, s ∼ κ on both the sides, we get
Es∼κ[DTV(π(τ)||πHθ (τ))] ≤ Eπ∼Π,s∼κ[DTV(π(τ)||π(τ))] + Eπ∼Π,s∼µ[DTV(π(τ)||πHθ (τ))] (32) ≤ Eπ∼Π,s∼κ[DTV(π(τ)||π(τ))] (33)
+ Eπ∼Π,s∼µ[ √ 1
2 DKL(π(τ)||πHθ (τ))] (34)
≤ ζ + √ 1
2 Eπ∼Π,τ∼π,s∼κ[log π(τ)− logEz∼H [πθ(τ |z)]] (35)
≤ ζ + √ 1
2 Eπ∼Π,s∼κ,τ∼π,z∼H [log π(τ)− log πθ(τ |z)]. (36)
The last two inequality used Jensen’s inequality. Let H(z) = Eπ∼Π,τ∼π[qφ(z|τ)]. Cancelling out the dynamics and using equation 14 we get,
Es∼κ[DTV(π(τ)||πHθ (τ))]
≤ ζ + √√√√1 2 Eπ∼Π,s∼κ,τ∼π,z∼qφ(z|τ) [ c−1∑
t=0
(log π(at|st)− log πθ(at|st, z)) ]
≤ ζ + √√√√1 2 ( Hc + √ SJ δ + J(θ, φ, ω) )
= ζ + √√√√1 2 ( Hc + c + √ SJ δ )
Corollary 4.1.1. If the optimal policy π∗ of M is ζ-common in Π, and ∥∥∥d π∗ c
κ ∥∥∥ ∞ ≤ ξ, then, with
high probability 1− δ,
SubOpt(θ) ≤ 2ξ (1− γc)(1− γ)Rmax
 ζ + √√√√1 2 ( c + √ SJ δ +Hc )  (37)
Proof. Expanding lemma 4.0.1 using the above assumption, we have
SubOpt(θ) ≤ |JRL(π∗,M)− JRL(πHθ ,M)| (38)
≤ 2 (1− γc)(1− γ)RmaxEs∼dπ ∗ c [DTV(π ∗(τ |s)||πHθ (τ |s))] (39)
≤ 2 (1− γc)(1− γ)Rmax ∥∥∥∥ dπ ∗ c κ ∥∥∥∥ ∞ Es∼κ[DTV(π∗(τ |s)||πHθ (τ |s))] (40)
≤ 2ξ (1− γc)(1− γ)RmaxEs∼κ[DTV(π ∗(τ |s)||πHθ (τ |s))] (41)
Now, we can use theorem 4.1 to prove the corollary.

heading: B.2 PERFORMANCE BOUNDS FOR OPAL
text: Theorem 4.2. Let πψ∗(z|s) be the policy obtained by CQL and let πψ∗,θ(a|s) refer to the policy when πψ∗(z|s) is used together with πθ(a|s, z). Let πβ ≡ {π ; π ∼ Π} refer to the policy generating Dr in MDPM and z ∼ πHβ (z|s) ≡ τ ∼ πβ,s0=s, z ∼ qφ(z|τ). Then, J(πψ∗,θ,M) ≥ J(πβ ,M)−κ with high probability 1− δ where
κ = O (
1
(1− γc)(1− γ)Es∼d πψ∗,θ M̂H (s)
[√ |Z|(DCQL(πψ∗ , πHβ )(s) + 1) ]) (42)
− α 1− γcEs∼d πψ∗ MH (s)
[ DCQL(πψ∗,θ, π H β )(s) ] (43)
Proof. We assume that the variational posterior qφ(z|τ) obtained after learning OPAL from D is same (or nearly same) as the true posterior p(z|τ). qφ can be used to define πHβ (z|s) as z ∼ πHβ ≡ τ ∼ πβ , z ∼ qφ(z|τ). This induces an MDPMH = (S,Z,Pz, rz, γc) where Z is the inferred latent space for choosing primitives, Pz and rz are the latent dynamics and reward function such that st+c ∼ Pz(st+c|st, zt) ≡ st+i+1 ∼ P(st+i+1|st+i, at+i), at+i ∼ π(at+i|st+i, zt) ∀i ∈ {0, 1, . . . , c − 1} and rz(st, zt) = ∑c−1 i=0 γ
ir(st+i, at+i), and γc is the new discount factor effectively reducing the task horizon by a factor of c. π(a|s, z) is the primitive induced by qφ and πβ . Since qφ captures the true posterior, π(a|s, z) is the optimal primitive you can learn and its
autoencoding loss, under true expectation, is ∗c = Eπ∼Π,τ∼π,z∼qφ(z|τ) [ −∑c−1t=0 log π(at|st, z) ] . Therefore, τ ∼ πβ ≡ z ∼ πHβ , τ ∼ π(·|·, z). πβ is used to collect the data Dr which induces an empirical MDP. We refer to the empirical MDP induced by Dr as M̂ = (S,A, P̂, µ̂, r, γ) where P̂(ŝ′|ŝ, â) = ∑ (s,a,s′)∼D 1[s=ŝ,a=â,s
′=ŝ′]∑ (s,a)∼D 1[s=ŝ,a=â]
and µ̂(ŝ) = ∑ s0∼D 1[s0=ŝ]
N . We use qφ to get Drhi from Dr which induces another empirical MDP M̂H . Using these definitions, we will try to bound |J(πψ∗,θ,M)− J(πβ ,M)|. Let’s break |J(πψ∗,θ,M)− J(πβ ,M)| into
|J(πψ∗,θ,M)− J(πβ ,M)| ≤ |J(πψ∗,θ,M)− J(πψ∗ ,MH)| (44) + |J(πψ∗ ,MH − J(πHβ ,MH)| (45) + |J(πHβ ,MH)− J(πβ ,M)| (46)
Since qφ captures the true variational posterior, τ ∼ πβ ≡ z ∼ πHβ , τ ∼ π(·|·, z) and therefore, |J(πHβ ,MH)− J(πβ ,M)| = 0. For bounding, |J(πψ∗ ,MH − J(πHβ ,MH)|, we use theorem 3.6 from Kumar et al. (2020b) and apply it toMH to get
|J(πψ∗ ,MH)− J(πHβ ,MH)| (47)
≤ 2 (
Cr,δ 1− γc + γcRmaxCP,δ (1− γc)(1− γ) ) E s∼dπψ∗,θ
M̂H (s)
[√ |Z| |D(s)|DCQL(πψ∗ , π H β )(s) + 1 ] (48)
− α 1− γcEs∼d πψ∗ MH (s)
[ DCQL(πψ∗,θ, π H β )(s) ] = κ2 (49)
Now, we will try to bound |J(πψ∗,θ,M) − J(πψ∗ ,MH)|. The only difference between the two is that the primitive πθ(a|s, z) is used in M and the primitive π(a|s, z) is used in MH . Therefore, we can write the above bound as |J(πψ∗,θ,M) − J(πψ∗,π(·|·,z),M)|. Let’s first bound their value function at a particular state s. Using Lemma 4.0.1, we get
|J(πψ∗,θ,M)− J(πψ∗,π(·|·,z),M)| (50)
≤ 2Rmax (1− γc)(1− γ)Es∼d πψ∗,π(·|·,z) c [DTV(πψ∗,π(·|·,z)(τ |s)||πψ∗,θ(τ |s))] (51)
≤ 2Rmax (1− γc)(1− γ)Es∼d πψ∗,π(·|·,z) c
[√ 1
2 DKL(πψ∗,π(·|·,z)(τ |s)||πψ∗,θ(τ |s))
] (52)
Now, we will try to bound DKL(πψ∗,π(·|·,z)(τ |s)||πψ∗,θ(τ |s)). We have
Ez∼πψ∗ (z|s),τ∼π(τ |s,z)
[ log πψ∗(z|s) ∏c−1 t=1 P(st|st−1, at−1) ∏c−1 t=0 π(at|st, z)
πψ∗(z|s) ∏c−1 t=1 P(st|st−1, at−1) ∏c−1 t=0 πθ(at|st, z)
] (53)
= Ez∼πψ∗ (z|s),τ∼π(τ |s,z)
[ c−1∑
t=0
log π(at|st, z)− log πθ(at|st, z) ]
(54)
= Ez∼πβ(z|s),τ∼π(τ |s,z) [( πψ∗(z|s) πHβ (z|s) ) c−1∑
t=0
log π(at|st, z)− log πθ(at|st, z) ]
(55)
≤ ∣∣∣∣∣ πψ∗(z|s) πHβ (z|s) ∣∣∣∣∣ ∞ Ez∼πβ(z|s),τ∼π(τ |s,z) [ c−1∑ t=0 log π(at|st, z)− log πθ(at|st, z) ]
(56)
≤ ∣∣∣∣∣ πψ∗(z|s) πHβ (z|s) ∣∣∣∣∣ ∞ ( c − ∗c + √ SJ δ ) (57)
The last equation comes from above definition of ∗c and equation 14. We will now try to bound∣∣∣πψ∗ (z|s)πHβ (z|s) ∣∣∣ ∞ using DCQL(πψ∗ , πHβ )(s). Using definition of DCQL(πψ∗ , π H β )(s), we have
DCQL(πψ∗ , π H β )(s) =
∑
z
πψ∗(z|s) ( πψ∗(z|s) πHβ (z|s) − 1 )
(58)
⇒ DCQL(πψ∗ , πHβ )(s) + 1 = ∑
z
πHβ (z|s) ( πψ∗(z|s) πHβ (z|s) )2 ≤ ∣∣∣∣∣ πψ∗(z|s) πHβ (z|s) ∣∣∣∣∣ 2 ∞ πHβ (z̄|s) (59)
where z̄ = arg maxz ( πψ∗ (z|s) πHβ (z|s) ) . To be concise, let
∆c = ( c − ∗c + √ SJ δ ) (60)
Combining above equations, we have
DKL(πψ∗(z|s)π(τ |s, z)||πψ∗(z|s)πθ(τ |s, z)) ≤ ∆c
√ DCQL(πψ∗ , πHβ )(s) + 1
πHβ (z̄|s) (61)
Using this to bound the returns, we get
|J(πψ∗,θ,M)− J(πψ∗,π(·|·,z),M)| (62)
≤ 2Rmax (1− γc)(1− γ)Es∼d πψ∗,π(·|·,z) c
  ( DCQL(πψ∗ , π H β )(s) + 1
πHβ (z̄|s)
) 1 4 √
1 2 ∆c
  (63)
= κ1 (64)
We get κ = κ1 + κ2. We apply O to get the notation in the theorem.

heading: C EXPERIMENT DETAILS

heading: C.1 OPAL EXPERIMENT DETAILS
text: Encoder The encoder qφ(z|τ) takes in state-action trajectory τ of length c. It first passes the individual states through a fully connected network with 2 hidden layers of sizeH and ReLU activation. Then it concatenates the proccessed states with actions and passes it through a bidirectional GRU with hidden dimension of H and 4 GRU layers. It projects the output of GRU to mean and log standard deviation of the latent vector through linear layers.
Prior The prior ρω(z|s) takes in the current state s and passes it through a fully connected network with 2 hidden layers of size H and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of the latent vector through linear layers.
Primitive Policy The primitive (i.e. decoder) πθ(a|s, z) has same architecture as the Prior but it takes in state and latent vector and produces mean and log standard deviation of the action. For kitchen environments, we use an autoregressive primitive policy with same architecture as used by EMAQ (Ghasemipour et al., 2020).
We use H = 200 for antmaze environments and H = 256 for kitchen environments. In both cases, OPAL was trained for 100 epochs with a fixed learning rate of 1e− 3, β = 0.1 (Lynch et al., 2020), Adam optimizer (Kingma & Ba, 2014) and a batch size of 50.

heading: C.2 TASK POLICY ARCHITECTURE
text: In all environments, for task policy, we used a fully connected network with 3 hidden layers of size 256 and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of the latent vector through linear layers.

heading: C.3 SAC HYPERPARAMETERS
text: We used SAC (Haarnoja et al., 2018) for online RL experiments in learning a task policy either in action space A or latent space Z . For the discrete primitives extracted from DDCO (Krishnan et al., 2017), we used Double DQN Van Hasselt et al. (2015). We used the standard hyperparameters for SAC and Double DQN as provided in rlkit code base (https://github.com/vitchyr/ rlkit) with both policy learning rate and q value learning rate as 3e− 4.

heading: C.4 CQL HYPERPARAMETERS
text: We used CQL (Kumar et al., 2020b) for offline RL experiments in learning a task policy either in action space A or latent space Z . We used the standard hyperparameters, as mentioned in Kumar et al. (2020b)., with minor differences. We used policy learning rate of 3e− 5, q value learning rate of 3e − 4, and primitive learning rate of 3e − 4. For antmaze tasks, we used CQL(H) variant with τ = 5 and learned α. For kitchen tasks we used CQL(ρ) variant with fixed α = 10. In both cases, we ensured α never dropped below 0.001.

heading: D CONNECTION BETWEEN OPAL AND VAE OBJECTIVES
text: We are given an undirected, unlabelled and diverse dataset D of sub-trajectories of length c. We would like to fit a sequential VAE model to D which maximizes
max θ
Eτ∼D[log pθ(τ |s0)] (65)
where s0 is initial state of the sub-trajectory. Let’s consider
log pθ(τ |s0) = log ∫ pθ(τ, z|s0)dz = log ∫ pθ(τ, z|s0)qφ(z|τ)
qφ(z|τ) dz (66)
(using Jensen’s inequality)
≥ ∫ qφ(z|τ)[log pθ(τ, z|s0)− log qφ(z|τ)]dz = Ez∼qφ(z|τ) [ log pθ(τ |z, s0)− log
qφ(z|τ) pθ(z|s0)
]
(67) Using the above equation, we have the following lower-bound for our objective function
max θ Eτ∼D[log pθ(τ |s0)] ≥ max θ,φ
Eτ∼D̄,z∼qφ(z|τ) [ log pθ(τ |z, s0)− log
qφ(z|τ) pθ(z|s0)
] (68)
= max θ,φ
Eτ∼D,z∼qφ(z|τ)[log pθ(τ |z, s0)]−DKL(qφ(z|τ)||pθ(z|s0)) (69)
We separate the parameters of decoder from prior and hence write pθ(z|s0) = ρω(z|s0). We can expand log pθ(τ |z, s0) = ∑c−1 t=1 logP(st|st−1, at−1) + ∑c−1 t=0 log πθ(at|st, z). Since P is fixed it can be removed from the objective function. Therefore, we can write the objective function as
max θ,φ Eτ∼D,z∼qφ(z|τ)
[ c−1∑
t=0
log πθ(at|st, z) ] − βDKL(qφ(z|τ)||ρω(z|s0)) (70)
where β = 1. This is similar to the autoencoding loss function we described in section 4.

heading: E ABLATION STUDIES
text: As shown in Table 5, we experimented with different choices of dim(Z) on antmaze-medium (diverse). Using the hyperparameters from Nachum et al. (2018a), we fixed c = 10. While
dim(Z) = 8, 16 gave similar performances, dim(Z) = 4 performed slightly worse. Therefore, we selected dim(Z) = 8 for our final model as it was simpler. Temporal abstraction actually helps: To empirically verify that the gain in performance was due to temporal abstraction and not better action space learned through latent space, we tried c = 1 (dim(Z) = 8) and found the performance to be similar to that of CQL (i.e. 55.3 ± 3.8) thereby empirically supporting the theoretical benefits of temporal abstraction.
We found dim(Z) = 8 and c = 10 to work well with other environments as well. However, we acknowledge that the performance of CQL+OPAL can be further improved by carefully choosing better hyperparameters for each environment or by using other offline hyperparameter selection methods for offline RL, which is a subject of future work.

heading: F ALTERNATIVE METHODS FOR EXTRACTING PRIMITIVES FROM OFFLINE DATA
text: We describe alternative methods for extracting a primitive policy from offline data. These methods are offline variants of CARML (Jabri et al., 2019) and DADS (Sharma et al., 2019). We tried these techniques in an early phase of our project and used the environment antmaze-medium (diverse) to evaluate these methods.
Let’s consider an offline undirected, unlabelled and diverse dataset D = {(sit, ait)c−1t=0}Ni=1. Let τ = (st) c−1 t=0 represent the state trajectory. To extract primitives, we first cluster the trajectories by maximizing the mutual information between the state trajectory τ and latent variable z (indicating cluster index) with respect to the parameters of joint distribution pφ,ω(τ, z) = pω(z)pφ(τ |z). For now, we consider pω(z) = Cat(p1, . . . , pk) (i.e. discrete latent variables sampled from a Categorical distribution) and represent z as one-hot vector of dimension k. The choice of pω(z) is consistent with the choices made in Jabri et al. (2019) and Sharma et al. (2019). Since z is discrete, we can use Bayes rule to calculate pφ,ω(z|τ) as
pφ,ω(z|τ) = pω(z)pφ(τ |z)∑k i=1 pω(zi)pφ(τ |zi) . (71)
Our objective function becomes
max φ,ω I(τ ; z) = max φ,ω
Eτ∼D,z∼pφ,ω(z|τ) [ log pφ(τ |z) p(τ) ] . (72)
Offline CARML and offline DADS differ only in how they model pφ(τ |z):
• Offline CARML We model pφ(τ |z) = ∏c−1 t=0 pφ(st|z) and hence, log pφ(τ |z) =∑c−1 t=0 log pφ(st|z). • Offline DADS We model pφ(τ |z) = p(s0) ∏c−1 t=1 pφ(st|st−1, z) and hence, log pφ(τ |z) =
log p(s0) + ∑c−1 t=0 log pφ(st|st−1, z). Here, we only model pφ(st|st−1, z) and not p(s0). Since, log is additive in nature, p(s0) will be ignored while calculating gradient.
To optimize equation 72, we use Algorithm 2 from Jabri et al. (2019). Once we have clustered the state trajectories τ with labels z by maximizing I(τ ; z), we can use behavioral cloning (BC) to learn πθ(a|s, z). Finally, we use pφ,ω(z|τ) to label the reward-labelled data Dr = {(sit, ait, rit)c−1t=0}Ni=1 with latents, and transform it intoDrhi = {(si0, zi, ∑c−1 t=0 γ irit, sc) N i=1}. The task policy πψ is trained onDrhi using Conservative Q Learning (CQL) (Kumar et al., 2020b). Since the primitive policy πθ is trained after pφ,ω(z|τ) is fully trained, it doesn’t need any additional finetuning.

heading: F.1 RESULTS
text: Using the hyperparameters from Nachum et al. (2018a), we used c = 10. We experimented with different values of k = 5, 10, 20 and found that k = 10, 20 works the best (see Table 6 for more
details). We went with k = 10 as our final model since it’s simpler. Offline CARML effectively uses only 6 skills as the other 4 skills had pω(z) = 0. Offline DADS uses all the skills. The results are described in Table 7. In addition to calculating the average success rate, we also calculate the average cumulative dense rewards for entire trajectory and the average cumulative dense rewards for the last 5 time steps. Here, the dense reward is negative l2 distance to the goal. The resulting trajectory clusters (using a subset of the dataset) from discrete skills are also visualized in Figure 4 where different colors represent different clusters.
Since offline CARML treats the states in the trajectory conditionally independent of each other given z, the clustering mainly focuses on the spatial location. Therefore, offline CARML isn’t able to separate out different control modes starting around the same spatial locations which explains its poor performance when combined with CQL. As we can see from Figure 5, offline CARML is able to make progress towards the goal, but gets stuck along the way due to poor separation of control modes. On the other hand, offline DADS treats the state transitions in the trajectory conditionally independent of each other given z and thus clusters trajectories with similar state transitions together. This allows it to more effectively separate out the control modes. Therefore, CQL+offline DADS slightly improves upon CQL but is still limited by discrete number of skills. Furthermore, increasing the number of skills from 10 to 20 gives similar performance. Moreover, in these methods, it’s intractable to use continuous skill space since we use Bayes rule to calculate pφ,ω(z|τ). Therefore, we decided to switch to learning a β-VAE (Higgins et al., 2016) style generative model with continuous skill space i.e. OPAL.

heading: F.2 TRAINING DETAILS
text: For clustering by optimizing equation 72, both offline CARML and offline DADS only considers the global x-y pose of the ant and ignores other dimensions of the state space. These methods fail to work when considering the full state space.
Offline CARML pφ(s|z) takes in the latent one-hot vector z and passes it through a fully connected network with 2 hidden layers of size H = 200 and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of the reduced state s (only global x-y pose considered) through linear layers.
Offline DADS pφ(st|st−1, z) has the same architecture as pφ(s|z) but also takes in the reduced state from the previous timestep.
Primitive Policy The primitive policy πθ(a|s, z) takes in the current state s and latent one-hot vector z and passes it through a fully connected network with 2 hidden layers of size H = 200 and ReLU activation. It then projects the output of the hidden layers to mean and log standard deviation of action through linear layers.
We perform the clustering for 25 epochs with a fixed learning rate of 1e − 3, Adam optimizer (Kingma & Ba, 2014) and a batch size of 50 using the Algorithm 2 from Jabri et al. (2019).
Task Policy For task policy πψ(s), we used a fully connected network with 3 hidden layers of size 256 and ReLU activation. It then projects the output of the hidden layers to the logits (corresponding to the components of discrete latent space) through linear layers.
CQL Hyperparameters We used the standard hyperparameters for CQL(H) with discrete action space, as mentioned in Kumar et al. (2020b).

Review summaries:
In the RL setting, this paper tackles the case where an agent may have access to large amounts of offline experience data. The objective of the work is to find an effective way to leverage this data for finding temporally extended primitive behaviors. The paper provides results that show how performing offline primitive learning can be leveraged for improving few-shot imitation learning as well as exploration and transfer on a variety of benchmark domains.