OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

1 INTRODUCTION . Reinforcement Learning ( RL ) systems have achieved impressive performance in a variety of online settings such as games ( Silver et al. , 2016 ; Tesauro , 1995 ; Brown & Sandholm , 2019 ) and robotics ( Levine et al. , 2016 ; Dasari et al. , 2019 ; Peters et al. , 2010 ; Parmas et al. , 2019 ; Pinto & Gupta , 2016 ; Nachum et al. , 2019a ) , where the agent can act in the environment and sample as many transitions and rewards as needed . However , in many practical applications the agent ’ s ability to continuously act in the environment may be severely limited due to practical concerns ( DulacArnold et al. , 2019 ) . For example , a robot learning through trial and error in the real world requires costly human supervision , safety checks , and resets ( Atkeson et al. , 2015 ) , rendering many standard online RL algorithms inapplicable ( Matsushima et al. , 2020 ) . However , in such settings we might instead have access to large amounts of previously logged data , which could be logged from a baseline hand-engineered policy or even from other related tasks . For example , in self-driving applications , one may have access to large amounts of human driving behavior ; in robotic applications , one might have data of either humans or robots performing similar tasks . While these offline datasets are often undirected ( generic human driving data on various routes in various cities may not be directly relevant to navigation of a specific route within a specific city ) and unlabelled ( generic human driving data is often not labelled with the human ’ s intended route or destination ) , this data is still useful in that it can inform the algorithm about what is possible to do in the real world , without the need for active exploration . In this paper , we study how , in this offline setting , an effective strategy to leveraging unlabeled and undirected past data is to utilize unsupervised learning to extract potentially useful and temporally extended primitive skills to learn what types of behaviors are possible . For example , consider a dataset of an agent performing undirected navigation in a maze environment ( Figure 1 ) . While the dataset does not provide demonstrations of exclusively one specific point-to-point navigation task , ∗Work done during an internship at Google Brain it nevertheless presents clear indications of which temporally extended behaviors are useful and natural in this environment ( e.g. , moving forward , left , right , and backward ) , and our unsupervised learning objective aims to distill these behaviors into temporally extended primitives . Once these locomotive primitive behaviors are extracted , we can use them as a compact constrained temporallyextended action space for learning a task policy with offline RL , which only needs to focus on task relevant navigation , thereby making task learning easier . For example , once a specific point-to-point navigation is commanded , the agent can leverage the learned primitives for locomotion and only focus on the task of navigation , as opposed to learning locomotion and navigation from scratch . We refer to our proposed unsupervised learning method as Offline Primitives for Accelerating offline reinforcement Learning ( OPAL ) , and apply this basic paradigm to offline RL , where the agent is given a single offline dataset to use for both the initial unsupervised learning phase and then a subsequent task-directed offline policy optimization phase . Despite the fact that no additional data is used , we find that our proposed unsupervised learning technique can dramatically improve offline policy optimization compared to performing offline policy optimization on the raw dataset directly . To the best of our knowledge , ours is the first work to theoretically justify and experimentally verify the benefits of primitive learning in offline RL settings , showing that hierarchies can provide temporal abstraction that allows us to reduce the effect of compounding errors issue in offline RL . These theoretical and empirical results are notably in contrast to previous related work in online hierarchical RL ( Nachum et al. , 2019b ) , which found that improved exploration is the main benefit afforded by hierarchically learned primitives . We instead show significant benefits in the offline RL setting , where exploration is irrelevant . Beyond offline RL , and although this isn ’ t the main focus of the work , we also show the applicability of our method for accelerating RL by incorporating OPAL as a preprocessing step to standard online RL , few-shot imitation learning , and multi-task transfer learning . In all settings , we demonstrate that the use of OPAL can improve the speed and quality of downstream task learning . 2 RELATED WORK . Offline RL . Offline RL presents the problem of learning a policy from a fixed prior dataset of transitions and rewards . Recent works in offline RL ( Kumar et al. , 2019 ; Levine et al. , 2020 ; Wu et al. , 2019 ; Ghasemipour et al. , 2020 ; Jaques et al. , 2019 ; Fujimoto et al. , 2018 ) constrain the policy to be close to the data distribution to avoid the use of out-of-distribution actions ( Kumar et al. , 2019 ; Levine et al. , 2020 ) . To constrain the policy , some methods use distributional penalties , as measured by KL divergence ( Levine et al. , 2020 ; Jaques et al. , 2019 ) , MMD ( Kumar et al. , 2019 ) , or Wasserstein distance ( Wu et al. , 2019 ) . Other methods first sample actions from the behavior policy and then either clip the maximum deviation from those actions ( Fujimoto et al. , 2018 ) or just use those actions ( Ghasemipour et al. , 2020 ) during the value backup to stay within the support of the offline data . In contrast to these works , OPAL uses an offline dataset for unsupervised learning of a continuous space of primitives . The use of these primitives for downstream tasks implicitly constrains a learned primitive-directing policy to stay close to the offline data distribution . As we demonstrate in our experiments , the use of OPAL in conjunction with an off-the-shelf offline RL algorithm in this way can yield significant improvement compared to applying offline RL to the dataset directly . Online skill discovery . There are a number of recent works ( Eysenbach et al. , 2018 ; Nachum et al. , 2018a ; Sharma et al. , 2019 ) which use unsupervised objectives to discover skills and use the discovered skills for planning ( Sharma et al. , 2019 ) , few-shot imitation learning , or online RL ( Eysenbach et al. , 2018 ; Nachum et al. , 2018a ) . However , these works focus on online settings and assume access to the environment . In contrast , OPAL focuses on settings where a large dataset of diverse behaviors is provided but access to the environment is restricted . It leverages these static offline datasets to discover primitive skills with better state coverage and avoids the exploration issue of learning primitives from scratch . Hierarchical policy learning . Hierarchical policy learning involves learning a hierarchy of policies where a low-level policy acts as primitive skills and a high-level policy directs the low-level policy to solve a task . While some works ( Bacon et al. , 2017 ; Stolle & Precup , 2002 ; Peng et al. , 2019 ) learn a discrete set of lower-level policies , each behaving as a primitive skill , other works ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018b ; 2019a ; Hausman et al. , 2018 ) learn a continuous space of primitive skills representing the lower-level policy . These methods have mostly been applied in online settings . However , there have been some recent variants of the above works ( Lynch et al. , 2020 ; Shankar & Gupta , 2020 ; Krishnan et al. , 2017 ; Merel et al. , 2018 ) which extract skills from a prior dataset and using it for either performing tasks directly ( Lynch et al. , 2020 ) or learning downstream tasks ( Shankar & Gupta , 2020 ; Krishnan et al. , 2017 ; Merel et al. , 2018 ) with online RL . While OPAL is related to these works , we mainly focus on leveraging the learned primitives for asymptotically improving the performance of offline RL ; i.e. , both the primitive learning and the downstream task must be solved using a single static dataset . Furthermore , we provide performance bounds for OPAL and enumerate the specific properties an offline dataset should possess to guarantee improved downstream task learning , while such theoretical guarantees are largely absent from existing work . 3 PRELIMINARIES . We consider the standard Markov decision process ( MDP ) setting ( Puterman , 1994 ) , specified by a tupleM = ( S , A , P , µ , r , γ ) where S represents the state space , A represents the action space , P ( s′|s , a ) represents the transition probability , µ ( s ) represents the initial state distribution , r ( s , a ) ∈ ( −Rmax , Rmax ) represents the reward function , and γ ∈ ( 0 , 1 ) represents the discount factor . A policy π in this MDP corresponds to a function S → ∆ ( A ) , where ∆ ( A ) is the simplex over A . It induces a discounted future state distribution dπ , defined by dπ ( s ) = ( 1 − γ ) ∑∞t=0 γtP ( st = s|π ) , where P ( st = s|π ) is the probability of reaching the state s at time t by running π on M. For a positive integer k , we use dπk ( s ) = ( 1 − γk ) ∑∞ t=0 γ tkP ( stk = s|π ) to denote the everyk-step state distribution of π . The return of policy π in MDP M is defined as JRL ( π , M ) = 1 1−γEs∼dπ , a∼π ( a|s ) [ r ( s , a ) ] . We represent the reward- and discount-agnostic environment as a tuple E = ( S , A , P , µ ) . We aim to use a large , unlabeled , and undirected experience dataset D : = { τ ri : = ( st , at ) c−1t=0 } Ni=1 associated with E to extract primitives and improve offline RL for downstream task learning . To account for the fact that the dataset D may be generated by a mixture of diverse policies starting at diverse initial states , we assume D is generated by first sampling a behavior policy π ∼ Π along with an initial state s ∼ κ , where Π , κ represent some ( unknown ) distributions over policies and states , respectively , and then running π on E for c time steps starting at s0 = s. We define the probability of a sub-trajectory τ : = ( s0 , a0 , . . . , sc−1 , ac−1 ) in D under a policy π as π ( τ ) = κ ( s0 ) ∏c−1 t=1 P ( st|st−1 , at−1 ) ∏c−1 t=0 π ( at|st ) , and the conditional probability as π ( τ |s ) = 1 [ s = s0 ] ∏c−1 t=1 P ( st|st−1 , at−1 ) ∏c−1 t=0 π ( at|st ) . In this work , we will show how to apply unsupervised learning techniques to D to extract a continuous space of primitives πθ ( a|s , z ) , where z ∈ Z , the latent space inferred by unsupervised learning . We intend to use the learned πθ ( a|s , z ) to asymptotically improve the performance of offline RL for downstream task learning . For offline RL , we assume the existence of a dataset Dr : = { τ ri : = ( st , at , rt ) c−1t=0 } Ni=1 , corresponding to the same sub-trajectories in D labelled with MDP rewards . Additionally , we can use the extracted primitives for other applications such as few-shot imitation learning , online RL , and online multi-task transfer learning . We review the additional assumptions for these applications in Appendix A .

Summary: In the RL setting, this paper tackles the case where an agent may have access to large amounts of offline experience data. The objective of the work is to find an effective way to leverage this data for finding temporally extended primitive behaviors. The paper provides results that show how performing offline primitive learning can be leveraged for improving few-shot imitation learning as well as exploration and transfer on a variety of benchmark domains.