Skill-based Meta-Reinforcement Learning

While deep reinforcement learning methods have shown impressive results in robot learning , their sample inefficiency makes the learning of complex , longhorizon behaviors with real robot systems infeasible . To mitigate this issue , metareinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn . Yet , the application has been limited to short-horizon tasks with dense rewards . To enable learning long-horizon behaviors , recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations . While these approaches yield improved sample efficiency , millions of interactions with environments are still required to solve complex tasks . In this work , we devise a method that enables meta-learning on long-horizon , sparse-reward tasks , allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions . Our core idea is to leverage prior experience extracted from offline datasets during meta-learning . Specifically , we propose to ( 1 ) extract reusable skills and a skill prior from offline datasets , ( 2 ) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors , and ( 3 ) rapidly adapt the meta-trained policy to solve an unseen target task . Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve longhorizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets , while prior approaches in RL , meta-RL , and multi-task RL require substantially more environment interactions to solve the tasks . 1 INTRODUCTION . In recent years , deep reinforcement learning methods have achieved impressive results in robot learning ( Gu et al. , 2017 ; Andrychowicz et al. , 2020 ; Kalashnikov et al. , 2021 ) . Yet , existing approaches are sample inefficient , thus rendering the learning of complex behaviors through trial and error learning infeasible , especially on real robot systems . In contrast , humans are capable of effectively learning a variety of complex skills in only a few trials . This can be greatly attributed to our ability to learn how to learn new tasks quickly by efficiently utilizing previously acquired skills . Can machines likewise learn to how to learn by efficiently utilizing learned skills like humans ? Meta-reinforcement learning ( meta-RL ) holds the promise of allowing RL agents to acquire novel tasks with improved efficiency by learning to learn from a distribution of tasks ( Finn et al. , 2017 ; Rakelly et al. , 2019 ) . In spite of recent advances in the field , most existing metaRL algorithms are restricted to short-horizon , dense-reward tasks . To facilitate efficient learning on long-horizon , sparse-reward tasks , recent works aim to leverage experience from prior tasks in the form of offline datasets without additional reward and task annotations ( Lynch et al. , 2020 ; Pertsch et al. , 2020 ; Chebotar et al. , 2021 ) . While these methods can solve complex tasks with substantially improved sample efficiency over methods learning from scratch , millions of interactions with environments are still required to acquire long-horizon skills . In this work , we aim to take a step towards combining the capabilities of both learning how to quickly learn new tasks while also leveraging prior experience in the form of unannotated offline data ( see Figure 1 ) . Specifically , we aim to devise a method that enables meta-learning on complex , long-horizon tasks and can solve unseen target tasks with orders of magnitude fewer environment interactions than prior works . We propose to leverage the offline experience by extracting reusable skills – short-term behaviors that can be composed to solve unseen long-horizon tasks . We employ a hierarchical meta-learning scheme in which we meta-train a high-level policy to learn how to quickly reuse the extracted skills . To efficiently explore the learned skill space during meta-training , the high-level policy is guided by a skill prior which is also acquired from the offline experience data . We evaluate our method and prior approaches in deep RL , skill-based RL , meta-RL , and multi-task RL on two challenging continuous control environments : maze navigation and kitchen manipulation , which require long-horizon control and only provides sparse rewards . Experimental results show that our method can efficiently solve unseen tasks by exploiting meta-learning tasks and offline datasets , while prior approaches require substantially more samples or fail to solve the tasks . In summary , the main contributions of this paper are threefold : • To the best of our knowledge , this is the first work to combine meta-reinforcement learning algorithms with task-agnostic offline datasets that do not contain reward or task annotations . • We propose a method that combines meta-learning with offline data by extracting learned skills and a skill prior as well as meta-learning a hierarchical skill policy regularized by the skill prior . • We empirically show that our method is significantly more efficient at learning long-horizon sparse-reward tasks compared to prior methods in deep RL , skill-based RL , meta-RL , and multi-task RL . 2 RELATED WORK . Meta-Reinforcement Learning . Meta-RL approaches ( Duan et al. , 2016 ; Wang et al. , 2017 ; Finn et al. , 2017 ; Yu et al. , 2018 ; Rothfuss et al. , 2019 ; Gupta et al. , 2018 ; Vuorio et al. , 2018 ; Nagabandi et al. , 2019 ; Clavera et al. , 2019 ; 2018 ; Rakelly et al. , 2019 ; Vuorio et al. , 2019 ; Yang et al. , 2019 ; Zintgraf et al. , 2019 ; Humplik et al. , 2019 ; Zintgraf et al. , 2020 ; Liu et al. , 2021 ) hold the promise of allowing learning agents to quickly adapt to novel tasks by learning to learn from a distribution of tasks . Despite the recent advances in the field , most existing meta-RL algorithms are limited to short-horizon and dense-reward tasks . In contrast , we aim to develop a method that can meta-learn to solve long-horizon tasks with sparse rewards by leveraging offline datasets . Offline datasets . Recently , many works have investigated the usage of offline datasets for agent training . In particular , the field of offline reinforcement learning ( Levine et al. , 2020 ; Siegel et al. , 2020 ; Kumar et al. , 2020 ; Yu et al. , 2021 ) aims to devise methods that can perform RL fully offline from pre-collected data , without the need for environment interactions . However , these methods require target task reward annotations on the offline data for every new tasks that should be learned . These reward annotations can be challenging to obtain , especially if the offline data is collected from a diverse set of prior tasks . In contrast , our method is able to leverage offline datasets without any reward annotations . Offline Meta-RL . Another recent line of research aims to meta-learn from static , pre-collected datasets including reward annotations ( Mitchell et al. , 2021 ; Pong et al. , 2021 ; Dorfman et al. , 2021 ) . After meta-training with the offline datasets , these works aim to quickly adapt to a new task with only a small amount of data from that new task . In contrast to the aforementioned offline RL methods these works aim to adapt to unseen tasks and assume access to only limited data from the new tasks . However , in addition to reward annotations , these approaches often require that the offline training data is split into separate datasets for each training tasks , further limiting the scalability . Skill-based Learning . An alternative approach for leveraging offline data that does not require reward or task annotations is through the extraction of skills – reusable short-horizon behaviors . Methods for skill-based learning recombine these skills for learning unseen target tasks and converge substantially faster than methods that learn from scratch ( Lee et al. , 2018 ; Hausman et al. , 2018 ; Sharma et al. , 2020 ) . When trained from diverse datasets these approaches can extract a wide repertoire of skills and learn complex , long-horizon tasks ( Merel et al. , 2020 ; Lynch et al. , 2020 ; Pertsch et al. , 2020 ; Ajay et al. , 2021 ; Chebotar et al. , 2021 ; Pertsch et al. , 2021 ) . Yet , although they are more efficient than training from scratch , they still require a large number of environment interactions to learn a new task . Our method instead combines skills extracted from offline data with meta-learning , leading to significantly improved sample efficiency . 3 PROBLEM FORMULATION AND PRELIMINARIES . Our approach builds on prior work for meta-learning and learning from offline datasets and aims to combine the best of both worlds . In the following we will formalize our problem setup and briefly summarize relevant prior work . Problem Formulation . Following prior work on learning from large offline datasets ( Lynch et al. , 2020 ; Pertsch et al. , 2020 ; 2021 ) , we assume access to a dataset of state-action trajectories D = { st , at , ... } which is collected either across a wide variety of tasks or as “ play data ” with no particular task in mind . We thus refer to this dataset as task-agnostic . With a large number of data collection tasks , the dataset covers a wide variety of behaviors and can be used to accelerate learning on diverse tasks . Such data can be collected at scale , e.g . through autonomous exploration ( Hausman et al. , 2018 ; Sharma et al. , 2020 ; Dasari et al. , 2019 ) , human teleoperation ( Schaal et al. , 2005 ; Gupta et al. , 2019 ; Mandlekar et al. , 2018 ; Lynch et al. , 2020 ) , or from previously trained agents ( Fu et al. , 2020 ; Gulcehre et al. , 2020 ) . We additionally assume access to a set of meta-training tasks T = { T1 , . . . , TN } , where each task is represented as a Markov decision process ( MDP ) defined by a tuple { S , A , P , r , ρ , γ } of states , actions , transition probability , reward , initial state distribution , and discount factor . Our goal is to leverage both , the offline dataset D and the meta-training tasks T , to accelerate the training of a policy π ( a|s ) on a target task T ∗ which is also represented as an MDP . Crucially , we do not assume that T ∗ is a part of the set of training tasks T , nor that D contains demonstrations for solving T ∗ . Thus , we aim to design an algorithm that can leverage offline data and meta-training tasks for learning how to quickly compose known skills for solving an unseen target task . Next , we will describe existing approaches that either leverage offline data or meta-training tasks to accelerate target task learning . Then , we describe how our approach takes advantage of the best of both worlds . Skill-based RL . One successful approach for leveraging task-agnostic datasets for accelerating the learning of unseen tasks is though the transfer of reusable skills , i.e . short-horizon behaviors that can be composed to solve long-horizon tasks . Prior work in skill-based RL called Skill-Prior RL ( SPiRL , Pertsch et al . ( 2020 ) ) proposes an effective way to implement this idea . Specifically , SPiRL uses a task-agnostic dataset to learns two models : ( 1 ) a skill policy π ( a|s , z ) that decodes a latent skill representation z into a sequence of executable actions and ( 2 ) a prior over latent skill variables p ( z|s ) which can be leveraged to guide exploration in skill space . SPiRL uses these skills for learning new tasks efficiently by training a high-level skill policy π ( z|s ) that acts over the space of learned skills instead of primitive actions . The target task RL objective extends Soft Actor Critic ( SAC , Haarnoja et al . ( 2018 ) ) , a popular off-policy RL algorithm , by guiding the high-level policy with the learned skill prior : max π ∑ t E ( st , zt ) ∼ρπ [ r ( st , zt ) − αDKL ( π ( z|st ) , p ( z|st ) ) ] . ( 1 ) Here DKL denotes the Kullback-Leibler divergence between the policy and skill prior , and α is a weighting coefficient . Off-Policy Meta-RL . Rakelly et al . ( 2019 ) introduced an off-policy meta-RL algorithm called probabilistic embeddings for actor-critic RL ( PEARL ) that leverages a set of training tasks T to enable quick learning of new tasks . Specifically , PEARL leverages the meta-training tasks for learning a task encoder q ( e|c ) . This encoder takes in a small set of state-action-reward transitions c and produces a task embedding e. This embedding is used to condition the actor π ( a|s , z ) and critic Q ( s , a , e ) . In PEARL , actor , critic and task encoder are trained by jointly maximizing the obtained reward and the policy ’ s entropyH ( Haarnoja et al. , 2018 ) : max π ET ∼pT , e∼q ( ·|cT ) [ ∑ t E ( st , at ) ∼ρπ|e [ rT ( st , at ) + αH ( π ( a|st , e ) ) ] ] . ( 2 ) Additionally , the task embedding output of the task encoder is regularized towards a constant prior distribution p ( e ) .

Summary: This paper combines skill-based with meta RL using training tasks.  The skill-based method (SPiRL, Pertsch et al. 2020) forms a skill embedding that represents $K$-step state-action trajectories.  An action policy provides an action distribution conditioned on the currently-active skill, and a (higher-level) skill policy is trained using RL with the skill embedding as its action space.  Simultaneously a skill prior conditioned on the starting state of a given skill is trained.  Once trained, the skill embedding, the skill-conditioned action policy, and the skill prior are held fixed.  The off-policy meta RL method (PEARL, Rakelly et al. 2019) forms a task embedding that represents small sets of skills active during a given task. (This is how I understand it; the paper does not explicitly state this. Rakelly et al. do not use skills; their task embedding represents small sets of state-action-state-reward sequences.)  A skill policy provides a skill distribution conditioned on the currently-active task.  Its training is regularized using the above skill prior. Once trained, the task embedding is held fixed.  Up to here, the system is trained in a task-agnostic manner.  To learn a target task, the system further trains the skill policy, conditioned on the target task, again regularized using the skill prior.