Year: 2021

Abstract:
While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, longhorizon behaviors with real robot systems infeasible. To mitigate this issue, metareinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve longhorizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.

text: While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, longhorizon behaviors with real robot systems infeasible. To mitigate this issue, metareinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve longhorizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.

heading: 1 INTRODUCTION
text: In recent years, deep reinforcement learning methods have achieved impressive results in robot learning (Gu et al., 2017; Andrychowicz et al., 2020; Kalashnikov et al., 2021). Yet, existing approaches are sample inefficient, thus rendering the learning of complex behaviors through trial and error learning infeasible, especially on real robot systems. In contrast, humans are capable of effectively learning a variety of complex skills in only a few trials. This can be greatly attributed to our ability to learn how to learn new tasks quickly by efficiently utilizing previously acquired skills.
Can machines likewise learn to how to learn by efficiently utilizing learned skills like humans? Meta-reinforcement learning (meta-RL) holds the promise of allowing RL agents to acquire novel tasks with improved efficiency by learning to learn from a distribution of tasks (Finn et al., 2017; Rakelly et al., 2019). In spite of recent advances in the field, most existing metaRL algorithms are restricted to short-horizon, dense-reward tasks. To facilitate efficient learning on long-horizon, sparse-reward tasks, recent works aim to leverage experience from prior tasks in the form of offline datasets without additional reward and task annotations (Lynch et al., 2020; Pertsch et al., 2020; Chebotar et al.,
2021). While these methods can solve complex tasks with substantially improved sample efficiency
over methods learning from scratch, millions of interactions with environments are still required to acquire long-horizon skills.
In this work, we aim to take a step towards combining the capabilities of both learning how to quickly learn new tasks while also leveraging prior experience in the form of unannotated offline data (see Figure 1). Specifically, we aim to devise a method that enables meta-learning on complex, long-horizon tasks and can solve unseen target tasks with orders of magnitude fewer environment interactions than prior works.
We propose to leverage the offline experience by extracting reusable skills – short-term behaviors that can be composed to solve unseen long-horizon tasks. We employ a hierarchical meta-learning scheme in which we meta-train a high-level policy to learn how to quickly reuse the extracted skills. To efficiently explore the learned skill space during meta-training, the high-level policy is guided by a skill prior which is also acquired from the offline experience data.
We evaluate our method and prior approaches in deep RL, skill-based RL, meta-RL, and multi-task RL on two challenging continuous control environments: maze navigation and kitchen manipulation, which require long-horizon control and only provides sparse rewards. Experimental results show that our method can efficiently solve unseen tasks by exploiting meta-learning tasks and offline datasets, while prior approaches require substantially more samples or fail to solve the tasks.
In summary, the main contributions of this paper are threefold:
• To the best of our knowledge, this is the first work to combine meta-reinforcement learning algorithms with task-agnostic offline datasets that do not contain reward or task annotations.
• We propose a method that combines meta-learning with offline data by extracting learned skills and a skill prior as well as meta-learning a hierarchical skill policy regularized by the skill prior.
• We empirically show that our method is significantly more efficient at learning long-horizon sparse-reward tasks compared to prior methods in deep RL, skill-based RL, meta-RL, and multi-task RL.

heading: 2 RELATED WORK
text: Meta-Reinforcement Learning. Meta-RL approaches (Duan et al., 2016; Wang et al., 2017; Finn et al., 2017; Yu et al., 2018; Rothfuss et al., 2019; Gupta et al., 2018; Vuorio et al., 2018; Nagabandi et al., 2019; Clavera et al., 2019; 2018; Rakelly et al., 2019; Vuorio et al., 2019; Yang et al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Zintgraf et al., 2020; Liu et al., 2021) hold the promise of allowing learning agents to quickly adapt to novel tasks by learning to learn from a distribution of tasks. Despite the recent advances in the field, most existing meta-RL algorithms are limited to short-horizon and dense-reward tasks. In contrast, we aim to develop a method that can meta-learn to solve long-horizon tasks with sparse rewards by leveraging offline datasets.
Offline datasets. Recently, many works have investigated the usage of offline datasets for agent training. In particular, the field of offline reinforcement learning (Levine et al., 2020; Siegel et al., 2020; Kumar et al., 2020; Yu et al., 2021) aims to devise methods that can perform RL fully offline from pre-collected data, without the need for environment interactions. However, these methods require target task reward annotations on the offline data for every new tasks that should be learned. These reward annotations can be challenging to obtain, especially if the offline data is collected from a diverse set of prior tasks. In contrast, our method is able to leverage offline datasets without any reward annotations.
Offline Meta-RL. Another recent line of research aims to meta-learn from static, pre-collected datasets including reward annotations (Mitchell et al., 2021; Pong et al., 2021; Dorfman et al., 2021). After meta-training with the offline datasets, these works aim to quickly adapt to a new task with only a small amount of data from that new task. In contrast to the aforementioned offline RL methods these works aim to adapt to unseen tasks and assume access to only limited data from the new tasks. However, in addition to reward annotations, these approaches often require that the offline training data is split into separate datasets for each training tasks, further limiting the scalability.
Skill-based Learning. An alternative approach for leveraging offline data that does not require reward or task annotations is through the extraction of skills – reusable short-horizon behaviors. Methods for skill-based learning recombine these skills for learning unseen target tasks and converge substantially faster than methods that learn from scratch (Lee et al., 2018; Hausman et al., 2018; Sharma et al., 2020). When trained from diverse datasets these approaches can extract a wide repertoire of skills and learn complex, long-horizon tasks (Merel et al., 2020; Lynch et al., 2020; Pertsch et al., 2020; Ajay et al., 2021; Chebotar et al., 2021; Pertsch et al., 2021). Yet, although they are more efficient than training from scratch, they still require a large number of environment interactions to learn a new task. Our method instead combines skills extracted from offline data with meta-learning, leading to significantly improved sample efficiency.

heading: 3 PROBLEM FORMULATION AND PRELIMINARIES
text: Our approach builds on prior work for meta-learning and learning from offline datasets and aims to combine the best of both worlds. In the following we will formalize our problem setup and briefly summarize relevant prior work.
Problem Formulation. Following prior work on learning from large offline datasets (Lynch et al., 2020; Pertsch et al., 2020; 2021), we assume access to a dataset of state-action trajectories D = {st, at, ...} which is collected either across a wide variety of tasks or as “play data” with no particular task in mind. We thus refer to this dataset as task-agnostic. With a large number of data collection tasks, the dataset covers a wide variety of behaviors and can be used to accelerate learning on diverse tasks. Such data can be collected at scale, e.g. through autonomous exploration (Hausman et al., 2018; Sharma et al., 2020; Dasari et al., 2019), human teleoperation (Schaal et al., 2005; Gupta et al., 2019; Mandlekar et al., 2018; Lynch et al., 2020), or from previously trained agents (Fu et al., 2020; Gulcehre et al., 2020). We additionally assume access to a set of meta-training tasks T = {T1, . . . , TN}, where each task is represented as a Markov decision process (MDP) defined by a tuple {S,A,P, r, ρ, γ} of states, actions, transition probability, reward, initial state distribution, and discount factor.
Our goal is to leverage both, the offline dataset D and the meta-training tasks T, to accelerate the training of a policy π(a|s) on a target task T ∗ which is also represented as an MDP. Crucially, we do not assume that T ∗ is a part of the set of training tasks T, nor that D contains demonstrations for solving T ∗. Thus, we aim to design an algorithm that can leverage offline data and meta-training tasks for learning how to quickly compose known skills for solving an unseen target task. Next, we will describe existing approaches that either leverage offline data or meta-training tasks to accelerate target task learning. Then, we describe how our approach takes advantage of the best of both worlds.
Skill-based RL. One successful approach for leveraging task-agnostic datasets for accelerating the learning of unseen tasks is though the transfer of reusable skills, i.e. short-horizon behaviors that can be composed to solve long-horizon tasks. Prior work in skill-based RL called Skill-Prior RL (SPiRL, Pertsch et al. (2020)) proposes an effective way to implement this idea. Specifically, SPiRL uses a task-agnostic dataset to learns two models: (1) a skill policy π(a|s, z) that decodes a latent skill representation z into a sequence of executable actions and (2) a prior over latent skill variables p(z|s) which can be leveraged to guide exploration in skill space. SPiRL uses these skills for learning new tasks efficiently by training a high-level skill policy π(z|s) that acts over the space of learned skills instead of primitive actions. The target task RL objective extends Soft Actor Critic (SAC, Haarnoja et al. (2018)), a popular off-policy RL algorithm, by guiding the high-level policy with the learned skill prior:
max π ∑ t E(st,zt)∼ρπ [ r(st, zt)− αDKL ( π(z|st), p(z|st) )] . (1)
Here DKL denotes the Kullback-Leibler divergence between the policy and skill prior, and α is a weighting coefficient.
Off-Policy Meta-RL. Rakelly et al. (2019) introduced an off-policy meta-RL algorithm called probabilistic embeddings for actor-critic RL (PEARL) that leverages a set of training tasks T to enable quick learning of new tasks. Specifically, PEARL leverages the meta-training tasks for learning a task encoder q(e|c). This encoder takes in a small set of state-action-reward transitions c and produces a task embedding e. This embedding is used to condition the actor π(a|s, z) and critic Q(s, a, e). In PEARL, actor, critic and task encoder are trained by jointly maximizing the obtained
reward and the policy’s entropyH (Haarnoja et al., 2018):
max π
ET ∼pT ,e∼q(·|cT ) [∑
t
E(st,at)∼ρπ|e [ rT (st, at) + αH ( π(a|st, e) )]] . (2)
Additionally, the task embedding output of the task encoder is regularized towards a constant prior distribution p(e).

heading: 4 APPROACH
text: We propose Skill-based Meta-Policy Learning (SiMPL), an algorithm for jointly leveraging offline data as well as a set of meta-training tasks to accelerate the learning of unseen target tasks. Our method has three phases: (1) skill extraction: we extract reusable skills and a skill prior from the offline data (Section 4.1), (2) skill-based meta-training: we utilize the meta-training tasks to learn how to leverage the extracted skills and skill prior to efficiently solve new tasks (Section 4.2), (3) target task learning: we fine-tune the meta-trained policy to rapidly adapt to solve an unseen target task (Section 4.3). An illustration of the proposed method is shown in Figure 2.

heading: 4.1 SKILL EXTRACTION
text: To acquire a set of reusable skills from the offline dataset D, we leverage the skill extraction approach proposed in Pertsch et al. (2020). Concretely, we jointly train (1) a skill encoder q(z|s0:K , a0:K−1) that embeds an K-steps trajectory randomly cropped from the sequences in D into a low-dimensional skill embedding z, and (2) a low-level skill policy π(at|st, z) that is trained with behavioral cloning to reproduce the action sequence a0:K−1 given the skill embedding. To learn a smooth skill representation, we regularize the output of the skill encoder with a unit Gaussian prior distribution, and weight this regularization by a coefficient β (Higgins et al., 2017):
max q,π Ez∼q [K−1∏ t=0
log π(at|st, z)︸ ︷︷ ︸ behavioral cloning
−βDKL ( q(z|s0:K , a0:K−1),N (0, I) ) ︸ ︷︷ ︸
embedding regularization
] . (3)
Additionally, we follow Pertsch et al. (2020) and learn a skill prior p(z|s) that captures the distribution of skills likely to be executed in a given state under the training data distribution. The prior is trained to match the output of the skill encoder: minpDKL ( bq(z|s0:K , a0:K−1)c, p(z|s0) ) . Here b·c indicates that gradient flow is stopped into the skill encoder for training the skill prior.

heading: 4.2 SKILL-BASED META-TRAINING
text: We aim to learn a policy that can quickly learn to leverage the extracted skills to solve new tasks. We leverage off-policy meta-RL (see Section 3) to learn such a policy using our set of meta-training tasks T. Similar to PEARL (Rakelly et al., 2019), we train a task-encoder that takes in a set of sampled transitions and produces a task embedding e. Crucially, we leverage our learned skills by training a task-embedding-conditioned policy over skills instead of primitive actions: π(z|s, e), thus equipping the policy with a set of useful pre-trained behaviors and reducing the meta-training task to learning how to combine these behaviors instead of learning them from scratch. We find that this usage of offline data through learned skills is crucial for enabling meta-training on complex, long-horizon tasks (see Section 5).
Prior work has shown that the efficiency of RL on learned skill spaces can be substantially improved by guiding the policy with a learned skill prior (Pertsch et al., 2020; Ajay et al., 2021). Thus, instead of regularizing with a maximum entropy objective as done in prior work on off-policy meta-RL (Rakelly et al., 2019), we propose to regularize the meta-training policy with our pre-trained skill prior, leading to the following meta-training objective:
max π
ET ∼pT ,e∼q(·|cT ) [∑
t
E(st,zt)∼ρπ|e [ rT (st, zt)− αDKL ( π(z|st, e), p(z|st) )]] . (4)
where α determines the strength of the prior regularization. We automatically tune α via dual gradient descent by choosing a target divergence δ between policy and prior (Pertsch et al., 2020).
To compute the task embedding e, we used multiple different sizes of c. We found that we can increase training stability by adjusting the strength of the prior regularization to the size of the conditioning set. Intuitively, when the high-level policy is conditioned on only a few transitions, i.e. when the set c is small, it has only little information about the task at hand and should thus be regularized stronger towards the task-agnostic skill prior. Conversely, when c is large, the policy likely has more information about the target task and thus should be allowed to deviate from the skill prior more to solve the task, i.e. have a weaker regularization strength.
To implement this intuition, we employ a simple approach: we define two target divergences δ1 and δ2 and associated auto-tuned coefficients α1 and α2 with δ1 < δ2. We regularize the policy using the larger coefficient α1 with small conditioning transition set and otherwise we regularize using the smaller coefficient α2. We found this technique simple yet sufficient in our experiments and leave the investigation of more sophisticated regularization approaches for future work.

heading: 4.3 TARGET TASK LEARNING
text: When a target task is given, we aim to leverage the meta-trained policy for quickly learning how to solve it. Intuitively, the policy should first explore different skill options to learn about the task at hand and then rapidly narrow its output distribution to those skills that solve the task. We implement this intuition by first collecting a small set of conditioning transitions c∗ from the target task by exploring with the meta-trained policy. Since we have no information about the target task at this stage, we explore the environment by conditioning our pre-trained policy with task embeddings sampled from the task prior p(e). Then, we encode this set of transitions into a target task embedding e∗ ∼ q(e|c∗). By conditioning our meta-trained high-level policy on this encoding, we can rapidly narrow its skill distribution to skills that solve the given target task: π(z|s, e∗). Empirically, we find that this policy is often already able to achieve high success rates on the target task. Note that only very few interactions with the environment for collecting c∗ are required for learning a complex, long-horizon and unseen target task with sparse reward. This is substantially more efficient than prior approaches such as SPiRL that require orders of magnitude more target task interactions for achieving comparable performance.
To further improve the performance on the target task, we fine-tune the conditioned policy with target task rewards while guiding its exploration with the pre-trained skill prior1:
max π
Ee∗∼q(·|c∗) [∑
t
E(st,zt)∼ρπ|e∗ [ rT ∗(st, zt)− αDKL ( π(z|st, e∗), p(z|st) )]] . (5)
In practice, we propose several techniques for stabilizing meta-training and fine-tuning: (1) adaptively regularizing the policy based on the size of the conditioning trajectory set as described in Section 4.2, (2) parameterizing the policy as a residual policy that outputs differences to the pre-trained skill prior instead of the approach from Pertsch et al. (2020) that directly fine-tunes the skill prior, and (3) initializing the Q-function and α parameter during fine-tuning with meta-trained parameters instead of randomly initialized networks. We discuss these techniques in detail in Section D.

heading: 5 EXPERIMENTS
text: Our experiments aim to answer the following questions: (1) Can our proposed method learn to efficiently solve long-horizon, sparse reward tasks? (2) Is it crucial to utilize offline datasets to achieve this? (3) How can we best leverage the training tasks for efficient learning of target tasks? (4) How does the training task distribution affect the target task learning?

heading: 5.1 EXPERIMENTAL SETUP
text: We evaluate our approach in two challenging continuous control environments: maze navigation and kitchen manipulation environment, as illustrated in Figure 3. While meta-RL algorithms are typically evaluated on tasks that span only a few dozen time steps and provide dense rewards (Finn et al., 2017; Rothfuss et al., 2019; Rakelly et al., 2019; Zintgraf et al., 2020), our tasks require to learn long-horizon behaviors over hundreds of time steps from sparse reward feedback and thus pose a new challenge to meta-learning algorithms.

heading: 5.1.1 MAZE NAVIGATION
text: Environment. This 2D maze navigation domain based on the maze navigation problem in Fu et al. (2020) requires long-horizon control with hundreds of steps for a successful episode and only provides sparse reward feedback upon reaching the goal. The observation space of the agent consists of its 2D position and velocity and it acts via planar, continuous velocity commands.
Offline Dataset & Meta-training / Target Tasks. Following Fu et al. (2020) we collect a taskagnostic offline dataset by randomly sampling start-goal locations in the maze and using a planner to
1Other regularization distributions are possible during fine-tuning, e.g. the high-level policy conditioned on task prior samples p(z|s, e ∼ p(e)) or the target task embedding conditioned policy p(z|s, e∗) before finetuning. Yet, we found the regularization with the pre-trained task-agnostic skill prior to work best in our experiments.
generate a trajectory that reaches from start to goal. Note that the trajectories are not annotated with any reward or task labels (i.e. which start-goal location is used for producing each trajectory). To generate a set of meta-training and target tasks, we fix the agent’s initial position in the center of the maze and sample 40 random goal locations for meta-training and another set of 10 goals for target tasks. All meta-training and target tasks use the same sparse reward formulation. More details can be found in Section F.1.

heading: 5.1.2 KITCHEN MANIPULATION
text: Environment. The FrankaKitchen environment of Gupta et al. (2019) requires the agent to control a 7-DoF robot arm via continuous joint velocity commands and complete a sequence of manipulation tasks like opening the microwave or turning on the stove. Successful episodes span 300-500 steps and the agent is only provided a sparse reward signal upon successful completion of a subtask.
Offline Dataset & Meta-training / Target Tasks. We leverage a dataset of 600 human-teleoperated manipulation sequences of Gupta et al. (2019) for offline pre-training. In each trajectory, the robot executes a sequence of four subtasks. We then define a set of 23 meta-training tasks and 10 target tasks that in turn require the consecutive execution of four subtasks (see Figure 3 for examples). Note that each task consists of a unique combination of subtasks. More details can be found in Section F.2.

heading: 5.2 BASELINES
text: We compare SiMPL to prior approaches in RL, skill-based RL, meta-RL, and multi-task RL.
• SAC (Haarnoja et al., 2018) is a state of the art deep RL algorithm. It learns to solve a target task from scratch without leveraging the offline dataset nor the meta-training tasks.
• SPiRL (Pertsch et al., 2020) is a method designed to leverage offline data through the transfer of learned skills. It acquires skills and a skill prior from the offline dataset but does not utilize the meta-training tasks. This investigates the benefits our method can obtain from leveraging the meta-training tasks.
• PEARL (Rakelly et al., 2019) is a state of the art off-policy meta-RL algorithm that learns a policy which can quickly adapt to unseen test tasks. It learns from the meta-training tasks but does not use the offline dataset. This examines the benefits of using learned skills in meta-RL.
• PEARL-ft demonstrates the performance of a PEARL (Rakelly et al., 2019) model further fine-tuned on a target task using SAC (Haarnoja et al., 2018).
• Multi-task RL (MTRL) is a multi-task RL baseline which learns from the meta-training tasks by distilling individual policies specialized in each task into a shared policy, similar to Distral (Teh et al., 2017). Each individual policy is trained using SPiRL by leveraging skills extracted from the offline dataset. Therefore, it utilizes both the meta-training tasks and offline dataset similar to our method. This provides a direct comparison of multi-task learning (MTRL) from the training tasks vs. meta-learning using them (ours).
More implementation details on the baselines can be found in Section E.

heading: 5.3 RESULTS
text: We present the quantitative results in Figure 4 and the qualitative results on the maze navigation domain in Figure 5. In Figure 4, SiMPL demonstrates much better sample efficiency for learning the unseen target tasks compared to all the baselines. Without leveraging the offline dataset and meta-training tasks, SAC is not able to make learning progress on most of the target tasks. While PEARL is first trained on the meta-training tasks, it still achieves poor performance on the target tasks and fine-tuning it (PEARL-ft) does not yield significant improvement. We believe this is because both environments provide only sparse rewards yet require the model to exhibit long-horizon and complex behaviors, which is known to be difficult for meta-RL methods (Mitchell et al., 2021).
On the other hand, by first extracting skills and acquiring a skill prior from the offline dataset, SPiRL’s performance consistently improves with more samples from the target tasks. Yet, it requires significantly more environment interactions than our method to solve the target tasks since the policy is optimized using vanilla RL, which is not designed to learn to quickly learn new tasks. While the multi-task RL (MTRL) baseline first learns a multi-task policy from the meta-training tasks, its

heading: SiMPL (Ours)
text: sample efficiency is similar to SPiRL on target task learning, which highlights the strength of our proposed method – meta-learning from the meta-training tasks for fast target task learning.
Compared to the baselines, our method learns the target tasks much quicker. Within only a few episodes the policy converges to solve more than 80% of the target tasks in the maze environment and two out of four subtasks in the kitchen manipulation environment. The prior-regularized fine-tuning then continues to improve performance. The rapidly increasing performance and the overall faster convergence show the benefits of leveraging meta-training tasks in addition to learning from offline data: by first learning to learn how to quickly solve tasks using the extracted skills and the skill prior, our policy can efficiently solve the target tasks.
The qualitative results presented in Figure 5 show that all the methods that leverage the offline dataset (i.e. SiMPL, SPiRL, and MTRL) effectively explore the maze in the first episode. Then, SiMPL converges with much fewer episodes compared to SPiRL and MTRL, underlining the effectiveness of meta-training. In contrast, PEARL-ft is not able to make learning progress, justifying the necessity of employing offline datasets for acquiring long-horizon, complex behaviors.

heading: 5.4 META-TRAINING TASK DISTRIBUTION ANALYSIS
text: In this section, we aim to investigate the effect of the meta-training task distribution on our skill-based meta-training and target task learning phases. Specifically, we examine the effect of (1) the number of tasks in the meta-training task distribution and (2) the alignment between a meta-training task
distribution and target task distribution. We conduct experiments and analyses in the maze navigation domain. More details on task distributions can be found in Section F.1.
Number of meta-training tasks. To investigate how the number of meta-training tasks affects the performance of our method, we train our method with fewer numbers meta-training tasks (i.e. 10 and 20) and evaluate it with the same set of target tasks. The quantitative results presented in Figure 6(a) suggest that even with sparser meta-training task distributions (i.e. fewer numbers of meta-training tasks), SiMPL is still more sample efficient compared to the best-performing baseline (i.e. SPiRL).
Meta-train / test task alignment. We aim to examine if a model trained on a meta-training task distribution that aligns better/worse with the target tasks would yield improved/deteriorated performance. To this end, we create biased meta-training / test task distributions: we create a meta-train set by sampling goal locations from only the top 25% portion of the maze (TTRAIN-TOP). To rule out the effect of the density of the task distribution, we sample 10 (i.e. 40 × 25%) meta-training tasks. Then, we create two target task distributions that have good and bad alignment with this meta-training distribution respectively by sampling 10 target tasks from the top 25% portion of the maze (TTARGET-TOP) and 10 target tasks from the bottom 25% portion of the maze (TTARGET-BOTTOM). Figure 6(b) and Figure 6(c) present the target task learning efficiency for models trained with good task alignment (meta-train on TTRAIN-TOP, learn target tasks from TTARGET-TOP) and bad task alignment (meta-train on TTRAIN-TOP, learn target tasks from TTARGET-BOTTOM), respectively. The results demonstrate that SiMPL can achieve improved performance when trained on a better aligned metatraining task distribution. On the other hand, not surprisingly, SiMPL and MTRL perform slightly worse compared to SPiRL when trained with misaligned meta-training tasks (see Figure 6(c)). This is expected given that SPiRL does not learn from the misaligned meta-training tasks. In summary, from Figure 6, we can conclude that meta-learning from either a diverse task distribution or a better informed task distribution can yield improved performance for our method.

heading: 6 CONCLUSION
text: We propose a skill-based meta-RL method, dubbed SiMPL, that can meta-learn on long-horizon tasks by leveraging prior experience in the form of large offline datasets without additional reward and task annotations. Specifically, our method first learns to extracts reusable skills and a skill prior from the offline data. Then, we propose to meta-trains a high-level policy that leverages these skills for efficient learning of unseen target tasks. To effectively utilize learned skills, the high-level policy is regularized by the acquired prior. The experimental results on challenging continuous control long-horizon navigation and manipulation tasks with sparse rewards demonstrate that our method outperforms the prior approaches in deep RL, skill-based RL, meta-RL, and multi-task RL. In the future, we aim to demonstrate the scalability of our method to high-DoF continuous control problems on real robotic systems to show the benefits of our improved sample efficiency.

heading: APPENDIX

heading: Table of Contents

heading: A Meta-reinforcement Learning Method Ablation 13

heading: B Learning Efficiency on Target Tasks with Few Episodes of Experience 14
text: C Investigating Offline Data vs. Target Domain Shift 15
D Implementation Details on Our Method 16 D.1 Model architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 D.2 Training details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
E Implementation Details on Baselines 18 E.1 SAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 E.2 PEARL and PEARL-ft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 E.3 SPiRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 E.4 Multi-task RL (MTRL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

heading: F Meta-training Tasks and Target Tasks. 19
text: F.1 Maze Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 F.2 Kitchen Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

heading: A META-REINFORCEMENT LEARNING METHOD ABLATION
text: In this section, we compare the learning efficiency of different meta-RL algorithms with respect to the length of the training tasks. Specifically, we hypothesize that our approach SiMPL, which extracts temporally extended skills from offline experience, is better suited for learning long-horizon tasks than prior meta-RL algorithms. To cleanly investigate the importance of the temporally extended skills vs. the importance of using prior experience we include two additional comparisons to methods that leverage prior experience for meta-RL but via flat behavioral cloning instead of through temporally extended skills:
• BC+PEARL first learns a behavior cloning (BC) policy through supervised learning from the offline dataset. Then, analogous to our approach SiMPL, during the meta-training phase, a task encoder and a meta-learned policy are meta-trained with the BC policy constrained SAC objective. For fair comparison, we use the same residual policy parameterization as described in Section D.1.3.
• BC+MAML follows the same learning procedure described above, but uses MAML (Finn et al., 2017) for meta-training instead of PEARL. We follow the original learning objective in Finn et al. (2017) (i.e. using REINFORCE (Williams, 1992) for task adaptation, and using TRPO (Schulman et al., 2017) for meta-policy optimization).
We compare these methods as well as the standard meta-RL approach PEARL (Rakelly et al., 2019) on three meta-training tasks distributions of increasing complexity in the maze navigation environment (see Figure 7): (1) short-range goals with small variance TTRAIN-EASY, (2) short-range goals with larger variance TTRAIN-MEDIUM, and (3) long-range goals with large variance TTRAIN-HARD, which we used in our original maze experiments. By increasing variance and length of the tasks in each task distribution, we can investigate the learning capability of the meta-RL algorithms.
We present the quantitative results in Figure 8 and the corresponding qualitative analysis in Figure 9. On the simplest task distribution we find that all approaches can learn to solve the tasks efficiently, except for BC+MAML. While the latter also learns to solve the task eventually (see performance upon convergence as dashed orange line in Figure 8(a)) it uses on-policy meta-RL and thus requires substantially more environment interactions during meta-training. We thus only consider the more sample efficient BC+PEARL off-policy meta-RL method in the remaining comparisons.
On the more complex task distributions TTRAIN-MEDIUM and TTRAIN-HARD, we find that using prior data for meta-learning is generally beneficial: both BC+PEARL and SiMPL learn more efficiently on the task distribution of medium difficulty TTRAIN-MEDIUM, as shown in Figure 8(b), since the policy pre-trained from offline data allows for more efficient exploration during meta-training. Importantly, on the hardest task distribution TTRAIN-HARD, as shown in Figure 8(c), which consists exclusively of long-horizon tasks, we find that only SiMPL is able to effectively learn, highlighting the importance of leveraging the offline data via temporally extended skills instead of flat behavioral cloning. This supports our intuition that the abstraction provided by skills is particularly beneficial for meta-learning on long-horizon tasks.

heading: B LEARNING EFFICIENCY ON TARGET TASKS WITH FEW EPISODES OF EXPERIENCE
text: In this section, we examine the data efficiency of the compared methods on the target tasks, specifically when provided with only a few (<20) episodes of online interaction with an unseen target task. Being able to learn new tasks this quickly is a major strength of meta-RL approaches (Finn et al., 2017; Rakelly et al., 2019). We hypothesize that our skill-based meta-RL algorithm SiMPL can learn similarly fast, even on long-horizon, sparse-reward tasks.
In our original evaluations in Section 5, we used 20 episodes of initial exploration to condition our meta-trained policy. In Figure 10, we instead compare performance of different approaches when only provided with very few episodes of online interactions. We find that SiMPL learns to solve the unseen tasks substantially faster than all alternative approaches. On the kitchen manipulation tasks our approach learns to almost solve two out of four subtasks within a time span equivalent to only a
few minutes of real-robot execution time. In contrast, prior meta-RL methods struggle at making progress at all on such long-horizon tasks, showing the benefit of combining meta-RL with prior offline experience.
C INVESTIGATING OFFLINE DATA VS. TARGET DOMAIN SHIFT
To provide more insights on comparing SiMPL and SPiRL (Pertsch et al., 2020), we evaluate SiMPL in the maze navigation task setup proposed in Pertsch et al. (2020). This tests whether our approach can scale to image-based observations: Pertsch et al. (2020) use 32 × 32px observations centered around the agent. Even more importantly, it allows us to investigate the robustness of the approach to the domain shifts between the offline pre-training data and the target task: we use the maze navigation
offline dataset from Pertsch et al. (2020) which was collected on randomly sampled 20× 20 maze layouts and test on tasks in the unseen, randomly sampled 40× 40 test maze layout from Pertsch et al. (2020). We visualize the meta-training task distribution in Figure 11(a) and the target task distribution in Figure 11(b).
We compare the performance of our method to the best-performing baseline, SPiRL (Pertsch et al., 2020), in Figure 11(c). Similar to the result presented in Figure 4, SiMPL can learn the target task faster by combining skills learned from the offline dataset with efficient meta-training. This shows that our approach can scale to image-based inputs and is robust to substantial domain shifts between the offline pre-training data and the target tasks.
D IMPLEMENTATION DETAILS ON OUR METHOD
In this section, we describe the additional implementation details on our proposed method. The details on model architecture is presented in Section D.1, followed by the training detailed described in Section D.2.

heading: D.1 MODEL ARCHITECTURE
text: We describe the details on our model architecture in this section.

heading: D.1.1 SKILL PRIOR
text: We followed architecture and learning procedure of Pertsch et al. (2020) for learning a low-level skill policy and a skill prior. Please refer to Pertsch et al. (2020) for more details on the architectures for learning skills and skill priors from offline datasets.

heading: D.1.2 TASK ENCODER
text: Following Rakelly et al. (2019), our task encoder is a permutation invarient neural network. Specifically, we adopt Set Transformer (Lee et al., 2019) that consists of layers [2× ISAB32 → PMA1 → 3 × MLP] for expressive and efficient set encoding. All the hidden layers are 128-dimensional and all attention layers have 4 attention heads. The encoder takes a set of high-level transitions as input, where each transition is a vector concatenation of high-level transition tuple. The output of the encoder is (µe, σe) which are the parameters of Gaussian task posterior p(e|c) = N (e;µe, σe). We varied task vector dimension dim(e) depends on task distribution complexity. dim(e) = 10 for Kitchen Manipulation, dim(e) = 6 for Maze Navigation with 40 meta-training tasks, and dim(e) = 5, otherwise.

heading: D.1.3 POLICY
text: We parameterize our policy with neural network. We employed 4-layer MLPs with 256 hidden units for Maze Navigation, and 6-layer MLPs with 128 hidden unit for Kitchen Manipulation experiment. Instead of direct parameterization of policy, the network output is added to skill-prior to make learning more stable. Specifically, the policy network takes concatenation of (s, e) as input, and then outputs residual parameters (µz, log σz) to skill-prior distribution p(z|s) = N (z|µp, σp). Resulting distribution by this residual parameterization is π(z|s) = N (z|µp + µz, exp(log σp + log σz))

heading: D.1.4 CRITIC
text: The critic network takes concatenation of s, e, and skill z as input and outputs an estimation of task-conditioned Q-value Q(s, z, e). We employ double Q networks (Van Hasselt et al., 2016) to mitigate Q-value overestimation. The architecture of critic follows the policy.

heading: D.2 TRAINING DETAILS
text: For all the network updates, we used Adam optimizer (Kingma & Ba, 2015) with a learning rate of 3e− 4, β1 = 0.9, and β2 = 0.999. We describe the training details of the skill-based meta-training phase in Section D.2.1 and the target task learning phase Section D.2.2.

heading: D.2.1 SKILL-BASED META-TRAINING
text: Our meta-training procedure is similar to the procedure adopted in (Rakelly et al., 2019). Encoder and critics networks are updated to minimize MSE between Q-value prediction and target Q value. Policy network is updated to optimize Equation 4 without updating the encoder network. All network are updated with the average gradients of 20 randomly sampled tasks. Each batch of gradients is computed from 1024 and 256 transitions for Maze Navigation and Kitchen Manipulation experiment, respectively. We train our models for 10000, 18000, and 16000 episodes for the Maze Navigation experiments with 10, 20, 40 meta-training tasks, respectively, and 3450 episodes for Kitchen Manipulation.
As stated in Section 4.2, we apply different regularization coefficients depending on the size of the conditioning transitions. In Maze Navigation experiment, we set target KL divergence to 0.1 for batch that is conditioned on size 4 transitions and 0.4 for batch conditioned on size 8192 transitions. In Kitchen Manipulation experiment, we set target KL divergence to 0.4 for bath conditioned with a size 1024 transitions while KL coefficient for batch conditioned on size 2 transitions is fixed to 0.3.

heading: D.2.2 TARGET TASK LEARNING
text: We initialize the Q function and the auto-tuning value α with the values that learned in the skill-based meta-training phase. The policy is initialized after observing and encoding 20 episodes obtained
from the task unconditioned policy rollouts. For the target task learning phase, the target KL δ is 1 for Maze Navigation, and 2 for Kitchen Manipulation experiments. To compute a gradient step, 256 high-level transitions are sampled from a replay buffer with size 20000. Note that we used same setup for baselines that uses SPiRL fine-tuning (SPiRL and MTRL).
E IMPLEMENTATION DETAILS ON BASELINES
In this section, we describe the additional implementation details on producing the results of the baselines.

heading: E.1 SAC
text: The SAC (Haarnoja et al., 2018) baseline learns to solve a target task from scratch without leveraging the offline dataset nor the meta-training tasks.
We initialize α to 0.1 and set the target entropy toH = −dim(A). To compute a gradient step, 4096 and 1024 environment transitions are sampled from a replay buffer for Maze Navigation and Kitchen Navigation experiments, respectively.

heading: E.2 PEARL AND PEARL-FT
text: PEARL (Rakelly et al., 2019) learns from the meta-training tasks but does not use the offline dataset. Therefore, we directly train PEARL models on the meta-training tasks without the phase of learning from offline datasets. We use gradients averaged from 20 randomly sampled tasks where each task gradient is computed by batch sampled from a per-task buffer. The target entropy is set to H = −dim(A) and α is initialized to 0.1. While the method proposed in Rakelly et al. (2019) does not fine-tune on target/meta-testing tasks, we extend PEARL to be fine-tuned on target tasks for a fair comparison, called PEARL-ft. Since PEARL does not use learned skills or a skill prior, the target task learning of PEARL is simply running SAC with task-encoded initialization. Similar to the target task learning of our method, we initialize the Q function and entropy coefficient α to the value learned during the meta-training phase. Also, we initialize the policy to the task conditioned policy after observing 20 episodes of experience from the task unconditioned policy rollouts. The hyperparameters used for fine-tuning are the same as SAC.

heading: E.3 SPIRL
text: Similar to our method, we initialize the high-level policy to skill-prior while fixing low-level policy for target task learning for SPiRL. α is initialized to 0.01 and we use the same hyperparameters for the SPiRL models as our method.

heading: E.4 MULTI-TASK RL (MTRL)
text: Inspired by Distral (Teh et al., 2017), our multi-task RL baseline is designed to first learns a set of individual policies, where each of them is specialized in one task; then, a shared/multi-task policy is learned by distilling the individual polices. Since it is inefficient to learn an individual policy from scratch, we learn each individual policy using SPiRL with learned skills and a skill prior. Then, we distill the individual policies using the following objective :
max π0
ET ∼pT [∑
t
E(st,zt)∼ρπ0 [ rT (st, zt)− αDKL ( π0(z|st, e), p(z|st) )]] . (6)
We use the same setup for α as our method, where α is auto-tuned to satisfy a target KL, δ = 0.1 for Maze Navigation and δ = 0.2 for Kitchen Manipulation.
While the target task learning phase for MTRL is similar to ours, except that MTRL is not initialized with a meta-trained Q function and learned α.
(a) Meta-training 40 Tasks (b) Meta-training 20 Tasks (c) Meta-training 10 Tasks (d) Target Tasks
Figure 12: Maze Meta-training and Target Task Distributions. The green dots represent the goal locations of meta-training tasks and the red dots represent the goal locations of target tasks. The yellow cross represent the initial location of the agent.
(a) TTRAIN-TOP (b) TTARGET-TOP (c) TTARGET-BOTTOM
Figure 13: Maze Meta-training and Target Task Distributions for Meta-training Task Distribution Analysis. The green dots represent the goal locations of meta-training tasks and the red dots represent the goal locations of target tasks. The yellow cross represent the initial location of the agent.

heading: F META-TRAINING TASKS AND TARGET TASKS.
text: In this section, we present the meta-training tasks and target tasks used in the maze navigation domain and the kitchen manipulation domain.

heading: F.1 MAZE NAVIGATION
text: The meta-training tasks and target tasks are visualized in Figure 12 and Figure 13.

heading: F.2 KITCHEN MANIPULATION
text: The meta-training tasks are:
• microwave→kettle→bottom burner→slide cabinet • microwave→bottom burner→top burner→slide cabinet • microwave→top burner→light switch→hinge cabinet • kettle→bottom burner→light switch→hinge cabinet • microwave→bottom burner→hinge cabinet→top burner • kettle→top burner→light switch→slide cabinet • microwave→kettle→slide cabinet→bottom burner • kettle→light switch→slide cabinet→bottom burner • microwave→kettle→bottom burner→top burner • microwave→kettle→slide cabinet→hinge cabinet • microwave→bottom burner→slide cabinet→top burner
• kettle→bottom burner→light switch→top burner • microwave→kettle→top burner→light switch • microwave→kettle→light switch→hinge cabinet • microwave→bottom burner→light switch→slide cabinet • kettle→bottom burner→top burner→light switch • microwave→light switch→slide cabinet→hinge cabinet • microwave→bottom burner→top burner→hinge cabinet • kettle→bottom burner→slide cabinet→hinge cabinet • bottom burner→top burner→slide cabinet→light switch • microwave→kettle→light switch→slide cabinet • kettle→bottom burner→top burner→hinge cabinet • bottom burner→top burner→light switch→slide cabinet
The target tasks are:
• microwave→bottom burner→light switch→top burner • microwave→bottom burner→top burner→light switch • kettle→bottom burner→light switch→slide cabinet • microwave→kettle→top burner→hinge cabinet • kettle→bottom burner→slide cabinet→top burner • kettle→light switch→slide cabinet→hinge cabinet • kettle→bottom burner→top burner→slide cabinet • microwave→bottom burner→slide cabinet→hinge cabinet • bottom burner→top burner→slide cabinet→hinge cabinet • microwave→kettle→bottom burner→hinge cabinet

Review summaries:
This paper combines skill-based with meta RL using training tasks.  The skill-based method (SPiRL, Pertsch et al. 2020) forms a skill embedding that represents $K$-step state-action trajectories.  An action policy provides an action distribution conditioned on the currently-active skill, and a (higher-level) skill policy is trained using RL with the skill embedding as its action space.  Simultaneously a skill prior conditioned on the starting state of a given skill is trained.  Once trained, the skill embedding, the skill-conditioned action policy, and the skill prior are held fixed.  The off-policy meta RL method (PEARL, Rakelly et al. 2019) forms a task embedding that represents small sets of skills active during a given task. (This is how I understand it; the paper does not explicitly state this. Rakelly et al. do not use skills; their task embedding represents small sets of state-action-state-reward sequences.)  A skill policy provides a skill distribution conditioned on the currently-active task.  Its training is regularized using the above skill prior. Once trained, the task embedding is held fixed.  Up to here, the system is trained in a task-agnostic manner.  To learn a target task, the system further trains the skill policy, conditioned on the target task, again regularized using the skill prior.

The paper proposes a meta-RL method to learn to solve long-horizon tasks with sparse rewards efficiently. To do so it builds on previous works, mainly SPiRL, to learn a set of skills and skills prior from an offline dataset. Jointly with the skills, a high-level policy generating action in the learned latent space, is trained with meta-RL. The goal of this policy is to learn to compose learned skills to solve new tasks efficiently. Given a small set of target task trajectories, the policy can meta-learn from it and learn quickly to solve the target task. Experimentally the proposed method is shown to be more sample-efficient on two tasks, maze navigation and kitchen manipulation.

This work considers a new setting where you can leverage both 1) offline "play" data that does not contain reward or task labels, and 2) meta-training tasks in order to quickly learn new tasks at meta-test time. To solve this setting, this work proposes to learn skills from 1) by using the SPiRL algorithm, and then learn a hierarchical policy on top of the learned skills over the meta-training tasks 2) using PEARL. Empirically, the combination of SPiRL and PEARL outperforms both SPiRL and PEARL on their own on a 2d maze navigation task and on a robot kitchen task.