Skill-based Meta-Reinforcement Learning
Authors: Taewook Nam
Shao-Hua Sun
Karl Pertsch
Sung Ju Hwang
Joseph J Lim
Venue: ICLR 2022 Poster (2022)
Keywords: meta-RL
meta-reinforcement learning
skill-based meta-reinforcement learning
meta-learning
skill-based RL

Abstract: While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, long-horizon behaviors with real robot systems infeasible. To mitigate this issue, meta-reinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve long-horizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.

Published as a conference paper at ICLR 2022
SKILL-BASED META-REINFORCEMENT LEARNING
Taewook Nam', Shao-Hua Sun', Karl Pertsch², Sung Ju Hwang" ,Jseph J. Lim 1.4g Korea Advanced Institute of Science and Technology' University of Southern California', AITRICS', Naver Lab
mamnan@kaist.aaa ac. kr, (shachuas, pert sch @usc. sjhwang820kaist. ac. kr, joe. limkkaiii www.kaissaac......
ABSTRACT
While deep reinforcement learning methods have shown impressive results robot learning, their sample inefficiency makes the learning of complex, long- horizon behaviors with real robot systems infeasible. To mitigate this issue, meta- reinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offfine datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve long- horizon novel target tasks by combining the strengths of leaanneg and the usage of offline datasets, while prior approaches in RL, meta-RL- and multi task RI. require substantially more environment interactions to solve the tasks.
INTRODUCTION
In recent years, deep reinforcement learning methods have achieved impressive results in robot learning (Gu et al., 2017; Andrychowicz et al., 2020; Kalashnikov et al., 2021). Yet, existing approaches are sample inefficient, thus rendering the learning of complex behaviors through trial and error learning infeasible, especially real robot systems. In contrast, humans are capable of effectively learning a variety of complex skills in only a few trials. This can be greatly attributed to our ability to learn how learn new tasks quickly by efficiently utilizing previously acquired skills.
Can machines likewise learn to how to learn by efficiently utilizing learned skills like humans? Meta-reinforremmet learning (meta-RL) holds the promise of allowing RL agents to acquire novel tasks with improved efficiency by learning to learn from distribution of tasks (Finn et al., 2017; Rakelly et al., 2019). In spite of recent advances in the field, most existing meta-RL algorithms are restricted to short-horizon, bense-reward tasks. To facilitate efficient learning on norghoorizon, sparse-reward tasks, recent works aim leverage experience from prior tasks in the form of offline datasets without additional reward and task annotations (Lynch et al., 2020; Pertsch et al., 2020; Chebotar et al., 2021). While these methods can solve complex tasks with substantially improved sample efficiency over methods learning from scratch, millions of interactions with environments still required to acquire long-horizon horizon skills.
I Work done while at USC
1 AI advisor at Naver Al Lab Project page: https: / github. 10/ 20/SiMP1.

---

Published as a conference paper at ICLR 2022
In this work, we aim take step towards com- bining the capabilities of both learning how to quickly learn prior experience in the form of unannotated of- aim to
Meta-Training Tasks
Tt
Fast Learning of New Long- Long-Horizon Task
tasks while also leveraging
55
Target
Task Agnostic Offline Data
fline data (see Figure 1). Specifically, devise a method that enables meta-learning on complex, long-horizon tasks and can solve un- seen target tasks with orders of magnitude fewer environment interactions than prior works.
Wkill-Based
26
Meta-RL
75
Figure 1: We propose method that jointly leverages (1) large offline dataset of prior experience collected across many tasks without reward or task annotations and (2) a set of meta- meta-training tasks to learn how to quickly solve unseen long-horizon tasks. Our method extracts reusable skills from the offline dataset and meta- learn policy quickly them for solving new tasks.
We propose to leverage the offline experience by extracting reusable skills short-term be- haviors that can be composed to solve unseen long-horizon tasks. We employ a hierarchical meta-learning scheme in which we meta-train high-level policy to learn how to quickly reuse the extracted skills. To efficiently explore the learned skill space during meta- teta--tainnng, the high-level policy is guided by a skill prior which is also acquired from the offline experience data.
We evaluate our method and prior approaches in deep RL, skill-based RL, meta-RL, and multi-task RL on two challenging continuous control environments: maze navigation and kitchen manipulation, which require long-horizon control and only provides sparse rewards. Experimental results show that our method can efficiently solve unseen tasks by exploiting meta-learning tasks and offline datasets, while prior approaches require substantially more samples fail to solve the tasks.
In summary, the main contributions of this paper are threefold:
To the best of our knowledge, this is the first work to combine mnttareeinfrreement learning algorithms with task-agnostic offline datasets that do not contain reward task annotations.
We propose a method that combines meta-learning with offline data by extracting learned skills and skill prior as well as meta-learning hierarchical skill policy regularized by the skill prior.
We empirically show that our method is significantly more efficient learning long- horizon sparse-reward tasks compared to prior methods in deep RL, skill-based RL, meta-RL, (Lt, and multi-task RI.
RELATED WORK
Rent--eeiforrm Learning. Meta-RL approaches (Duan et al., 2016; Wang al., 2017; Finn et al., 2017; Yu et al., 2018; Rothfuss et al., 2019; Gupta et al., 2018; Vuorio et al., 2018; Nagabandi
et al. 2019; Clavera et al., 2019; 2018; Rakelly al., 2019; Vuorio al., 2019; Yang al., 2019; Zintgraf et al., 2019; Humplik et al., 2019; Zintgraf et al., 2020; Liu et al., 2021) hold the promise of allowing learning agents quickly adapt to novel tasks by learning learn from distribution of tasks. Despite the recent advances in the field, most existing meta-RL algorithms are limited to short-horizon and dense-reward tasks. In contrast, we aim develop a method that can meta-learn to solve long-horizon tasks with sparse rewards by leveraging offline datasets.
Offline datasets. Recently, many works have investigated the usage of offline datasets for agent training. In particular, the field of offline reinforcement learning (Levine et al., 2020: Siegel et al., 2020: Kumar al., 2020; Yu et al., 2021) aims to devise methods that can perform RL fully offline from pre-collected data, without the need for environment interactions. However, these methods require target task reward annotations on the offline data for every new tasks that should be learned. These reward annotations can be challenging obtain, especially if the offline data is collected from a diverse set of prior tasks. In contrast, our method is able to leverage offline datasets without any reward annotations.
Offline Meta-RL. Another recent line of research aims to meta-learn from static, pre-collected datasets including reward annotations (Mitchell et al., 2021; Pong et al., 2021; Dorfman et al., 2021). After meta- training with the offline datasets, these works aim to quickly adapt to a new task with only small amount of data from that new task. In contrast the aforementioned offline RI. methods
2

---

Published as a conference paper at ICLR 2022
these works aim adapt wnseen tasks and assume access to only limited data from the new tasks. However, in addition to reward annotations, these approaches often require that the offline training data is split into separate datasets for each training tasks, further limiting the scalability.
Skill-based Learning. An alternative approach for leveraging offline data that does not require reward or task annotations is through the extraction of skills reusable short-horizon behaviors. Methods for skill-based learning recombine these skills for learning unseen target tasks and converge substantially faster than methods that learn from scratch (Lee et al., 2018; Hausman et al., 2018; Sharma et al., 2020; Sun, 2022). When trained from diverse datasets these approaches can extract wide repertoire of skills and learn complex, long-hooe morizhnrzzon tasks (Merel et al.. 2020; Lynch et al., 2020; Pertsch et al., 2020; Ajay et al., 2021; Chebotar et al., 2021; Pertsch et al., 2021). Yet, although they are more efficient than training from scratch, they still require a large number of environment interactions to learn new task. Our method instead combines skills extracted from offline data with meta-learning, leading to significantly improved sample efficiency.
PROBLEM FORMULATION AND PRELIMINARIES
Our approach builds on prior work for meta-learning and learning from offline datasets and aims to combine the best of both worlds. In the following we will formalize our problem setup and briefly summarize relevant prior work.
Problem Formulation. Following prior work on learning from large offline datasets (Lynch et al., 2020; Pertsch et al., 2020; 2021). we assume access to a dataset of state-action trajectories D {sz, .t.... which is collected either across a wide variety of tasks or as "play data" with no particular task in mind. We thus refer to this dataset as task-agnostic. With a large number of data collection tasks, the dataset covers a wide variety of behaviors and can be used to accelerate learning on diverse tasks. Such data can be collected at scale, e.g. through autonomous exploration (Hausman al., 2018; Sharma et al., 2020; Dasari et al., 2019), human teleoperation (Schaal et al., 2005; Gupta et al., 2019; Mandlekar et al., 2018; Lynch et al., 2020), or from previously trained agents (Fu et al., 2020; Gulcehre et al., 2020). We additionally assume access set of meta- training tasks T = (T (Tr..... ........77 TN.: h. where each task is represented as a Markov decision process (MDP) defined by a tuple (S,A, r.p3) P:7) ) of states, actions, transition probability, reward, initial state distribution, and discount factor.
Our goal is to leverage both, the offline dataset D and the meta-training tasks T, to accelerate the training of a policy π (a]ss) 3) on target task 7 which is also represented as an MDP. Crucially, we do not assume that T is part of the set of training tasks T, nor that D contains demonstrations for solving T* Thus, we aim to design an algorithm that can leverage offline data and meta-training tasks for learning how to quickly compose known skills for solving an unseen target task. Next, we will describe existing approaches that either leverage offline data or meta- meta-traininn training tasks to accelerate target task learning. Then, we describe how our approach takes advantage of the best of both worlds.
Skill-based RL. One successful approach for leveraging task-agnostic datasets for accelerating the learning of unseen tasks is though the transfer of reusable skills, i.e. short-horizon behaviors that can be composed to solve long-horizon tasks. Prior work in skill-based RL called Skill-Prior RL (SPiRL, Pertsch et al. (2020)) proposes an effective way to implement this idea. Specifically, SPiRL uses task-agnostic dataset to learns two models: (1) skill policy T ss, 2) that decodes a latent skill representation a into a sequence of executable actions and (2) a prior over latent skill variables p(2 s) which can be leveraged to guide exploration in skill space. SPiRL uses these skills for learning new tasks efficiently by training a high-level skill policy T T=2ss) that acts over the space of learned skills instead of primitive actions. The target task RL objective extends Soft Actor Critic (SAC, Haarnoja et al. (2018)), a popular off-policy RL algorithm, by guiding the high-level policy with the learned skill prior:
max Σ Beensee [r(s., 21) aDi (T(2)s))
(1)
Here DKL denotes the weighting coefficient.
divergence between the policy and skill prior, and a is a
Off-Policy Meta-RL. Rakelly et al. (2019) introduced an off-policy meta-RL algorithm called probabilistic embeddings for actor-critic RL (PEARL) that leverages set of training tasks T to
3

---

Published as a conference paper at ICLR 2022
① Skill Extraction
Skill-based Meta- Meta-Training
Target Task Learning
%
$2
$3
S4
Policy
Meta- Traininn
Policy
do
Regularize
gie c)
gielo)
F5
Pror:
Jo
g(a a)
approx
plz %
a(2 e,
.|z st
T
T
J
xta 5,2)
Transfer
xía (2
T
xía s.
J5
Figure 2: Method Overview. Our proposed skill-based meta-RL method has three phases. (1) Skill Extraction: learns reusable skills from snippets of task- task-agnostic offline data through skill extractor (yellow) and low-level skill policy (blue). Also trains prior distribution over skill embeddings (green). (2) Skill-based Meta-training: Meta-traing a high-level skill policy (red) and encoder (purple) while using pre-traine trainer low-level policy. The pre-trained skill prior is used to regularize the high-level policy during mneta-triiing and guide exploration. (3) Target Task Learning: Leverages the meta-trained hierarchical policy for quick learning of an unseen target task. After conditioning the policy by encoding few transitions c² from the target task T continue fine-tuning the high-level skill policy the target task while regularizing with the pre-trained skill prior.
enable quick learning of new tasks. Specifically, PEARL leverages the meta-training tasks for learning task encoder qeelc). This encoder takes in a small set of state-action- transitions c and produces a task embedding This embedding is used to condition the actor π [a]s, 2) and critic Q(s, a, e). In PEARL, actor, critic and task encoder are trained by jointly maximizing the obtained reward and the policy's entropy H (Haarnoja et al., 2018):
( E(o [rr(ss, [r(ls an) aHH(T(alss....... e))]
max E TTpT engg(leT))
(2)
Additionally, the task embedding output of the task encoder is regularized towards a constant prior distribution p(e).
4 APPROACH
We propose Skill-based Meta-Policy Learning (SiMPL), an algorithm for jointly leveraging offline data as well as set of meta-trainin tasks to accelerate the learning of unseen target tasks. Our method has three phases: (1) skill extraction: we extract reusable skills and a skill prior from the offline data (Section 4.1), (2) bkiled meta-training: we utilize the meta-training tasks to learn how to leverage the extracted skills and skill prior to efficiently solve new tasks (Section 4.2), (3) target task learning: we fine-tune the meta-trained policy to rapidly adapt to solve an unseen target task (Section 4.3). An illustration of the proposed method is shown in Figure
4.1 SKILL EXTRACTION
To acquire a of reusable skills from the offline dataset D, we leverage the skill extraction approach proposed in Pertsch et al. (2020). Concretely, we jointly train (1) skill encoder q(2 (g:K-1) that embeds an K K-steps trajectory randomly cropped from the sequences in D into cow-dimensional skill embedding z, and (2) low-level skill policy π (a, [S,, 2) that is trained with behavioral cloning to reproduce the action sequence do:k-1 given the skill embedding. To learn smooth skill repre- sentation, we regularize the output of the skill encoder with unit Gaussian prior distribution, and weight this regularization by a coefficient (Higgins et al., 2017):
K
max Egnny 9:z
II
log (ac)st,, DKL Su:k au:K (1.
(3)
behuvioral cloning
embedking regularization
Additionally, we follow Pertsch et al. (2020) and learn skill prior p(z(s) that captures the distribution of skills likely to be executed in given state under the training data distribution. The prior is trained to
4

---

Published as a conference paper at ICLR 2022
match the output of the skill encoder: min, DKL (19(2)su:K do:K-1)], (0..p(((ssss) p(2)so)). Here that gradient flow is stopped into the skill encoder for training the skill prior.
[.] indicates
4.2 SKILL-BASED META-TRAINDE TRAINING
We aim to learn a policy that can quickly learn to leverage the extracted skills to solve new tasks. We leverage off off-policy meta-RI. (see Section 3) to learn such a policy using our of meta-training tasks T. Similar to PEARL (Rakelly et al., 2019), we train a task-encoder that takes in a set of sampled transitions and produces task embedding e. Crucially, we leverage our learned skills by training we---eeeedinn-odddiie
policy over skills instead of primitive actions: [z]|s, e), thus equipping the policy with set of useful pre-trained behaviors and reducing the meta-training task to learning how to combine these behaviors instead of learning them from scratch. We find that this usage of offline data through learned skills is crucial for enabling meta-training on complex, long- mogghooizon tasks (see Section 5).
Prior work has shown that the efficiency of RL on learned skill spaces can be substantially improved by guiding the policy with learned skill prior (Pertsch et al., 2020; Ajay et al., 2021). Thus, instead of regularizing with maximum entropy objective as done prior work on off-policy meta-RL (Rakelly et al., 2019), we propose regularize the meta-training policy with our pre-trained skill prior, leading to the following meta-training objective:
Σ E (s (aree/aaarrrrsssss. (1r((1,)) 2)) aDka p()ees)))
max BT (eT)
(4)
where a determines the strength of the prior regularization. We automatically tune a via dual gradient descent by choosing a target divergence 7 between policy and prior (Pertsch et al., 2020).
To compute the task embedding e, we used multiple different sizes of c. We found that we can increase training stability by adjusting the strength of the prior regularization to the size of the conditioning set. Intuitively, when the high-level policy conditioned on only a few transitions, i.e. when the set is small, it has only little information about the task at hand and should thus be regularized stronger towards the task-agnostic skill prior. Conversely, when c is large, the policy likely has more information about the target task and thus should be allowed to deviate from the skill prior more to solve the task, i.c. have a weaker regularization strength.
To implement this intuition, we employ a simple approach: we define two target divergences o, and 52 and associated auto-tuned coefficients 01 and 02 with 61 < 88. We regularize the policy using regularize using the
the larger coefficient 0ı with small conditioning transition set and otherwise smaller coefficient 022. We found this technique simple yet sufficient in our experiments and leave the investigation of more sophisticated regularization approaches for future work.
4.3 TARGET TASK LEARNING
When target task is given, we aim leverage the meta-trained policy for quickly learning how to solve it. Intuitively, the policy should first explore different skill options to learn about the task at hand and then rapidly narrow its output distribution to those skills that solve the task. We implement this intuition by first collecting small set of conditioning transitions c* from the target task by exploring with the meta-trained policy. Since we have no information about the target task this stage, we explore the environment by conditioning our pre-trained policy with task embeddings sampled from the task prior p(e). Then, we encode this set of transitions into a target task embedding e* ~ q(elc") ). By conditioning our meta-trained high-level policy this encoding, we can rapidly narrow its skill distribution to skills that solve the given target task: IT (=)s,
Empirically, we find that this policy is often already able to achieve high success rates on the target task. Note that only very few interactions with the environment for collecting c* are required for learning a complex, long-horizon and unseen target task with sparse reward. This is substantially more efficient than prior approaches such as SPiRL that require orders of magnitude more target task interactions for achieving comparable performance.
5

---

Published as a conference paper at ICLR 2022
Meta-training Tasks
Target
Meta-training Tasks
botton burner
top burner
dide cabline
Pinge calliet
bette
light swisch
allce cata
cottom burmer
Target Tasks
In croward
botton bumnrr
ligt switch
top purner
bettl
light awicch
Allcw cattined
Pinge coninet
Meteiratinng Tankor
Terget Thoics
X Agere
(a) Maze Navigation
(b) Kitchen Manipulation
evaluate our proposed framework in two domains that require the learning of complex, Long-horizon behaviors from sparse rewards. These environments are substantially more complex than those typically used to evaluate meta-RL Reta-RL algorithms. (a) Maze Navigation: The agent needs to navigate for hundreds of steps reach unseen target goals and only receives binary reward upon task success. (b) Kitchen sequence of four subtasks, spanning hundreds of time steps, and only receives a sparse reward upon completion of each subtask.
Figure 3: Environments.
Manipulation: The 7DoF agent needs execute
To further improve the performance on the target task, fine-tune the conditioned policy with target task rewards while guiding its exploration with the pre-trained skill prior':
max Re (g(ler)) n
Σ
Proneeerpee [IT- 21)
DD ( ) (.),((()))
(5)
More implementation details on our method can be found in Section
EXPERIMENTS
Our experiments aim to answer the following questions: (1) Can our proposed method learn to efficiently solve long-horizon, sparse reward tasks? (2) Is it crucial to utilize offline datasets to achieve this? (3) How can we best leverage the training tasks for efficient learning of target tasks? (4) How does the training task distribution affect the target task learning?
5.1 EXPERIMENTAL SETUP
We evaluate our approach in two challenging continuous control environments: maze navigation and kitchen manipulation environment, illustrated in Figure 3. While meta-RL algorithms typically evaluated on tasks that span only a few dozen time steps and provide dense rewards (Finn al., 2017; Rothfuss al., 2019; Rakelly al., 2019; Zintgraf al., 2020), our tasks require to learn long-horizon behaviors over hundreds of time steps from sparse reward feedback and thus pose new challenge to meta-learning algorithms.
5.1.1 MAZE NAVIGATION
Environment. This 2D maze navigation domain based on the maze navigation problem in Fu et al. (2020) requires hong-horizon control with hundreds of steps for successful episode and only provides sparse reward feedback upon reaching the goal. The observation space of the agent consists its 2D position and velocity and it acts via planar, continuous velocity commands.
Offline Dataset & Meta-training / Target Tasks. Following Fu et al. (2020) we collect a task- agnostic offline dataset by randomly sampling start-goal locations in the maze and using planner to generate a trajectory that reaches from start to goal. Note that the trajectories are not annotated with any reward task labels (i.e. which start-goal location is used for producing each trajectory). To generate a set of meta-training and target tasks, we fix the agent's initial position in the center of the maze and sample 40 random goal locations for meta-training and another of 10 goals for target
'Other regularization distributions are possible during fine-tuning, e.g. the high-level policy conditioned on task prior samples p(=|s, p(e)) e or the target task embedding conditioned policy p(z)s, e*) before finetuning. Yet, found the regularization with the pre-trained task-agnostic skill prior work hest in experiments.

---

Published as a conference paper at ICLR 2022
tasks. All meta-training and target tasks use the same sparse reward formulation. More details can be found in Section G.I.
5.1.2 KITCHEN MANIPULATIOO
Environment. The FrankaKitchen environment of Gupta al. (2019) requires the agent to control 7-DoF robot arm via continuous joint velocity commands and complete a sequence of manipulation tasks like opening the microwave or turning on the stove. Successful episodes span 300-500 steps and the agent is only provided a sparse reward signal upon successful completion of a subtask.
Offline Dataset & Meta-training / Target Tasks. We leverage dataset of 600 telepeeteeeperatedd manipulation sequences of Gupta et al. (2019) for offline pre-training. In each trajectory, the robot executes a sequence of four subtasks. We then define a set of 23 meta-training tasks and 10 target tasks that in turn require the consecutive execution of four subtasks (see Figure for examples). Note that each task consists a unique combination of subtasks. More details can be found in Section G.2.
5.2 BASELINES
We compare SiMPL to prior approaches in RL, skill-based RL, meta-RL, and multi-task RL.
SAC (Haarnoja al., 2018) is state of the art deep RL algorithm. It learns solve target task from scratch without leveraging the offline dataset nor the meta-training tasks.
SPiRL (Pertsch et al., 2020) is method designed to leverage offline data through the transfer of learned skills. It acquires skills and a skill prior from the offline dataset but does not utilize the meta-training tasks. This investigates the benefits method can obtain from leveraging the meta-training tasks.
PEARL (Rakelly et al., 2019) is a state of the art off-policy meta-RL. algorithm that learns policy which can quickly adapt to unseen test tasks. learns from the meta- training tasks but does not use the offline dataset. This examines the benefits of using learned skills in meta-RL.
PEARL-ft demonstrates the performance a PEARL (Rakelly et al., 2019) model further fine-tuned on a target task using SAC (Haarnoja et al., 2018).
multi-task RL baseline which learns from the meta-training tasks by distilling individual policies specialized in each task into shared policy, similar to Distral (Teh et al., 2017). Each individual policy is trained using SPiRL by leveraging skills extracted from the offline dataset. Therefore, it utilizes both the meta-training tasks and offline dataset similar to our method. This provides a direct comparison of multi-task learning (MTRL) from the training tasks vs. meta-learning using them (ours).
Multi-task RL (MTRL)
More implementation details on the baselines can be found in Section F.
5.3 RESULTS
We present the quantitative results in Figure 4 and the qualitative results on the maze navigation domain in Figure 5. In Figure 4, SiMPL demonstrates much better sample efficiency for learning the unseen target tasks compared to all the baselines. Without leveraging the offline dataset and meta-training tasks, SAC not able to make learning progress on most of the target tasks. While PEARL is first trained on the meta-training tasks, it still achieves poor performance on the target tasks and fine-tuning (PEARL-ft) does not yield significant improvement. We believe this is because both environments provide only sparse rewards yet require the model to exhibit long- morioorizon and complex behaviors, which is known to be difficult for meta-RL methods (Mitchell et al., 2021).
On the other hand, by first extracting skills and acquiring skill prior from the offline dataset, SPiRL's performance consistently improves with more samples from the target tasks. Yet, requires significantly more environment interactions than our method to solve the target tasks since the policy is optimized using vanilla RL, which is not designed to learn to quickly learn new tasks. While the multi-task RL (MTRL) baseline first learns multi-task policy from the meta-training tasks, its sample efficiency is similar to SPiRL on target task learning, which highlights the strength of our proposed method meta-learning from the meta-trainin tasks for fast target task learning.
Compared to the baselines, our method learns the target tasks much quicker. Within only a few episodes the policy converges to solve more than 80% of the target tasks in the maze environment and
7

---

Published as a conference paper at ICLR 2022
Maze Navigation
Kitchen Manipulation
30
Completed Sublacks
0.8
Success Rate
2.5
0.0
0.4
1.3
02
1.0
20
100
200
300
400
500
20
100
150 200
Episode
Episode
SMPLL (Ours)
SPIRL
PEARL-R SAC
Figure 4: Target Task Learning Efficiency. SiMPL demonstrates better sample efficiency compared to all the basclines, verifying the efficacy of meta-learning on ong-horizon tasks by leveraging skills and skill prior extracted from an offline dataset. For both the two environments, we train each model each target task with different random seeds. SiMPL and PEARL-ft ft first collect 20 episodes environment interactions (vertical dotted line) for conditioning the meta-trained policy before fine-tuning it on target tasks.
Episode
Episode
Episode 100
Episode
Episode
Episode 100
SIMPL (Ours)
PEARL-Rt
SPIRL
MTRL
Meta-treining Tasks
X
Target Task
Agent Trajectory
Figure 5: Qualitative Results. All the methods that leverage the ofline dataset (i.e. SiMPL, SPiRL, and MTRL) effectively explore the maze in the first episode. Then, SiMPL converges with much fewer episodes compared to SPiRL and MTRL. In contrast, PEARL-ft is not able make learning progress.
two out of four subtasks in the kitchen manipulation environment. The prior-regularized fine-tuning then continues to improve performance. The rapidly increasing performance and the overall faster convergence show the benefits of leveraging meta-training tasks in addition learning from offline data: by first learning to learn how to quickly solve tasks using the extracted skills and the skill prior, our policy can efficiently solve the target tasks.
The qualitative results presented in Figure 5 show that all the methods that leverage the offline dataset (i.e. SiMPL. SPiRL, and MTRL) effectively explore the maze in the first episode. Then. SiMPL converges with much fewer episodes compared to SPiRL and MTRL, underlining the effectiveness of meta-training. contrast, PEARL-ft is not able to make learning progress, justifying the necessity of employing offline datasets for acquiring long- horizon, complex behaviors.
5.4 META-TRAINING TASK DISTRIBUTION ANALYSIS
In this section, we aim investigate the effect of the meta-training task distribution on our skill-based meta-training and target task learning phases. Specifically, examine the effect of (1) the number of tasks in the meta- trtaa-traning task distribution and (2) the alignment between a meta-training task distribution and target task distribution. We conduct experiments and analyses in the maze navigation domain. More details on task distributions can be found Section G.
Number of meta-training tasks. To investigate how the number of meta- meta-training tasks affects the performance our method, we train our method with fewer numbers meta- meta-training tasks (i.e. and 20) and evaluate it with the same set of target tasks. The quantitative results presented in Figure 6(a)
8

---

Published as a conference paper at ICLR 2022
10
1.0
Successs Rale
0.6
Success Rate
0.5
Surceess Rates
0.8
0.0
88
SIPPL 40 Tasks Tasks
04
0.4
SIMPL 10 Tasks
SIMPL MTRL
SIMPL
MTRL
MTRL
20
100
220
900
400
500
20
100
220
300
400
500
100
220
300
400
500
Episode
Episode
Episode
(a) Sparser Task Distribution
(b) TTRAIN-TOP TTARGIT
(c) TTRAIN-TOP TTARGI-BOFTOM
Figure 6: Meta-training Task Distribution Analysis. (a) With sparser meta- veta-training task distributions (i.e. fewer numbers meta- seta-training tasks). SiMPL still achieves better sample efficiency compared to SPiRL, highlighting the henefit leveraging meta-training tasks. (h) When trained on a meta-training task distribution that aligns better with the target task distribution, SiMPI. achieves improved performance. (c) When trained on a meta- teta-training task distribution that is mis-aligned with the target tasks, SiMPI. yields worse performance. For all the analyses, train each model each target task with 3 different random seeds.
suggest that even with sparser meta-training task distributions (i.e. fewer numbers of meta-training tasks), SiMPL still more sample efficient compared the best-performing baseline (i.e. SPiRL).
Meta-train / test task alignment. We aim to examine if a model trained on a meta-training task distribution that aligns better/worse with the target tasks would yield improved/deterrooatte perfor- mance. To this end, we create biased meta- veetatraiinig test task distributions: we create meta-train set by sampling goal locations from only the top 25% portion of the maze (Ttrain-Toi) To rule out the effect of the density of the task distribution, we sample 10 (i.e. 40 25%) meta-training tasks. Then, we create two target task distributions that have good and bad alignment with this meta-training distribution respectively by sampling 10 target tasks from the top 25% portion of the maze (TTaRCeT-ToP) and 10 target tasks from the bottom 25% portion of the maze TTARGET-BOrTON.
Figure 6(b) and Figure 6(c) present the target task learning efficiency for models trained with good task alignment (meta-train on TTRAINTTOP learn target tasks from TIARGET-TOP) and bad task alignment (meta-train TTRAIN-TOP. learn target tasks from TTARGET-BOTTONA respectively. The results demonstrate that SiMPL. can achieve improved performance when trained on better aligned meta- training task distribution. On the other hand, not surprisingly, SiMPL and MTRL perform slightly worse compared to SPiRL when trained with misaligned meta-training tasks (see Figure 6(c)). This expected given that SPiRL does not learn from the misaligned meta-training tasks. In summary, from Figure 6, we can conclude that meta-learning from either a diverse task distribution or a better informed task distribution can yield improved performance for our method.
6 CONCLUSION
We propose skill-based meta-RL method, dubbed SiMPL, that can meta-learn on long-horizon tasks by leveraging prior experience in the form of large offline datasets without additional reward and task annotations. Specifically, our method first learns to extracts reusable skills and skill prior from the offline data. Then, we propose meta-trains a high-level policy that leverages these skills for efficient learning of unseen target tasks. To effectively utilize learned skills, the high-level policy is regularized by the acquired prior. The experimental results challenging continuous control long- long-horizon navigation and manipulation tasks with sparse rewards demonstrate that our method outperforms the prior approaches in deep RI., shill-based RI., meta-RI., and multi-task RL. In the future, we aim to demonstrate the scalability of our method to high-DoF continuous control problems on real robotic systems show the benefits of our improved sample efficiency.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-2018RIASAAO055 KAIST-NAVER Hypercreative Al Center, and Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2019-4) 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)). The authors are grateful for the fruitful discussion with the members of USC CLVR lab.
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. International Conference on Learning Representations, 2021.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020.
John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard Turner. Memory efficient meta-learning with large images. In Neural Information Processing Systems, 2021.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.07749 2021.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Conference Robot Learning, 2018
Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt dynamic, real-world environments through meta-reinforcemment learning, In International Conference Learning Representations, 2019.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2017.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta learning of exploration. In Neural Information Processing Systems, 2021.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL?: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:161 (..6111.000 1.02779, 2016.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a multi-domain representation for few-shot classification. In European Conference on Computer Vision, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. International Conference on Machine Leurning, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,
2020. 2020.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference on Robotics and Automation, 2017.
10

---

Published as a conference paper at ICLR 2022
Caglar Gulcehre, Ziyu Wang. Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolma, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006.13888 2020.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Neural Information Processing Systems, 2018.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference Learning Representations, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv: 1905.0642. 2019.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint arXhv:2104.08112 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European Conference on Computer Vision, 2020.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Neural Information Processing Systems, 2020.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- permutation-invarian neural networks. In International
former: A framework for attention Conference on Machine Learning, 2019.
Youngwoon Lec, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph J Lim. Com- posing complex skills by learning transition policies. In International Conference on Learning Representations, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meiafrreinorremeet learning without sacrifices. In International Conference on Machine Learning. 2021.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2020.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, 2018
11

---

Published as a conference paper at ICLR 2022
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: Reusable neural controllers for vision-guided whole-body tasks. ACM Transactions Graphics, 2020.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, 2021.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl. In International Conference on Learning Representations, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, 2020.
Karl Pertsch, Youngwoon Lee. Yue Wu, and Joseph J. Lim. Demonstration-guudde reinforcement learning with learned skills. In Conference on Robot Learning. 2021.
Vitchyr H Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. arXiv preprint 1Xiv:2207703974. 2021.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International Conference on Machine Learning, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal meta-policy search. In International Conference Learning Representations, 2019.
Stefan Schaal, Jan Peters, Jun Nakanishi, and Auke Ijspeert. Learning movement primitives. In Paolo Dario and Raja Chatila (eds.), Robotics Research, 2005.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1777.06344 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference Learning Representations, 2020.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
Shao-Hua Sun. Program-Guided Framework for Interpreting and Acquiring Complex Skills with Learning Robots. PhD thesis, University of Southern California, 2022.
Yee Teh, Victor Bapst, Wojeiech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Neural Information Processing Systems, 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. International Conference on Learning Representations, 2020.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI Conference Artificial Intelligence, 2016.
Risto Vuorio. Shao-Hua Sun, Hexiang Hu, and Joseph Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv: 1eXiv:/8/2..0 1812,07 2018.
Risto Vuorio, Shao-Hun Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via awske modulation. In Neural Information Processing Systems, 2019.
12

---

Published as a conference paper at ICLR 2022
Jane X Wang. Zeb
Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement leam. In Annual Meeting of the Cognitive Science Society (CogSci), 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992.
Yuxiang Yang, Ken Caluwaerts, Atil Iscen, Jie Tan, and Chelsea Finn. Norml: No-reward meta learning. International Conference on Autonomous Agents and Multiagent Systems, 2019.
Tianhe Yu, Chelsea Finn, Annie Xic. Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. arXiv preprint arXiv:/802.00557, 2018.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.0883 2021.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes- aadetivee deep via meta meta-learning In International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
Table of Contents
A Meta-reinforcement Learning Method Ablation
14
B Learning Efficiency on Target Tasks with Few Episodes of Experience
15
Investigating Offline Data vs. Target Domain Shift
17
Extended Related Work
17
E
Implementation Details on Our Method
Model architecture
Training details
F
Implementation Details Baselines
19
F.1
SAC
19
F2
PEARL and PEARL-ft
19
F3
SPiRL
20
F4
Multi-task RL (MTRL)
20
G Meta-training Tasks and Target Tasks.
20
G.1 Maze Navigation
20
G.2 Kitchen Manipulation
20
A META META-REINFOREE REINFORCEMENT LEARNING METHOD ABLATION
In this section, we compare the learning efficiency of different meta-RI. algorithms with respect to the length of the training tasks. Specifically, we hypothesize that our approach SiMPL, which extracts temporally extended skills from offline experience, is better suited for learning long-horizon tasks than prior meta-RL algorithms. To cleanly investigate the importance of the temporally extended skills the importance of using prior experience include two additional comparisons to methods that leverage prior experience for meta-RL but via flat behavioral cloning instead through temporally extended skills:
BC+PEARL first learns behavior cloning (BC) policy through supervised learning from the offline dataset. Then, analogous to our approach SiMPI., during the meta-training phase, task encoder and meta-learned policy are meta-trained with the BC policy constrained SAC objective. For fair comparison, use the same residual policy parameterization as described in Section E.1.
BC+MAML follows the same learning procedure described above, but MAML (Finn et al., 2017) for meta-training instead of PEARL. We follow the original learning objective in Finn al. (2017) (i.e. using REINFORCE (Williams, 1992) for task adaptation, and using TRPO (Schulman et al., 2017) for meta-policy optimization).
We compare these methods well as the standard meta-RI. approach PEARL (Rakelly et al., 2019) on three meta-training tasks distributions of increasing complexity the maze navigation environment (see Figure 7): (1) short-range goals with small variance TTRAINEASY. (2) short-range goals with larger variance TTRAIN-MEDIUM, and (3) long-range goals with large variance TTRAIN-HADD which we used in our original maze experiments. By increasing variance and length of the tasks in each task distribution, we can investigate the learning capability of the meta-RL. RL algorithms.
We present the quantitative results in Figure 8 and the corresponding qualitative analysis in Figure 9. On the simplest task distribution we find that all approaches can learn to solve the tasks efficiently, except for BC+MAMI. While the latter also learns to solve the task eventually (see performance
14

---

Published as a conference paper at ICLR 2022
(a) TTRAIN-ASY
(b) TTRAIN-MEDUM MEDIUM
(c) TTRAI-HARD
Figure 7: Task Distributions for Task Length Ablation. We propose three meta-training task distributions of increasing difficulty to compare different meta-RL. algorithms: TIRAIN EASy uses short-horizon tasks with adjacent goal locations, making exploration easier during meta- meta-training. TIRAN MEdIUM similar task horizon but increases the goal position variance, TARDN-HARD contains long-horizon tasks with high variance in goal position thus is the hardest the lested task distributions.
BC+PEARL
1.
PEARL
PEARL
Sucoess Rate
Successs Rate
Successs Rate
MW A mm
EC+PEARL
0.4
0.4
0.4
SIMPL
BC+MAML
6.0
0.0
6.
20
300
820
500
1200
1500
20
500
1500
2000
3000
4530
20
2000
4000
0020
8000
10000
Episode
Episode
Episode
TTRAI-FASY
(b) TTEAIN-MERU
(c) TRAIN-HARD
Figure 8: Meta-Trann Ceta-Training Performance for Task Length Ablation. We find that meta-learning ap- proaches can solve the simplest task distribution, but using prior experience BC+PEARL. and SiMPL. helps for the more challenging distributions (b) and (c). We find that only our approach, which uses the prior data by extracting temporally extended skills, is able to learn the challenging long- mor-horizon tasks efficiently.
upon convergence as dashed orange line in Figure 8(a)) it uses on-policy meta-RL and thus requires substantially more environment interactions during meta-training. We thus only consider the more sample efficient BC+PEARL off- policy meta-RL method in the remaining comparisons.
On the more complex task distributions TTRAIN-MEDIL and TtrAin-HaK we find that using prior data for meta-learning generally beneficial: both BC+PEARL. and SiMPI. learn more efficiently on the task distribution of medium difficulty TTRAIN MEDIUM, as shown in Figure 8(b), since the policy pre-trained from offline data allows for more efficient exploration during meta-training. Importantly, on the hardest task distribution TTRAINHARD. long-horizon tasks, we find that only SiMPL is able to effectively learn, highlighting the importance of leveraging the offline data via temporally extended skills instead of flat behavioral cloning. This supports our intuition that the abstraction provided by skills particularly beneficial for meta-learning on long-horize horizon tasks.
shown in Figure 8(c), which consists exclusively of
B LEARNING EFFICIENCY ON TARGET TASKS WITH FEW EPISODES OF EXPERIENCE
In this section, we examine the data efficiency of the compared methods on the target tasks, specifically when provided with only afew (<20) episodes of online interaction with an unseen target task. Being able to learn Rakelly et al., 2019). We hypothesize that our skill-based meta-RL RL algorithm SiMPL can learn similarly fast, even on long-horizon, sparse-reward tasks.
tasks this quickly is major strength of meta-RL RL approaches (Finn et al., 2017;
15

---

Published as a conference paper at ICLR 2022
(a) PEARL on TTRAIN-EASY
(b) BC+PEARL on TTRAINEASY
(c) SiMPL on TTRAN-EASY
(d) PEARL on TTRAIN-MUDUU
(e) BC+PEARL on TTRAIN-MUUUM
(f) SiMPL on TERAIN-MEDUU
(g) PEARL on TTRAIN-HARD
(h) BC+PEARL on TTRAN-HARD
(i) SiMPL on TTRAIS-HARD
Figure 9: Qualitative Result of Meta-reinforcement Learning Method Ablation. Top. All methods can learn to solve short-horizon tasks TIKAN-EASY Middle. On medium--oriioon tasks TIRANNMEBUM. Memux PEARL struggles at exploring further, while BC+PEARL. exhibits more consistent exploration yet still fails to solve some of the tasks. SiMPL can explore well and solve all the tasks. Bottom. On long-horizon tasks TTRAN- HARD. single task the left. BC+PEARL explores slightly
PEARL falls into local minimum, focusing only better and can solve more tasks, SiMPL can effectively learn all the tasks.
In our original evaluations in Section 5, we used 20 episodes of initial exploration to condition our meta-trained policy. In Figure 10, we instead compare performance of different approaches when only provided with very few episodes of online interactions. We find that SiMPL learns to solve the unseen tasks substantially faster than all alternative approaches. On the kitchen manipulation tasks our approach learns to almost solve two out of four subtasks within time span equivalent to only few minutes of real-robot execution time. In contrast, prior meta-RL RL methods struggle making progress at all on such long-horizon tasks, showing the benefit combining meta-RA RL with prior offline experience.
16

---

Published as a conference paper at ICLR 2022
Maze Navigation
Kitchen Manipulation
0.5
Completed Subtasks
2.0
Success Rate
0.4
1.5
0.3
0.2
1.0
0.1
0.5
0.0
Episode
Episode
SIMPL (Ours)
SPIRL
MTRL
PEARL
SAC
Figure 10: Performance with few episodes of target task interaction. We find that our skill- killl-baedd meta-RL. RL approach SiMPL is able leamn complex, long-horize tasks within few episodes of online interaction with new task while prior meta-RL approaches and non- non-meta-lle meta-learnng baselines require many more interactions or fail to learn the task altogether.
C INVESTIGATING OFFLINE DATA VS. TARGET DOMAIN SHIFT
To provide more insights comparing SiMPL and SPiRL (Pertsch et al., 2020), we evaluate SiMPL the maze navigation task setup proposed Pertsch et al. (2020). This tests whether our approach scale image-basse based observations: Pertsch al. (2020) use 32 32px observations centered around the agent. Even more importantly, it allows us to investigate the robustmess of the approach to the domain shifts between the offline pre-training data and the target task: offline dataset from Pertsch et al. (2020) which was collected on randomly sampled 20 x 20 maze layouts and test tasks in the unseen, randomly sampled 40 x 40 test maze layout from Pertsch et al. (2020). We visualize the meta-training task distribution in Figure 1 (a) and the target task distribution in Figure lb
the maze navigation
We compare the performance of our method to the best-performing baseline, SPiRL (Pertsch et al., 2020), in Figure (c). Similar to the result presented Figure 4, SiMPL can learn the target task faster by combining skills learned from the offline dataset with efficient meta-training. This shows that our approach can scale to image-based inputs and robust to substantial domain shifts between the offline pre- training data and the target tasks.
Note that the above results are obtained by comparing our proposed method and SPiRL with the exact same setup used in the SPiRL paper (Pertsch al., 2020). Specifically, we used the same initial position the agent as well as sampled the tasks of comparable complexity to the ones used in the SPiRL paper for our evaluation (please their evaluation). While the used test tasks do not fully cover the entire maze, they are already considerably long-how morioooi requiring average 710 steps until completion while only providing sparse goal-reaching rewards.
Figure 13 in the SPiRL paper for tasks used in
To further explore the performance of our proposed method and SPiRL, we have experimented with learning from goals sampled across the entire maze. Yet, SPiRL cannot learn such target tasks and our proposed method consequently does not converge well the meta-training tasks. This highlights the limitation of skill-based RL methods and can potentially be addressed by learning a more expressive skill prior, e.g. using flow models (Dinh et al., 2017), but this is outside the scope of this work.
D EXTENDED RELATED WORK
We present an extended discussion of the related work in this section.
Pre-training in Meta-learning. Leveraging pre-trained models for improving meta-learning meth- ods has been explored in Bronskill et al. (2021); Dvorik et al. (2020); Kolesnikov et al. (2020); Triantafillou et al. (2020) with focus few-shot image classification. One can also view proposed framework as reneoreeifor learning method with a pre- pre-training phase. Specifically, in the pre-training phase, we propose to first extract reusable skills and skill prior from offline
17

---

Published as a conference paper at ICLR 2022
Image-based Maze Navigation
SPIRL
Success Rale
0.8
0.0
200
400
620
1900
1000
Episode
(a) TTRAINIMAGH-BASID
(b) TAARGIF--NAAGI-BARDD
(c) Target Task Learning Efficiency
Figure 11: Image-Basee Maze Navigation with Distribution Shift. (a-b): Meta-training and target task distributions. The green dots represent the goal locations of meta-training tasks and the red dots represent the goal locations target tasks. The yellow cross represent the initial location the agent, which is equivalent one in Pertsch al. (2020). (c): Performance on the target task. Our approach SiMPL can leverage skills learned from offline data for efficient meta-RL on the maze navigation task and is robust to the domain shift between offline data environments and the target environment.
datasets without reward or task annotations self-supervised fashion. Then, our proposed method meta-learns from of meta-training tasks, which significantly accelerates learning on unseen target tasks.
E IMPLEMENTATION DETAILS OUR METHOD
In this section, we describe the additional implementation details our proposed method. The details on model architecture is presented in Section E. followed by the training detailed described in Section E.2.
E. MODEL ARCHITECTURE
We describe the details on our model architecture in this section.
E.1.1
SKILL PRIOR
We followed architecture and learning procedure of Pertsch et al. (2020) for learning a low-level skill policy and skill prior. Please refer Pertsch al. (2020) for more details the architectures for learning skills and skill priors from offline datasets.
E.1.2 TASK ENCODER
Following Rakelly et al. (2019), task encoder is permutation invarient neural network. Specifi- cally, we adopt Set Transformer (Lee et al., 2019) that consists of layers [2 x ISAB32 - PMA, - 3 x MLP] for expressive and efficient set encoding. All the hidden layers are 128-dimensional and all attention layers have 4 attention heads. The encoder takes set of high-level transitions as input, where each transition is vector concatenation of high-level transition tuple. The output of the encoder (Je, which are the parameters of Gaussian task posterior p(e|c) = =W(e;; (ee a.). We varied task vector dimension dim(e) depends task distribution complexity. dim(e) 10 for Kitchen Manipulation, dim(e) = 6 for Maze Navigation with 40 meta-tres meta-training tasks, and dim(e) = 5. otherwise.
1.1 POLICY
We parameterize our policy with neural network. We employed 4-layer MLPs with 256 hidden units for Maze Navigation, and 6-layer MLPs with 128 hidden unit for Kitchen Manipulation experiment. Instead of direct parameterization of policy, the network output is added to skill-prior to make learning more stable. Specifically, the policy network takes concatenation of (s, e) input, and then
18

---

Published as a conference paper at ICLR 2022
outputs residual parameters log. 0.) to skill-prior distribution p(z]s) N(2l op). Resulting distribution by this residual parameterization is (((ss) = (2lppp + Ms: exp(log op + log lz)g
E.1.4 CRITIC
The critic network takes concatenation 8, e, and skill 2 as imput and outputs an estimation of task-conditioned Q-value Q(s, e). We employ double Q networks (Van Hasselt et al., 2016) to mitigate Q-value overestimation. The architecture of critic follows the policy.
E.2 TRAINING DETAILS
For all the network updates, we used Adam optimizer (Kingma & Ba, 2015) with a learning rate of
3e 4, 81 = 0.9, and B2 0.999. We describe the training details of the skill-based meta-training phase in Section E.2.1 and the target task learning phase Section E.2.2.
E.2.1 SKILLLBASED META-TRAINING
Our aatt-training procedure similar to the procedure adopted in (Rakelly al., 2019). Encoder and critics networks are updated to minimize MSE between Q-value prediction and target Q value. Policy network is updated to optimize Equation without updating the encoder network. All network are updated with the average gradients of 20 randomly sampled tasks. Each batch of gradients is computed from 1024 and 256 transitions for Maze Navigation and Kitchen Manipulation experiment, respectively. We train our models for 10000, 18000, and 16000 episodes for the Maze Navigation experiments with 10, 20, 40 meta-training tasks, respectively, and 3450 episodes for Kitchen Manipulation.
As stated in Section 4.2, we apply different regularization coefficients depending on the size of the conditioning transitions. In Maze Navigation experiment, batch that is conditioned on size transitions and 0.4 for batch conditioned on size 8192 transitions. In Kitchen Manipulation experiment, we target KL divergence 0.4 for batch conditioned with size 1024 transitions while KL coefficient for batch conditioned on size 2 transitions is fixed to 0.3.
target KI. divergence 0.1 for
E.2.2 TARGET TASK LEARNING
We initialize the Q function and the auto-tuning value a with the values that learned in the skill-based meta-training phase. The policy is initialized after observing and encoding 20 episodes obtained from the task unconditioned policy rollouts. For the target task learning phase, the target KI. is 1 for Maze Navigation, and 2 for Kitchen Manipulation experiments. To compute gradient step, 256 high-level transitions are sampled from replay buffer with size 20000. Note that we used same setup for baselines that uses SPiRL fine-tuning (SPiRL. and MTRL).
IMPLEMENTATION DETAILS ON BASELINES
In this section, we describe the additional implementation details on producing the results of the baselines.
F.1 SAC
The SAC (Haarnoja et al., 2018) baseline learns to solve a target task from scratch without leveraging the offline dataset nor the meta-training tasks.
We initialize a to 0.1 and set the target entropy to H = - dim( To compute gradient step, 4096 and 1024 environment transitions are sampled from replay buffer for Maze Navigation and Kitchen Navigation experiments, respectively.
F.2 PEARL AND PEARL-FT
PEARL (Rakelly et al., 2019) learns from the meta-training tasks but does not use the offline dataset. Therefore, directly train PEARL. models the meta-training tasks without the phase of learning
19

---

Published as a conference paper at ICLR 2022
from offline datasets. We gradients averaged from 20 randomly sampled tasks where each task gradient is computed by batch sampled from per-task buffer. The target entropy is set to H. = --dimm (A) and is initialized to 0.1.
While the method proposed in Rakelly et al. (2019) does not fine-tune on target/meta-testingg tasks, we extend PEARL be fine-tuned target tasks for fair comparison, called PEARL-ft. Since PEARL does not use learned skills or a skill prior, the target task learning of PEARL is simply running SAC with lask-encoded initialization. Similar to the target task learning of method, initialize the Q function and entropy coefficient a to the value learned during the meta-training phase. Also, we initialize the policy to the task conditioned policy after observing episodes of experience from the task unconditioned policy rollouts. The hyperparameters used for fine-tuning are the same as SAC.
F.3 SPIRL
Similar to our method, we initialize the high-level policy to skill-prior while fixing low-level policy for target task learning for SPiRL. is initialized to 0.01 and use the same hyperparameters for the SPiRL models as our method.
F.4 MULTI-TASK RL (MTRL)
Inspired by Distral (Teh et al., 2017), our multi-task RL baseline is designed to first learns set of individual policies, where each of them is specialized in one task; then, a shared/multi-taz policy is learned by distilling the individual polices. Since it is inefficient to learn an individual policy from scratch, we learn each individual policy using SPiRL with learned skills and skill prior. Then, distill the individual policies using the following objective:
max E T~pp TO
Σ Banssson (rr 21) a (ma(z))s,,))) e))
(6)
We use the same setup for our method, where is auto-tuned to satisfy target KL, 5 = 0.1 for Maze Navigation and = 0.2 for Kitchen Manipulation.
While the target task learning phase for MTRL is similar ours, except that MTRL is not initialized with a meta trained Q function and learned a.
G META- TASKS AND TARGET TASKS.
In this section, we present the meta-training tasks and target tasks used in the maze navigation domain and the kitchen manipulation domain.
G.1
MAZE NAVIGATION
The meta-training tasks and target tasks are visualized in Figure 12 and Figure 13.
G.2
KITCHEN MANIPULATION
The meta-training tasks are:
m -rr.aaae-eeeeeleebottttt burner-islide cabinet
mi crcroaaee-bbttom burner-ttop burner-sslide cabinet
mi crooavvvetto burner-tligt switch-thinge cabinet
kettle-bbottom burner-tlight switch-thinge cabinet
microwave-bottoon burner-thinge cabinet-rtop burner
kettle-*top burner-tlight switch-sslide cabinet
microwae--kettle-aldde cabinet- burner
kettle-tlight switch-sslide
burner
20

---

Published as a conference paper at ICLR 2022
(a) Mcta-training 40 Tasks (b) Meta-training 20 Tasks (c) Meta-training 10 Tasks
(d) Target Tasks
Figure 12: Maze Meta-training and Target Task Distributions. The green dots represent the goal locations of meta-training tasks and the red dots represent the goal locations of target tasks. The yellow cross represent the initial location of the agent.
...
(a) TTRAI--TP
(b) TTARGEN-TOP
(c) TrARam-RarTON
Figure 13: Maze Meta-training and Target Task Distributions for Meta-training Task Distribution Anal- ysis. The green dots represent the goal locations of meta- meta-training tasks and the red dots represent the goal locations of target tasks. The yellow cross represent the initial location of the agent.
microvave-katttle--botto burner-ttop burner
mi cllcvave-keettl--ssliee cabinet-thing cabinet
mi cicrowaee-bottom burrer-tslide cabinet-ttop burner
bacpll--bbtttommmbrnreeedeee switch-ttop burner
microwave-kettll-ttoo burner-tlihtt switch
mi milcroaave--eettle--llgt switch-hhine cabinet.
mi crooaae- burrer-+light switch-sslide cabinet
kettle-Abottom burner-ttop tto--rrrttombburree----oo burner-tlight switch
microwave-tlight switch->slide cabinet-hhing cabinet
microwave-botttom burner-ttop burner-thinge cabinet
kettle-tbottom burner-tslide cabinet-thing cabinet
bottom burner-ttop burner-tslide cabinet-tlight switch
microvave-ketttle-lioht switch-pslid cabinet
kettle-bbottom burner-ttop burner-thing cabinet
bottom burner-top burner-light switch-vslide cabinet
The target tasks are:
microwave-boottm burner-tlight switch-stop burner
mi crowave burmer-ttop burner-Plight switch
kettle-tbottom burner-+light switch-sslide cabinet
mi ti burner-thinge cabinet
kettle- burner-4slide cabinet-ttop burner
21

---

Published as a conference paper at ICLR 2022
kettle-+light switch->slide cabinet-hinne cabinet
kettle-bbottom burner-ttop burner-tslide cabinet
microwave-bottom burner-sslide cabinet-thinge cabinet
bottom burner-ttop burner-tslide cabinet-hhing cabinet
-ictoowwee-ktttle--otttoom burner-thinge cabinet
22


Reviews:
[{"review_id": "tjJCIHBQczi", "reviewer": "Reviewer_AfGF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nThe paper is well written and the method is clearly explained. To my understanding it is a clean extension of SPiRL where the main point is to formulate the high level policy learning as a meta-RL problem. \n\n* The meta-RL formulation allows to exploit available target task trajectories at test time efficiently. Indeed as shown in the experiments of Figure 4, when using 20 episodes to warm up the high-level policy, the initial success rate of SiMPL is ~2x higher than other methods after 20 rollouts on the target task environment.\n\n* The meta-training task distribution analysis shown in Figure 6 is also an interesting experiment. Figure 6-(b,c) show the bias learned from the meta-trained dataset and the impact it has on the proposed method. This experiment is insightful and shows the advantage of the method if the train-test tasks are well aligned.\n\n## Weaknesses\nThe method is close to SPiRL, e.g. it reuses the skill extraction from SPiRL and the main novelty comes from the skill-based meta-training part. It is unclear to me what is the real gain coming from the meta-RL method given the current experimental results which is either too restrictive or a bit unfair.\n\n* The method seems to be tested in maze environments more restrictive than SPiRL, e.g. the train-test maze is the same, whereas SPiRL train on a various set of small mazes and test on larger mazes (c.f. Figure 3 left). Is the method trained on a fixed, pre-defined maze or does it work on a large set of mazes ? Could you evaluate the method on the same environments as SPiRL and compare to it ?\n\n* The proposed method use extra annotations at test time which are target trajectories. These trajectories are not available to the strongest baseline SPiRL, so I wonder if the comparisons of Figure 4 are fair. A fairer experiment would be to also provide these extra annotations (target trajectories) to SPiRL. For example, you could add the available target trajectories to the replay buffer of SAC before running SPiRL. This way you could measure SPiRL vs SiMPL in a fair setup where both have access to the same amount of information.\n\n* Overall the proposed method provides a good warm-start but eventually SiMPL and SPiRL methods seem to be converging to the same success rate after a limited number of steps. SPiRL seems already quite sample-efficient and the gains coming from SiMPL are a bit marginal after 200 episodes.", "summary_of_the_paper": "The paper proposes a meta-RL method to learn to solve long-horizon tasks with sparse rewards efficiently. To do so it builds on previous works, mainly SPiRL, to learn a set of skills and skills prior from an offline dataset. Jointly with the skills, a high-level policy generating action in the learned latent space, is trained with meta-RL. The goal of this policy is to learn to compose learned skills to solve new tasks efficiently. Given a small set of target task trajectories, the policy can meta-learn from it and learn quickly to solve the target task. Experimentally the proposed method is shown to be more sample-efficient on two tasks, maze navigation and kitchen manipulation.", "summary_of_the_review": "The paper is well-written and the method is clearly explained. The proposed method builds on SPiRL and proposed a meta-RL learning of the high-level policy. It shows reasonable improvements when the number of samples is limited (maze and kitchen tasks) and also provides insightful experiments (meta-training task distribution analysis).\nHowever the performances are quikcly matched by concurrent baselines after not that many steps and the comparison to SPiRL seems unfair in the current setup. Mainly the set of mazes solved seem more restrictive than SPiRL, is it a restriction of the method? Also the extra annotations used by meta-RL during evaluation (target trajectories), is not made available to the strongest baseline the authors compare to, SPiRL. The experimental section could be strengthen by evaluating SiMPL against SPiRL where the extra-data provided to SiMPL is also provided to SPIRL, and by evaluating on a richer set of environments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bzmxh_3R17K", "reviewer": "Reviewer_6qBE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this work is its strong experimental results in the new setting it proposes.\n- The proposed algorithm improves substantially over reasonably chosen baselines.\n- Additionally, this work includes two relatively informative ablations which illustrate how the proposed approach performs as a function of number of meta-training tasks, and as a function of the alignment between the meta-training tasks and the meta-testing tasks.\n\nOverall, there are no serious flaws with this work.\n\nAt the same time, the main contribution of this work is just a straightforward combination of two existing algorithms, SPiRL and PEARL. Considering and proposing the new setting of combining offline play data with the meta-RL setting is interesting, but feels a little incremental.\n\nAdditional minor comments:\n- The notation in this work is difficult to read in places. For example, $z_T$ is used to denote the task embedding vector for PEARL, while $z_s$ is used to denote the skill variables. These are completely separate concepts, but the notation makes them seem related. Similarly, $\\mathcal{T}$ is used to denote a set of tasks, which looks very similar to the trajectory $\\tau$, even though they are completely different. I would recommend using more visually distinct notation.\n- The PEARL-ft baseline seems somewhat strange to me. PEARL already prescribes the meta-test time behavior -- why should we change that to SAC instead? PEARL also seems to perform surprisingly poorly on its own, although it appears that this might just be due to the horizon length and reward sparsity. Approximately how many timesteps is each of these tasks?\n- This work is missing citations to some of the foundational meta-RL works, such as [1, 2], as well as meta-RL works that do consider sparse reward tasks, such as [3, 4].\n\n[1]: RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Duan et al., 2016.\n\n[2]: Learning to reinforcement learn. Wang et al., 2016.\n\n[3]: Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices. Liu et al., 2021.\n\n[4]: NoRML: No-Reward Meta Learning. Yang et al., 2020.", "summary_of_the_paper": "This work considers a new setting where you can leverage both 1) offline \"play\" data that does not contain reward or task labels, and 2) meta-training tasks in order to quickly learn new tasks at meta-test time. To solve this setting, this work proposes to learn skills from 1) by using the SPiRL algorithm, and then learn a hierarchical policy on top of the learned skills over the meta-training tasks 2) using PEARL. Empirically, the combination of SPiRL and PEARL outperforms both SPiRL and PEARL on their own on a 2d maze navigation task and on a robot kitchen task.", "summary_of_the_review": "Overall, I think this work is quite reasonable and does not have any serious issues. I find it only borderline on the accept side, due to its incremental nature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_8_i7NFeA74", "reviewer": "Reviewer_1NZY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n----------\n\n- The paper tackles the problem of learning over a long-horizon, which is an important challenge.\n- The experiments show a clear improvement over methods like SPiRL and PEARL. \n- Additional experiments on number of tasks and task misalignment are a valuable source of information.\n- The paper is well written and easy to follow.\n\nWeaknesses\n----------\n\n- Novelty and contribution. The method seems to be a variant of Rakelly et al. (2019) which uses the learnable priors from Pertsch et al. (2020). Overall, the difference with those methods is negligible. The major contribution of the paper is the use of a pre-training stage before meta-training which is a marginal contribution, since pre-training has been already used in meta-learning. I am open to change my mind on this point if the authors can provide concrete evidences of the relevance and novelty of their work.\n\n- Baselines. It is missing a comparison against standard meta-training methods. The authors state that meta-agents cannot learn long-horizon behaviours. However, I did not see any convincing evidence to backup this claim in the environments considered in the paper. A basic experiment would be to pre-train a standard backbone using offline data (e.g. via behavioural cloning), and then use this backbone to train a classical meta-learner such as MAML (Finn et al. 2017), CAVIA (Zintgraf et al. 2019), or more recent variants. The backbone could be finetuned or frozen at meta-training time. Pre-training the backbone of the meta-learner will guarantee the same exposure to the training environment resulting in a fairer comparison with the other methods.\n\n- Domain shift. The authors have explored the performance under misaligned train/test tasks (see Section 5.4, Figures 6b and 6c) but there is not much regarding misalignment between pre-train/meta-train. In Section 3 (Problem Formulation) the authors state that they do not assume that the offline data is part of the set of training tasks nor that it contains demonstrations for solving them. However, it seems to me that the offline data is collected from the same environment, enforcing a strong inductive bias on the pre-trained model. It is not clear if the method can still work under a mild/strong domain shift. In other words, it is not clear the type of constraints one should impose on the offline data in order to guarantee robust performances at test time. It would be beneficial to see an empirical comparison using different offline data to check how they affect the performance. Different sets of offline data could be produced via domain randomization over the base environment. This is an important factor to explore, since it could lead to a performance drop similarly to the one observed in the misaligned tasks experiments.\n\n- Long-horizon. One of the main claim of the authors is that the method is able to cope with long-horizon. It would be interesting to quantify the performance of the method under environments with an increasing horizon. In particular, it would be useful to know when the method starts to be beneficial compared to standard meta-learner. In the maze environment this could be achieved by gradually increasing the distance between the starting point and the reward.\n\nPost rebuttal\n----------------\n\nThe authors have provided a new set of experiments that (partially) answer my concerns and I will raise my score to 6. Overall the paper is still short in terms of novelty and the new experiments have been carried on a simplistic environment. I have provided a more detailed feedback in the comments section.\n\nReferences\n----------\n\nFinn, C., Abbeel, P., & Levine, S. (2017, July). Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (pp. 1126-1135). PMLR.\n\nPertsch, K., Lee, Y., & Lim, J. J. (2020). Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944.\n\nRakelly, K., Zhou, A., Finn, C., Levine, S., & Quillen, D. (2019, May). Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning (pp. 5331-5340). PMLR.\n\nZintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., & Whiteson, S. (2019, May). Fast context adaptation via meta-learning. In International Conference on Machine Learning (pp. 7693-7702). PMLR.", "summary_of_the_paper": "In this paper the authors propose to learn long-horizon policies by following three stages: (i) pretrain a model on offline data to acquire a set of skills, (ii) meta-train a policy to use those skills, and (iii) adapt the meta-trained policy to solve an unseen target task. A series of experiments on maze navigation and manipulation tasks is provided to evaluate the method.", "summary_of_the_review": "Given the marginal novelty and the lack of empirical evidences to backup the authors' claim I do not recommend to accept the paper. I am open to change my mind if the authors can provide a convincing rebuttal and more solid experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T2bmFVEaC7", "reviewer": "Reviewer_jZvC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work succeeds at combining the best of skill-based RL and meta-learning using task embeddings.  This combination introduces meta-learning at the skill level.  The results make a good case for the efficacy of the approach.\n\nOn the downside, the resulting system is quite a complicated beast.  While the ideas and methods are quite clear at a high level, the details are hard to follow, at least without deep familiarity with the Pertsch and Rakelly work.  Some of the notation is unclear, confusing or even wrong:\n- In skill-based RL (p. 3), both policies are called \"skill policies\".  I'd call only the higher-level policy a skill policy, and the lower-level policy a (skill-conditioned) action policy.\n- $z_s$ is called a \"latent skill representation\", while $z_T$ is called a \"task embedding\".  This (and other wording) makes two very similar concepts look different.  Call both of them \"embeddings\" to highlight the commonalities between skill-based and meta RL (and making their crucial differences stand out even better).\n- The embeddings $z_s$ and $z_T$ are distinguished by their subscripts.  These subscripts are typeset like variables and are easily confused with states and targets, respectively, that use the same respective identifiers.  However, these subscripts are not variables but labels that distinguish the two different uses of $z$.  Typeset them as text (not math italic), make them superscripts to further distinguis them from variable subscripts ($z^{\\textrm{s}}$, $z^{\\textrm{T}}$) or even write them out: $z^{\\textrm{skill}}$, $z^{\\textrm{Task}}$.\n    \n  To further complicate things, in $z_T$ in Eqn. 2, $T$ is not a label but represents an actual training task (sampled from the distribution $p_T$ of training tasks, where $T$ again is a label, and characterized by its reward function $r_T$, where $T$ is this training task).\n\n  One way to clarify this is the $z_{T, \\textrm{target}}$ notation in Sec. 4.3.  Follow this or one of my suggestions throughout.\n\n  This excessive overloading and abuse of notation makes this paper much harder to read than necessariy.\n- In Eqns. 1, 2, 4 and 5 the argument of the summation includes the regularizer; a parenthesis around its two terms is missing.\n- In Eqn. 2, the maximization should also run over the task encoder $q$ (and also over the critic, but the critic is anonymous here, so omitting it should not cause confusion).  Similarly in Eqn. 4(?).\n- Below Eqn. 2, \"skill\" should read \"task\".\n- In Eqn. 3, the plus should be a minus.\n- The skill prior is conditioned on the starting state of a skill.  This is called $s_0$ in Fig. 2 and $s_1$ at the bottom of p. 4.\n\nSome other issues that should be clarified:\n- Do I understand correctly that in the Kitchen task, the only significance of a subtask is that the overall Task's reward function is structured to emit a reward after the completion of each subtask (in order)?\n- In Fig. 4 and 6, SiMPL and PEARL first meta-train for 20 episodes and then train on the target task? Or what is the special significance of the 20th episode?\n\nAnother remark: It is bad news but unsurprising that SiMPL does not perform well if the training tasks are not representative of the target tasks. It would be highly interesting to try your method on Kitchen-like tasks where subtasks depend implicitly on other subtasks, without intermediate rewards.  In such hierarchical/combinatorial settings, there is no such immediate notion of \"representative\", and I can imagine that your method can (meta-)learn to combine skills to solve such hierarchical tasks.  Demonstrating this would be a big win.", "summary_of_the_paper": "This paper combines skill-based with meta RL using training tasks.\n\nThe skill-based method (SPiRL, Pertsch et al. 2020) forms a skill embedding that represents $K$-step state-action trajectories.  An action policy provides an action distribution conditioned on the currently-active skill, and a (higher-level) skill policy is trained using RL with the skill embedding as its action space.  Simultaneously a skill prior conditioned on the starting state of a given skill is trained.  Once trained, the skill embedding, the skill-conditioned action policy, and the skill prior are held fixed.\n\nThe off-policy meta RL method (PEARL, Rakelly et al. 2019) forms a task embedding that represents small sets of skills active during a given task. (This is how I understand it; the paper does not explicitly state this. Rakelly et al. do not use skills; their task embedding represents small sets of state-action-state-reward sequences.)  A skill policy provides a skill distribution conditioned on the currently-active task.  Its training is regularized using the above skill prior. Once trained, the task embedding is held fixed.\n\nUp to here, the system is trained in a task-agnostic manner.  To learn a target task, the system further trains the skill policy, conditioned on the target task, again regularized using the skill prior.", "summary_of_the_review": "The paper proposes a nontrivial and effective way of combining skills extracted from offline training data with meta-learning from training tasks.  Following the details is hampered by notational ambiguities and inconsistencies.\n\nPost-discussion edit: I upped my score to in response to the authors' improvements, although limited novelty remains a concern, as expressed by my fellow reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this paper lies in the novel setting that is being considered: offline data without rewards is combined with meta-training tasks to quickly adapt to new long-horizon tasks at meta-test time. Within this setting, it is shown that the combination of SPiRL and PEARL outperforms the individual algorithms. The technical contribution is limited, as now new methods are introduced. Nevertheless, the setting considered is interesting and the empirical evaluation is solid. For these reasons, I recommend acceptance.
Decision: Accept (Poster)