Demonstration-Guided Reinforcement Learning with Learned Skills
Authors: Karl Pertsch
Youngwoon Lee
Yue Wu
Joseph J Lim
Venue: CoRL2021 Poster (2021)
Keywords: Reinforcement Learning
Imitation Learning
Skill-Based Transfer Learning

Abstract: Demonstration-guided reinforcement learning (RL) is a promising approach for learning complex behaviors by leveraging both reward feedback and a set of target task demonstrations. Prior approaches for demonstration-guided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin to a human trying to imitate a completely unseen behavior by following the demonstrator's exact muscle movements. Naturally, such learning will be slow, but often new behaviors are not completely unseen: they share subtasks with behaviors we have previously learned. In this work, we aim to exploit this shared subtask structure to increase the efficiency of demonstration-guided RL. We first learn a set of reusable skills from large offline datasets of prior experience collected across many tasks. We then propose Skill-based Learning with Demonstrations (SkiLD), an algorithm for demonstration-guided RL that efficiently leverages the provided demonstrations by following the demonstrated skills instead of the primitive actions, resulting in substantial performance improvements over prior demonstration-guided RL approaches. We validate the effectiveness of our approach on long-horizon maze navigation and complex robot manipulation tasks.

Demonstration-GGuddee Reinforcement Learning with Learned Skills
Karl Pertsch," Youngwoon Lee, Yue Wu, Joseph J. Lim
University Southern California attps://clver com/skile
Abstract: Demonstratioonguuidee reinforcement learning (RL) is promising ap- proach for learning complex behaviors by leveraging both reward feedback and a set of target task demonstrations. Prior approaches for demonstration-guuided RL treat every new task as an independent learning problem and attempt to follow the provided demonstrations step-by-step, akin human trying to imitate a com- pletely unseen behavior by following the demonstrator s exact muscle movements. Naturally, such learning will be slow, but often new behaviors are not completely unseen: they share subtasks with behaviors we have previously learned. In this work, we aim exploit this shared subtask structure to increase the efficiency of demonstration-guine
RL. We first learn a set of reusable skills from large offline datasets of prior experience collected across many tasks. We then propose Skill-based Learning with Demonstrations (SkiLD), demonstration--uuidee RL algorithm that leverages the provided demonstrations by following the demon- strated skills instead of the primitive actions, resulting in substantial performance improvements over prior demonstration guided RL approaches. We validate its effectiveness on long- hong-horizon maze navigation and robot manipulation tasks.
Keywords: Reinforcement Learning, Imitation Learning. Skill-Based Transfer
Introduction
Humans are remarkably efficient at acquiring new skills from demonstrations: often single demonstration of the desired behavior and a few trials of the task are sufficient to master it [1, 2, 3]. To allow for such efficient learning, we can leverage a large number of previously learned behaviors [2, 3]. Instead of imitating precisely each of the demonstrated muscle movements, humans can extract the performed skills and use the rich repertoire of already acquired skills to efficiently reproduce the desired behavior.
Task- Aaskk-Agisticc Offline Data
Pask-Specific
Demonstrations
Policy
Environment
Demonstration-Guidee RL
We leverage large, task agnostic datasets collected across many different tasks for efficient demonstration-guidee reinforcement learning by (1) acquiring a rich motor skill repertoire from such offline data and (2) understanding and imitating the demonstrations based on the skill repertoire.
Figure 1:
Demonstrations are also commonly used in re- inforcement learning (RL) to guide exploration
and improve sample efficiency [4, 5, 6, 7, 8]. RI. ap-
However, such demonstration-guidce proaches attempt to learn tasks from scratch:
analogous to human trying imitate com-
pletely unseen behavior by following every demonstrated muscle movement, they try to imitate the primitive actions performed in the provided demonstrations. As with humans, such step-by-step imitation leads to brittle policies [9], and thus these approaches require many demonstrations and environment interactions learn new task.
"ooresponndncce to pertsch@use edu t Work done during an internship NAVER AI Lab AI Advisor NAVER Al Lab
5th Conference on Robot I Learning (CoRI. 2021), London, UK.

---

We propose to improve the efficiency of demonstration-uuiee guided RL by leveraging prior experience in the form of an offline "task- agnossicc experience dataset, collected not on one but across many tasks (see Figure 1). Given such a dataset, our approach extracts reusable skills: robust short-horizon behaviors that can be recombined to learn new tasks. Like a human imitating complex behaviors via the chaining of known skills, we can use this repertoire of skills for efficient demonstration-gul -uided RL on a new task by guiding the policy using the demonstrated skills instead of the primitive actions.
Concretely, we propose Skill-based Learning with Demonstrations (SkiLD), a demonstration-guiide RL algorithm that learns short-horizon skills from offline datasets and leverages them for following demonstrations of a new task. Across challenging navigation and robotic manipulation tasks this significantly improves the learning efficiency over prior demonstration-gun -ugiiodd RL approaches.
In summary, the contributions of our work are threefold: (1) we introduce the problem of leveraging task-agnostic offline datasets for accelerating demonstration-gul -igiidd RL unseen tasks, (2) propose SkiLD,, skill-based algorithm for efficient demonstration- guided RL. and (3) we show the effectiveness of our approach on a maze navigation and two complex robotic manipulation tasks,
2 Related Work
Imitation learning. Learning from Demonstration, also known as imitation learning [10], is common approach for learning complex behaviors by leveraging a set of demonstrations. Most prior approaches for imitation learning are either based on behavioral cloning (BC, [11]), which uses supervised learning to mimic the demonstrated actions, or inverse reinforcement learning (IRI, [12, 13]), which infers reward from the demonstrations and then trains a policy to optimize it. However, BC commonly suffers from distribution shift and struggles to learn robust policies [9], while IRL's joint optimization of reward and policy can result in unstable training.
Demonstration-guider RL. A number of prior works aim mitigate these problems by combining reinforcement learning with imitation learning. They can be categorized into three groups: (1) ap- proaches that use BC to initialize and regularize policies during RL training [6, 7], (2) approaches that place the demonstrations in the replay buffer of an off-policy RL algorithm [4, 5], and (3) approaches that augment the environment rewards with rewards extracted from the demonstrations [8, 14. 15].
While these approaches improve the efficiency of RL, they treat each task as an independent learning problem and thus require many demonstrations to learn effectively, which especially expensive since new set of demonstrations needs to be collected for every new task.
Online RL with offline datasets. As an alternative expensive task-specific demonstrations, multiple recent works have proposed to accelerate reinforcement learning by leveraging lask-agnostic experience in the form of large datasets collected across many tasks [16, 17, 18, 19, 20, 21J. In contrast to demonstrations, such task-agnostic datasets can be collected cheaply from a variety of sources like autonomous exploration [22, 23] or human tele-operation [24, 25, 26]. but will lead to slower learning than demonstrations since the data is not specific to the downstream task.
Skill-based RL. One class of approaches for leveraging such offline datasets that is particularly suited for learning long-horizon behaviors is skill-based RL [22, 27, 28, 29, 30, 31, 24, 32, 26, 33, 16]. These methods extract reusable skills from task-agnostic datasets and learn new tasks by recombining them. Yet, such approaches perform reinforcement learning over the set of extracted skills to learn the downstream task. Although being more efficient than RL over primitive actions, they still require many environment interactions to learn long-horizon tasks. In our work we combine the best of both worlds: by using large, task- aggsstic datasets and small number of task task-specific demonstrations, we accelerate the learning of long-horizon tasks while reducing the number of required demonstrations.
Approach
Our goal is to use skills extracted from task aask-agnostie prior experience data to improve the efficiency of demonstration -uidee RL on a new task. We aim to leverage a set of provided demonstrations by following the performed skills as opposed to the primitive actions. Therefore, we need a model that can (1) leverage prior data to learn rich set of skills and (2) identify the skills performed in the demonstrations in order to follow them. Next, we formally define our problem, summarize relevant prior work RL with learned skills and then describe our remmonstrationguidde RL approach.

---

Skill Extraction
Prior & Posterior Training
Downstream RL
D
D Deemo
Environment
4g
41 02 0g
Dis)
T %)
La s. a)
qutel
Regularization
Pre- Trained
Prior
Posteriar
x,,0,1s,2)
PaCel (y)
g.(2118)
Tyla, 2)
Figure 2: Our approach, SkiLD, combines task-agnostic experience and task specsfseeeiiccee demonstrations to efficiently learn target tasks in three steps: (1) extract skill representation from task-agnostic offline data, (2) learn task-agnostic skill prior from task-agnostic data and task-specific skill posterior from demonstrations, and (3) learn high-level skill policy for the target task using prior knowledge from both task-agnostic offline data and -sec-fpe demonstrations. Left: Skill embedding model with skill extractor (yellow) and closed-loop skill policy (blue). Middle: Training of skill prior (green) from lask-agnostic data and skill posterior (purple) from demonstrations. Right: Training of high-level skill policy (red) downstream task using the pre-trained skill representation and regularization via the skill prior and posterior, mediated by the demonstration discriminator D(s).
3.1 Preliminaries
Problem Formulation We assume access to two types of datasets: large task-agnostic offline dataset and a small task-specific demonstration dataset. The aask-agnostic dataset = {s,, ...... consists of trajectories of meaningful agent behaviors, but includes demonstrations of the target task. We only assume that its trajectories contain short-horizon behaviors that can be reused to solve the target task. Such data can be collected without particular task in mind using míx of sources, e.g., via human teleoperation, autonomous exploration, or through policies trained for other tasks. Since can be used to accelerate many downstream task that utilize similar short-term behaviors we call it task-agnostic, In contrast, the task-specific data is a much smaller set of demonstration trajectories Doemo {se at ...} that are specific single target task.
The downstream learning problem is formulated as a Markov decision process (MDP) defined by a tuple (S, A, T, R. p, of states, actions, transition probabilities, rewards, initial state distribution, and discount factor. We aim to learn a policy na (a(ss with parameters 0 that maximizes the discounted
sum of rewards J (0) E, DT Jr] = Ex, ET T-11 >rre]. where T' is the episode horizon.
Skill Prior RL Our goal is to extract skills from task-agnostic experience data and reuse them for demonstration-guuider RL. Prior work has investigated the reuse of learned skills for accelerating RL [16]. In this section, we will briefly summarize their proposed approach Skill Prior RL (SPiRL) and then describe how our approach improves upon in the demonstration guided RL setting.
SPiRL defines a skill as a sequence of H consecutive actions a = {ar...., H where the skill horizon H is hyperparameter. It uses the task-agnostic data to jointly learn (1) generative model of skills p( )2), that decodes latent skill embeddings into executable action sequences a, and (2) a state-conditioned prior distribution p(z[s) over skill embeddings. For learning new downstream task, SPiRL trains a high-level skill policy To (2|() whose outputs get decoded into executable actions using the pre-trained skill decoder. Crucially, the learned skill prior is used to guide the policy during downstream RL by maximizing the following divergence-regularizee RL. objective:
J(O) = Euz Σ r(s,, 2x) - a DxI.
(T (To(Z)))))) p(Z)))))))
Here, the KL-divergence term ensures that the policy remains close to the learned skill prior, guiding exploration during RI. By combining this guided exploration with temporal abstraction via the leamed skills, SPiRL substantially improves the efficiency of RL on long-horizon tasks.
3

---

3.2 Skill Representation Learning
We leverage SPiRL's skill embedding model for learning our skill representation. We follow prior work on skill-based RL [26, 19] and increase the expressiveness of the skill representation by replacing SPiRL's low-level skill decoder p(a(z) with a closed-loop skill policy Ta(a]s, 2) with parameters & that is conditioned on the current environment state. In our experiments we found this closed-loop decoder improve performance (see Section for an empirical comparison).
Figure 2 (left) summarizes our skill learning model. It consists of two parts: the skill inference network (u S0:H 1: 00.H --: with parameters and the closed-loop skill policy To(a St, z(). Note that in contrast to SPiRL the skill inference network is state-conditioned to account for the state-conditioned low-level policy. During training we randomly sample an H!-step state-action trajectory from the task-agnostic dataset and pass it to the skill inference network, which predicts the low-dimensional skill embedding 2. This skill embedding then input into the low-level policy To To(a ( St. 2) for every input state. The policy is trained to imitate the given action sequence, thereby learning to reproduce the behaviors encoded by the skill embedding 2.
The latent skill representation is optimized using variational inference, which leads to the full skill learning objective:
H-2 II
max E, 22w
log log -3( log go 80:11 Gu: -2) log p(:)) t=0
embedding regularization
behavioral cloning
We use a unit Gaussian prior p(z) and weight the embedding regularization term with factor / [34].
3.3 Demonstration-Guidee RL with Learned Skills
To leverage the learned skills for accelerating demonstration-guidee RL on a new task, we use a hierarchical policy learning scheme (see Figure 2, right): high-level policy T=(=[s) outputs latent skill embeddings that get decoded into actions using the pre-trained low-level skill policy. We freeze the weights of the skill policy during downstream training for simplicity.
Goal
[D]
demo
Start
Our goal is to leverage the task-specific demonstrations to guide learning of the high-level policy the new task. In Section 3.1, we showed how SPiRL [16] leverages a learned skill prior p(z[s) to guide exploration. However, this prior is task-agnostic, i.e., it encourages exploration of be explored, independent
Figure 3: We leverage prior experience data D and demonstration data D Demo- Our policy is guided by the task-specific skill posterior (c within the support of the demonstrations (green) and by the task task-agnostic skill prior Pa (z(s) other- wise (red). The agent also receives reward bonus for reaching states in the demonstration support.
all skills that are meaningful of which task the agent is trying to solve. Even though SPiRL's objective makes learning with a large number of skills more efficient, it encourages the policy to explore many skills that are not relevant the downstream task.
In this work, we propose to extend the skill prior guided approach and leverage target task demonstrations to addi- tionally learn a task-specific skill distribution, which we call skill posterior (;(2(s)) with parameters Ç (in contrast the skill prior it is conditioned the target task, hence "posterior"). We train this skill posterior by using the pre-trained skill inference model q (= 8n: (0:H12) 2) to extract the embeddings for the skills performed in the demonstration sequences (see Figure 2, middle):
min E ((s.a) DKL (2)s ao.. 9: (2(so)))
(3)
A naive approach for leveraging the skill posterior is to simply use it to replace the skill prior in Equation 1, i.e., to regularize the policy to stay close to the skill posterior every state. However, the trained skill posterior is only accurate within the demonstration support [Daemo.. because by definition it was only trained on demonstration sequences. Since [Daemol < [D] (see Figure 3), the skill posterior will often provide incorrect guidance in states outside the demonstrations' support.

---

Instead, we propose to use three-part objective that guides the policy to (1) follow the skill posterior within the support of the demonstrations, (2) follow the skill prior outside the demonstration support, and (3) reach states within the demonstration support. To determine whether a given state is within the support of the demonstration data we train a learned discriminator D(s) as a binary classifier using samples from the demonstration and task-agnostic datasets, respectively.
In summary, our algorithm pre-trains the following components: (1) the low-level skill policy To (als,, z). (2) the task-agnostic skill prior p(z[s), (3) the task-specific skill posterior q< (=(2s) and (4) the learned discriminator (s). Only the latter two need be re-trained for a new target task.
Once all components are pre- pre-trained, we use the discriminator's ' output to weight terms in our objective that regularize the high-level policy To m=(=s)) towards the skill prior posterior. Additionally, provide a reward bonus for reaching states which the discriminator classifies as being within the demonstration support. This results in the following term J, for SkiL.D's full RL objective:
Jr F(st; =+) ,x DxL. (To((z))))) 96 (Fxl=:)) D(se) DKI. (To (x), p(z (s)) (1 (sr)))
posterior regularization
prior regularization
with F(re; 2.) = (1 - 6) r(0, 2)) + K [log D(s,) log (1 D(s)))
(4)
discriminator reward
The weighting factor x is a hyperparameter; a and Og are either constant or tuned automatically via dual gradient descent [35]. The discriminator reward follows common formulations used in adversarial imitation learning [36, 37, 8, 38]. Our formulation combines IRL-like and BC-like objectives by using learned rewards and trying to match the demonstration's is skill distribution.
For policy optimization, we use a modified version of the SPiRI algorithm [16], which itself is based on Soft Actor- Critic [39]. Concretely, we replace the environment reward with the discriminator- augmented reward and all prior divergence terms with our new, weighted prior-posterior-divereee terms from equation 4 (for the full algorithm see appendix, Section A).
4 Experiments
In this paper, we propose to leverage a large offline experience dataset for efficient demonstration- guided RL. We aim to answer the following questions: (1) Can the use of task-agnostic prior experience improve the efficiency of demonstration--uuin
RL? (2) Does the reuse of pre-trained skills reduce the number of required target-specific demonstrations? (3) In what scenarios does the combination of prior experience and demonstrations lead to the largest efficiency gains?
4.1 Experimental Setup and Comparisons
To evaluate whether our method SkiLD can efficiently use task-agnostic data, we compare it to prior demonstration--uuide RL approaches on three complex, long-horizon tasks: 2D maze navigation task, a robotic kitchen manipulation task and a robotic office cleaning task (see Figure 4, left).
Maze Navigation. We adapt the maze navigation task from Pertsch al. [16] and increase task complexity by adding randomness to the agent's initial position. The agent needs to navigate through maze for hundreds of time steps using planar velocity commands to receive sparse binary reward upon reaching a fixed goal position. We collect 3000 task-agnostic trajectories using a motion planner that finds paths between randomly sampled start-goal pairs. For the target task we collect demonstrations for an unseen start-goal pair.
Robot Kitchen Environment. We the environment of Gupta al. [24] in which 7DOF robot arm needs to perform a sequence of four subtasks, such as opening the microwave or switching on the light, in the correct order. The agent observes a fow-dimensional state representation and receives a binary reward upon completion of each consecutive subtask. We use 603 teleoperated sequences performing various subtask combinations (from Gupta et al. [24]) as agnossii experience D and
*We found that using the pre-trained discriminator weights led to stable training. but it is possible to perform full adversarial training by finetuning ((s) with rollouts from the downstream task training. We report results for initial experiments with discriminator finetuning Section E and leave further investigation for future work.
5

---

separate set of 20 demonstrations for one particular sequence of subtasks, which we define as our target task (see Figure 4, middle).
Navigation
Robot Office Environment. A 5 DOF robot arm needs to clean an office environment by placing objects in their target bins or putting them drawer. observes the poses of its end-effector and all objects in the scene and re- ceives binary rewards for the completion of each subtask. We collect 2400 training trajectories by perturbing the objects initial positions and performing random subtasks using scripted poli- cies. We also collect 50 demonstrations for the unseen target task with unseen object locations and subtask sequence.
Succets Racc
40
0.00
0.33
6.35
8.3
6.9
Environment steps (1M)
Kitchen Envirenment
Completee ploted Subrasks
We compare our approach to multiple prior semoostrationnguudee RL approaches that rep- resent the different classes of existing algo- rithms introduced in Section SkiLD, these approaches are not designed to leverage task-agnostic prior experience: BC + RL initializes a policy with behavioral cloning of the demonstrations, then continues to apply BC loss while finetuning the policy with Soft Actor-Critic (SAC, [39]), representative of [6, 7]. GAIL + RL [8] combines rewards from the environment and adversarial imitation learning (GAIL, [13]), and optimizes the policy using PPO [40]. Demo Replay initializes the replay buffer of an SAC agent with the demonstrations and uses them with prioritized replay during up- dates, representative of [4]. We also compare our approach to RL-only methods to show the benefit of using demonstration data: SAC [39] is tftt-off-th-art model-free RL algorithm, neither uses offline experience nor demon- strations. SPiRL [16] extracts skills from task- agnostic experience and performs prior-guided RL on the target task (see Section 3.1 Finally, Skill BC+RL combines skills learned from task- agnostic data with target task demonstrations: it encodes the demonstrations with the pre-trained skill encoder and runs BC for the high-level skill policy, then finetunes on the target task using SAC. For further details on the environments, data collection, and implementation, see appendix Section B.
Environment (2M)
Office Environment
In contrast to
Completed Subtasks
2:38
0.55
1.50
steps
SKiLD (Ours)
SPIRL
BC+RL
Replay
BC+RL
GAIL+RL
SAC
Figure Left: Test environments, top to bot- tom: 2D maze navigation, robotic kitchen manip- ulation and robotic office cleaning. Right: Target task performance vs environment steps. By using vask-agnostic experience, our approach more ef- ficiently leverages the demonstrations than prior demonstration-
RL approaches across all tasks. The comparison SPiRL shows that demon- strations improve efficiency even if the agent has access large amounts of prior experience.
4.2 Demonstration-Guidee RL with Learned Skills
Maze Navigation. Prior demonnstraionnguuided RL approaches struggle on the task (see Figure right) since rewards are sparse and only five demonstrations are provided. With such small coverage, behavioral cloning of the demonstrations' primitive actions leads brittle policies which are hard to finetune. The Replay agent improves over SAC without demonstrations and partly succeeds at the task, but learning is slow. The GAIL+RL approach is able follow part of the demonstrated behavior, but fails to reach the final goal (see Figure 8 for qualitative results). SPiRL and Skill BC+RL leverage task-agnostic data to learn to occasionally solve the task, but train slowly: SPiRL's learned, task-agnostic skill prior and Skill BC+RL's uniform skill prior during SAC finetuning encourage the exploration of many iask-irrlevant skills contrast, our approach SkiLD leverages the task-
'e train SPiRL with the closed-loop policy representation from Section for fair comparison and better performance. For an empirical comparison of open and closed-loop skill representations SPIRI, see Section C. PPerformance of SPIRL differs from Pertsch al. [16] due increased task complexity, Section B.4.

---

Demonstrations
p (demoo| state)
(policy posterior)
[polliyy || prior)
Figure 5: Visualization of our approach the maze navigation task (visualization states collected by rolling out the skill prior). Left: the given demonstration trajectories; Middle left: output of the demonstration discriminator (s) (the greener, the higher the predicted probability of state to be within demonstration support, red indicates low probability). Middle right: policy divergences to the skill posterior and Right: divergence the skill prior (blue indicates small and red high divergence). The discriminator accurately infers the demonstration support, the policy successfully follows the skill posterior only within the demonstration support and the skill prior otherwise.
specific skill posterior quickly explore the relevant skills, leading to significant efficiency gains (see Figure 5 for qualitative analysis and Figure for a comparison of SkiLD vs. SPiRL exploration).
Robotic Manipulation. We show the performance comparison on the robotic manipulation tasks in Figure (right)". Both tasks are more challenging since they require precise control of a high-DOF manipulator. We find that approaches for demonstration-guiet guided RL. that do not leverage task-agnostic experience struggle to learn either of the tasks since following the demonstrations step-by-step is inefficient and prone accumulating errors. SPiRL, in contrast, able to learn meaningful skills from the offline datasets, but struggles to explore the task-relevant skills and therefore learns slowly, Worse yet, the uniform skill prior used in Skill BC+RL's SAC finetuning is even less suited for the target task and leads the policy to deviate from the BC initialization early on in training, preventing the agent from learning the task altogether (for pure BC performance, see appendix, Figure 13). Our approach, however, uses the learned skill posterior to guide the chaining of the extracted skills and thereby learns to solve the tasks efficiently, showing how SkiLDD effectively combines agnn-tgo and task-specific data for demonstration-guint RL.
4.3 Ablation Studies
In Figure (left) we test the robustness of our ap- proach to the number of demonstrations in the maze navigation task and compare to BC+RL, which found to work best across different demonstration set sizes. Both approaches bene- fit from more demonstrations, but approach able to learn with much fewer demonstrations by using prior experience. While BC+RL leams each low-level action from the demonstrations, SkiLD merely learns to recombine skills it has already mastered using the offline data, thus re- quiring less dense supervision and fewer demon- strations. We also ablate the components of our RL objective on the kitchen task Figure 6 (right). Removing the discriminator reward bonus ("no-GAle (ho-GAlL") slows convergence since the agent lacks a dense reward signal. Naively replacing the skill prior in the SPiRL objective of Equation| 1 with the learned skill posterior ("post-only") fails since the agent follows the skill posterior outside its support. Removing the skill posterior and optimizing a discriminator bonus augmented reward using SPiRL ("no-post") fails because the agent cannot efficiently explore the rich skill space. Finally, we show the efficacy of our approach in the pure imitation setting, without environment rewards, in appendix, Section
Succea Dore
SMIL (Bund)
ne-WWi
SILD (Ours) BC+AL
pest pestienly
nelpost
10
1.3
(1M)
Figure 6: Ablation studies. We test the perfor- mance of SkiLD for different sizes of the demon- stration dataset [Dáemol on the maze navigation task (left) and ablate the components of our objective on the kitchen manipulation task (right).
For qualitative robot manipulation videos, see https: http://ssiieessgglll.ee www.go.....vvieeeeoeoooooooooooe www/view/wwilll..cce.p... 1etesiill-mmm----- demo-rl. rl.
7

---

4.4 Robustness to Partial Demonstrations
Completed S.basts
Most prior approaches that aim to follow demon- strations of a target task, assume that these demonstrations show the complete execution of the task. However, we can often encounter sit- uations in which the demonstrations only show incomplete solutions, e.g. because the agent's and demonstration's initial states do not align or because only have access demonstrations for a subtask within a long-horizon task. Thus, SkiLD is designed to handle such partial demon- strations: through the discriminator weighting it relies on demonstrations only within their sup- port and falls back to following the task-agnostic skill prior otherwise. Thus it provides bust framework that seamlessly integrates task- specific and task-agnostic data sources. We test this experimentally in the kitchen environment: we train SkiLD with partial demonstrations in which we remove one of the subskills. The results in Figure 7 (left) show that "SkiLD-Partial" is able to leverage the partial demonstrations to improve efficiency over SPiRL, which does not leverage demonstrations. Expectedly, using the full demonstrations the SkiLD framework ("SkiLDDFull") leads to even higher learning efficiency.
84LD (Dus))
6PR
SPAL
Environment stops (IM)
Environment stops stops (1M)
Figure 7: Left: Robustness to partial demonstra- tions. SkiLD can leverage partial demonstrations by seamlessly integrating task-agnostic and task- specific datasets (see Section 4.4). Right: Analysis of data vs. task alignment. The benefit of using demonstrations in addition to prior experience di- minishes if the prior experience is closely aligned with the target task (solid), but gains are high when data and task are not well-aligned (dashed).
4.5 Data Alignment Analysis
We aim to analyze in what scenarios the use of demonstrations in addition to task-agnostic experience is most beneficial. In particular, we evaluate how the alignment between the distribution of observed behaviors in the task-agnostic dataset and the target behaviors influences learning efficiency. We choose two different target tasks in the kitchen environment, one with good and one with bad alignment between the behavior distributions (see Section F), and compare our method, which uses demonstrations, to SPiRL, which only relies on the task-agnostic data.
In the well-aligned case (Figure 7, right, solid lines), we find that both approaches learn the task efficiently. Since the skill prior encourages effective exploration on the downstream task, the benefit of the additional demonstrations leveraged in our method is marginal. In contrast, if task-agnostic data and downstream task are not well-aligned (Figure 7, right, dashed), SPiRL struggles learn the task since it cannot maximize task reward and minimize divergence from the mis-aligned skill prior at the same time, Our approach learns more reliably by encouraging the policy to reach demonstration-Liks states and then follow the skill posterior, which by design is well-aligned with the target task.
In summary, our analysis finds that approaches which leverage both task agno-sgi data and demon- strations, improve over methods that use either of the data sources alone across all tested tasks. We find that combining the data sources is particularly beneficial in two cases:
Diverse Task Agnostic Data. Demonstrations can focus exploration on task-relevant skills if the task-agnostic skill prior explores a too large set of skills (see Section 4.2).
Mis-Aligned Task-Agnostic Data. Demonstrations can compensate mis-alignment be- tween task ags-agnostiic data and target task by guiding exploration with the skill posterior instead of the mis-alignn aliined prior.
Conclusion
We proposed SkiLD, an approach for demonstration-guiel RL that is able to leverage aask-agnostic experience datasets and taec-fpe demonstrations accelerated learning of unseen tasks. In three challenging environments SkiLD learns new tasks more efficiently than both, prior demonstration- guided RL approaches that are unable to leverage task-agnostic data, as well as skill-based RL methods that cannot effectively incorporate demonstrations, Future work should combine task-agnostic data and demonstrations for efficient learning in the real world and investigate aggain-agnostic measures for data-task alignment to quantify the usefulness of prior experience for target tasks.

---

Acknowledgments
This research is supported by the USC Annenberg Fellowship, NAVER AI Lab, and NSF NRI- 2024768. We would like to thank the lab members of the USC CLVR lab for valuable discussions throughout this project.
References
[1] H. Bekkering, A. Wohlschlager, and M. Gattis. Imitation of gestures in children is goal-directed. The Quarterly Journal of Experimental Psychology: Section A, 53(1)1133-164, 2000.
[2] S. A. Al-Abood, K. Davids, and S. J. Bennett. Specificity of task constraints and effects of visual demonstrations and verbal instructions in directing learners' search during skill acquisition. Journal of motor behavior, 33(3):295-305, 2001.
[3] N. J. Hodges, A. M. Williams, S. J. Hayes, and G. Breslin. What is modelled during observa- tional learning? Journal of sports sciences, 25(5):531-545, 2007.
[4] M. Vecerik, T. Hester, J. Scholz, Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint arXiv: 170..08877 2017.
[5] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Oshand, et al. Deep q-learning from demonstrations. In AAAI, 2018.
[6] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
[7] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292-6299. IEFE, 2018
[8] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Robotics: Science and Systems, 2018.
[9] S. Ross, G. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction no-regret online learning. In International Conference on Artificial Intelligence and Statistics, pages 627-635, 2011.
[10] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483,,48444000 2009.
[11] D. A. Pomerleau. Alvinm: An autonomous land vehicle in neural network. In Proceedings of Neural Information Processing Systems (NeurIPS), pages 305-313, 1989.
[12] P. Abbeel and A. Ng.. Apprenticeship learning via inverse reinforcement learning. In ICML, 2004.
[13] J. Ho and S. Ermon. Generative adversarial imitation learning. NewrIPS, 2016.
[14] X. B. Peng, P. Abbeel, S. Levine, and M. van Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018
[15] J. Merel, Y. Tassa, D. TB, S. Srinivasan, J. Lemmon, Z. Wang, G. Wayne, and N. Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.0220.. 2017.
[16] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.

---

[17] N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. ICIRR, 2020.
[18] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09355 2020.
[19] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
[20] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. ICLR, 2021.
[21] A. Singh, A. Yu, J. Yang. J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. CoRL., 2020.
[22] K. Hausman, J. T. Springenberg, Z. Wang. N. Heess, and M. Riedmiller. Learning embedding space for transferable robot skills. In ICLR, 2018.
[23] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. ICLR, 2020.
[24] A. Gupta, V. Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRL, 2019.
[25] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In CoRL, 2018.
[26] C. Lynch, M. Khansari, T. Xiao, V, Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In CoRL, 2020.
[27] Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. ICLR, 2019.
[28] T. Kipf, Y. Li, H. Dai. V. Zambaldi, E. Grefenstette, P. Kohli, and P. Battaglia. Compositional imitation learning: Explaining and executing one task at a time. ICML, 2019.
[29] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: Reusable neural controllers for vision-guided whole-body tasks. ACM. Trans. Graph., 2020.
[30] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In ICLR, 2019.
[31] W. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics-aware embeddings. ICLR, 2020.
[32] Y. Lee, J. Yang, and J. J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In ICLR, 2020.
[33] K. Pertsch, O. Rybkin, J. Yang, S. Zhou, K. Derpanis, J. Lim, K. Daniilidis, and A. Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. L4DC, 2020.
[34] I. Higgins, I. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. beta-VAF, Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
[35] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: arXiv: 1812.05905 2018.
[36] C. Finn, P. Christiano, P. Abbeel, and S. Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. NeurlPS Workshop on Adversarial Training, 2016.
10

---

[37] Fu. K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In ICLR, 2018.
[38] I. Kostrikov, K. K. Agrawal, D. Dwibedi, Levine, and J. Tompson. Dissriminator-acto--riitt Addressing sample inefficiency and reward bias in adversarial imitation learning. In ICLR, 2019.
[39] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. ICML, 2018.
[40] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint 0rXiv:1707.06.. 2017.
[41] L. Liu, H. Jiang, P. He. W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. ICLR, 2020.
[42] D. Kingma and J. Ba, Adam: method for stochastic optimization. ICLR, 2015.
[43] J. Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0 2000:0000979,, 2020.


Reviews:
[{"review_id": "dmGvEDCU2Cx", "reviewer": "Reviewer_5LPL", "summary": "This paper describes two mechanisms, which when combined can provide a good imitation learning algorithm to quickly learn a skill-based policy for a novel skill. \nThere are two main components:\nA pretraining method (Skill representation Learning) for skill extraction from unstructured experience data.  This produces \\pi(a|s,z) a skill-state-conditioned policy that will be used as the low-level controller later on.  This is learned through a variational learning objective with a regularization that forces the task embedding to be consistent during a particular trajectory.\n\nA learning method train a policy purely from demonstrations (without a reward function) in a demonstration-data-efficient manner.  This method requires the training of:\nD(s) - a demonstration/non-demonstration discriminator.\nq_z(z|s), the task-based policy being learned.\nP_a(z|s) - not entirely sure what this is, perhaps a model that provides z-probability conditioned on s_t but over the whole dataset? Or perhaps over a particular action sequence.... This could warrant being clarrified.\n\nOnce these are trained, a final policy pi_theta(z|s) is trained by maximizing a discriminator reward for a given task, and making sure that the policy is following the posterior predictor (q_z) when on-distrubition relative to task data, and following p_a(z|s) when out of distribution relative to task-specific data.\n\nThe authors show that this approach can find controllers that can complete more than 3 subtasks on average after 1-2 million training steps on the environment.  Thes tasks have no reward provided by the environment, and even if there was a sparse reward the exploration problem would render them unsolvable.  \nThe authors show that given the various models above one can learn a new policy pi_theta(z|s) which can operate in the skill-space MDP to solve demonstration-defined tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found the paper pretty clear to read, straight-to-the-point, and the experiments quite convincing.  The method makes intuitive sense, and although it has a lot of moving parts each one seems to be well-justified and not overdone.  I also found the related work well put together and providing sufficient context to situate the proposed work relative to other papers.\n\nMy main comments on weaknesses would be that because there are a lot of moving parts, I had to go back and forth quite a bit, and I still have a couple elements that I'm not quite sure about (p_a(z|s) for example).  I've put specific fixes like this in the 'Issues' section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend acceptance, but please try and fix the issues mentioned in the next section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. Although overall the paper was easy to read, it would be very helpful to have a small table or some form of recap of all the different models: skill prior, skill posterier, skill inference model, policy etc. and what exactly each one learns.  I spent a lot of time going back and forth between sections to try to decipher what is going on.  Could you provide some form of recap before or around Eq. 4 to summarize all these, perhaps in some sort of table?\n\n2. Small nit but on l. 162 you introduce the skill posterior but don't define the zeta subscript, is zeta a skill?  That was my deduction but this cost me some concentration points and could be easily clarified."}}, {"review_id": "bZFmtqVwgSL", "reviewer": "Reviewer_RYNo", "summary": "The main contribution of the paper is to combine the skill-based RL with the demonstrations-based RL. The demo-based RL can suffer from the covariate shift problem. In contrast, the skill-based RL still needs to explore a relatively large options space. The idea is to combine both with a discriminator weight to avoid their disadvantages. Simply speaking, if the current state is covered by the demonstrations, the learned policy should behave like BC. If the current state is out of the demonstrations, the learned policy should behave like SPiRL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea is interesting and clearly explained in the paper. The experiments are thorough. However, in the paper, the skill is learned beforehand and fixed during the downstream training. I think that all three experiments can be considered standard RL problems with a small set of \"actions\".  These kinds of problems are difficult for the agents without considering the hierarchy of tasks.  But the experiments are only skill sequencing problems if skills are well trained. I noticed that the low-level policy described in the paper is dependent on the state. According to the authors, the one in SPiRL is, however, not dependent on the state. Is this also one reason why the proposed method outperforms SPiRL? Furthermore, compared to SPiRL, the proposed method still needs a set of demonstrations for different tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is well written. The idea is interesting. The question is whether this kind of method can be used or generalized for non-skill-sequencing problems. I think that it might be also difficult to extract skills automatically from demonstrations.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "It would be good if the authors can compare the proposed method with SPiRL with the same set of skills. They should all depend on the states. If this is already the case, the authors should clarify it in the paper. (If I missed some sentences in the paper, please just correct me.) \n\nI would suggest giving more details in the paper (but not only in supplementary materials) about the state and action definition for each experiment and what skills are learned. For example, for the maze navigation, it is not clear in the paper whether the goal position is in the state or not."}}, {"review_id": "TTIA-U_NYBe", "reviewer": "Reviewer_fuNi", "summary": "This work builds on top of SPiRL (Pertsch et al. 2020), with the insight that SPiRL encourages policies to explore many behaviors not relevant for the downstream task we actually want to solve. To address this, the authors propose a reasonable suggestion: fine-tune RL on the target task, but use the high-level skill spaces learned by SPiRL to benefit from the data efficiency and restricted search space. \n\nThe recipe is as follows: just as in SPiRL, first learn a VAE for short-horizon actions. This yields a skill posterior q(z|s,a), and a skill-conditioned action decoder pi(a|s,z) (which is later re-used as an “action decoder” for the policy). There is a small modification where the state is also passed to the “low-level policy”. One can also train the task-agnostic prior p(z|s) in this step, by predicting the inferred posteriors directly from state.\n\nNext, *repeat* this for the target demo data, learning a task-specific skill posterior q(z|s). The task-agnostic prior p(z|s) and posterior q(z|s) are used in regularizing the high-level policy pi(z|s) being trained with RL. The full RL objective adds 3 distribution matching losses on top of the existing RL reward optimization: the raw reward is using a GAIL-style discriminator reward against target task demonstrations, encouraging the high level policy to match the posterior, and encouraging the policy to also lie within the prior (because the “skill posterior will often provide incorrect guidance outside of the demonstrations’ support”). Some of the loss coefficients are tuned automatically via dual gradient descent. Learned discriminator used to distinguish whether state is within support of the target task demonstration, and is thus used to tune which loss (posterior or prior regularization) matters more. \n\nThe authors evaluate this method on 3 simulated tasks, showing that it outperforms SPiRL and some other baseline algorithms (e.g. BC+RL) by a large margin.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n1. Each of the pieces of the method (the use of high-level skills, the use of discriminator to detect OoD, the use of a task-agnostic skill prior to “catch” states outside of demonstration support, the use of a skill posterior to keep it close to the task-specific skill distribution) are reasonably motivated.\n2. The re-use of various skill encoder components to form reward constraints and “loss switching” mechanisms is interesting. \n\nWeaknesses:\n1. Although the extensions to SPiRL presented here are interesting, there are too many moving parts to this method to make it practical. Depending on how one counts the pieces of a VAE encoder/decoder and GAN as one or several models, there are at least 4 models that need to be trained for this to work, each with their own set of hyperparameters. Furthermore, certain pieces are re-used in different places (e.g. the state discriminator is used for both reward shaping and switching between the penalizing the skill prior or skill posterior), which sounds like it makes it hard to debug if something is not working.\n2. No real robot results. Tuning on-policy RL algorithms on real robots is challenging, and often SAC alone has enough hyperparameters to make things quite difficult and collect sufficient data. A demonstration that this method can be plausibly scaled to real robots is what I’d like to see.\n3. A discriminator is used to switch between penalizing posterior / prior, but a discriminator in and of itself does not guarantee out-of-distribution errors. In fact, early on in training the state discriminator model may be misspecified.\n4. It is not easy to assess the effectiveness or completeness of the latent skills until one trains the policy itself. \n5. The paper’s admission that the task-specific skill posterior is not always accurate outside of the support of the task-specific demos seems like a major flaw in the method, requiring the user to bolt on several additional losses like the skill prior and discriminator loss. Going through this trouble of learning yet another couple models doesn’t imbue the robot with new capabilities beyond being able to solve the target task better, so it begs the question of whether one model can simply be tuned to solve the task well (for example, simply using PPO in simulation).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do not spot major theoretical flaws with each of the components being proposed, but the method feels like 3 bolted-on learned components needed to get a specific idea (\"fine-tune SPiRL with RL\") to work well. It feels like to me that there can be a simplification of the method that makes it more practical for real robots. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The fact that SPiRL and the proposed method are the only methods that do okay on the Kitchen and Office environments suggest to me that the baselines have not been tuned very well. For instance, it’s surprising that SAC and GAIL + RL do so poorly on the Maze Navigation environments.. Relay Policy Learning (Gupta ‘19 CORL) achieved high success rates on the kitchen environment using what essentially amounts to behavior cloning, which suggests that there is something sub-optimal in the baseline implementations of this method. I would like to see some convincing analysis as to why the baselines do so abysmally, given that other papers have reported decent goal-conditioned behavior cloning baselines.\n\nI would like the authors to address the problem setting and data requirements in which this method is an ideal choice over other learning approaches, in particular ones that are simpler like GCBC or PPO brute-forced at a sim environment. For instance, is this done in the context of solving in sim and then transferring to real? How much target task data is available? What is a passable amount of overlap between the task-agnostic dataset distribution and the target task distribution? \n"}}]

Author rebuttal:

Meta-review: === comments before the discussion ===

The paper presents a demonstration-guided RL method, that leverages pre-trained skills. 
In the initial review, the scores are split. 
I recommend the authors to carefully address the concerns raised by reviewers. For example, following points seems to be important points:
- Reviewers 5LPL and fuNi shared the same impression that there are several moving parts in the proposed method. 
- The reviewer fuNi raised concerns related to baselines in experiments, which is important for evaluating the method. 
- The reviewer fuNi also raised concerns related to the out-of-distribution errors.

=== comments after the discussion ===

The author response clarified unclear points in the paper. Although there are several moving parts in the proposed method, the empirical results show the good performance in the kitchen task, and the pros outweigh the cons of the paper. Thus, the area chair recommends the acceptance of the paper.
Decision: Accept (Poster)