Learning and Retrieval from Prior Data for Skill-based Imitation Learning
Authors: Soroush Nasiriany
Tian Gao
Ajay Mandlekar
Yuke Zhu
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Skill Learning
Robot Manipulation

Abstract: Imitation learning offers a promising path for robots to learn general-purpose tasks, but traditionally has enjoyed limited scalability due to high data supervision requirements and brittle generalization. Inspired by recent work on skill-based imitation learning, we investigate whether leveraging prior data from previous related tasks can enable learning novel tasks in a more robust, data-efficient manner. To make effective use of the prior data, the agent must internalize knowledge from the prior data and contextualize this knowledge in novel tasks. To that end we propose a skill-based imitation learning framework that extracts temporally-extended sensorimotor skills from prior data and subsequently learns a policy for the target task with respect to these learned skills. We find a number of modeling choices significantly improve performance on novel tasks, namely representation learning objectives to enable more predictable and consistent skill representations and a retrieval-based data augmentation procedure to increase the scope of supervision for the policy. On a number of multi-task manipulation domains, we demonstrate that our method significantly outperforms existing imitation learning and offline reinforcement learning approaches. Videos and code are available at https://ut-austin-rpl.github.io/sailor

Learning and Retrieval from Prior Data for Skill-based Imitation Learning
Soroush Nasiriany', Tian Gao Ajay Mandlekar", Yuke Zhu'
The University of Texas at Austin, IISS, Tsinghua, NNVIDIA Research
Imitation learning offers a promising path for robots to learn general- purpose behaviors, but traditionally has exhibited limited scalability due to high data supervision requirements and brittle generalization. Inspired by recent ad- vances in multi-task imitation learning, we investigate the use of prior data from previous tasks to facilitate learning novel tasks in robust, data-efficient manner. To make effective use of the prior data, the robot must internalize knowledge from past experiences and contextualize this knowledge in novel tasks. To that end, we develop a skill-based imitation learning framework that extracts temporally extended sensorimotor skills from prior data and subsequently learns policy for the target task that invokes these learned skills. We identify several key design choices that significantly improve performance tation learning objectives to enable more predictable skill representations and a retrieval-based data augmentation mechanism to increase the scope of supervi- sion for policy training. On a collection of simulated and real-world manipulation domains, demonstrate that our method significantly outperforms existing im- itation learning and offline reinforcement learning approaches. Vidcos and code are available at https: //tt//uttaae austin-rpl. gi thub. io/sailor
Abstract:
novel tasks, namely represen-
Keywords: Imitation Learning, Skill Learning, Robot Manipulation
Introduction
A long-standing dream in robotics is to enable robots to perform general-purpose tasks in diverse environments. In recent years, imitation learning has led to great progress towards this goal. Recent work has demonstrated that well-engineered behavioral cloning methods can achieve competitive performance on diverse manipulation tasks [1, 3]. Despite this promise, these imitation learning algorithms have been confined to relatively small-scale domains. For real-world problems. current imitation learning algorithms often demand large number of task demonstrations, which can be difficult and costly to obtain.
Realizing these limitations, a series of recent work seeks to prior interaction data to improve the sample efficiency of imitation learning on new tasks. Such prior data come in various forms, includ- ing task-agnostic exploratory "play" data [4] or demonstrations previously collected for different tasks [5]. An open question is how best to extract knowledge from these large prior datasets and use this knowledge facilitate learning novel tasks. One promising approach is skill-based imitation learning [6. 7]. which aims to learn a latent space of short-horizon sub- uub-trajectories from the prior data (called skill learning) and subsequently learn a policy to invoke the skills to solve a specific downstream task (called policy learning). This approach offers several appealing advantages: First, the policy benefits from the temporal abstraction encapsulated by the skills, allowing the policy to focus on higher-level reasoning about what behavior to perform rather than how to execute that be- havior; and second, by reasoning about the target task relation skills trained on prior data, the robot implicitly distills knowledge from the rich diverse interactions of the prior data into the policy. Even so, existing skill-based imitation learning approaches bring modest improvements over simple behavioral cloning methods. This work aims at identifying the underlying limitations of existing approaches and designing new methods address these limitations.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

1. Skill representation learning
2. Retrieval for policy learning
3. Policy learning
Target Task Data
Aggregated Policy Learning Data
Prior Data
Sample
Policy
Z
2 0
Retrieved Prior Data
Figure 1: Overview. We present a skill-based imitation learning framework that uses prior data to effectively learn novel tasks. First, we learn latent skill model on the prior data, with objectives to ensure predictable skill representation. Given target task demonstrations, from the prior data, expanding supervision for the policy. We then train policy which outputs latent skills.
this latent space retrieve similar behaviors
We argue that practical approach should encompass two key properties. First, the latent skill space should serve as a predictable representation for downstream policy learning, allowing the policy to accurately infer which appropriate skill to use in new situations. While prior methods [8, 9, 6, 7] have widely adopted variational autoencoders to transform sub-trajectories into embeddings, we empirically show that this learning objective alone is insufficient for constructing well structured latent space. As a result, the policy tends to execute irrelevant skills. Second, the robot should take advantage of the prior data to learn both the skills and the policy. A major limitation of existing based imitation learning approaches [6. 7] is that they primarily focus on using prior data for skill learning but not policy learning. In these approaches, policies learned small number of target task demonstrations are prone to severe overfitting and covariate shift.
To address these limitations, we introduce Skill-Augmenice Imitation Learning with prior Retrieval (SAILOR). To improve the predictability of our skill representations, SAILOR uses an auxiliary temporal predictability objective to estimate the temporal distance between sub-trajectories in the demonstration sequences. To use prior data for policy learning. we introduce a novel retrieval-based data augmentation procedure that selectively retrieves data relevant to the target task. Specifically, we consider a subset of sub-trajcctories from the prior dataset that have high latent skill similarity wide variety
with sub-trajectorier in the target task demonstrations (see Fig. 1). We evaluate of manipulation tasks in simulation and the real world, and show that SAILOR significantly out- performs tftte-of-thearre imitation learning and offline reinforcement learning approaches. Through comprehensive analysis, we highlight the roles that prior data, our representation learning objective, and retrieval-bass data augmentation have in data-efficient learning of robust manipulation policies.
2 Related Work
Learning from Prior Data. There is a large body of work on learning manipulation tasks us- ing human demonstrations [1, 10, 11, 12, 13, 14]. While promising, most of these works learn tasks independently, without re-using knowledge from prior tasks. As a result, they have high data requirements and exhibit brittle generalization in complex long-horizon tasks. To address these lim- itations, several lines of work have investigated leveraging large offline prior datasets to facilitate learning downstream robotic tasks. These prior datasets include task-agnostic play data [4, 15], demonstrations for related tasks [5], self self-supervised -genrreeee data [16], or combination of these [17]. Alternatively to these, large video datasets are an appealing choice [18, 19, 20]. Recent work has leveraged these datasets to learn pre-trained visual representations for downstream control tasks [21, 22]. Yet despite their appeal, after experimenting with one such approach, R3M [21], we found that it can sometimes hinder downstream performance. One hypothesis is that the prior data on which these methods are trained on exhibit significant domain shift compared to downstream tasks, limiting transfer. In this work, we instead consider large multi-task robotic datasets.

---

Multi- Task Imitation Learning. Multi-task imitation learning methods offer a promising way learn from diverse robotic data. These include cask-conditooned [5]. language-conditioned [23, 4, 24, 25], and skill-hased [6. 7] imitation learning. While most task-conditioned and language- conditioned approaches seek to learn single policy that performs well across a series of tasks, we focus on learning task-specific policies by utilizing a large multi-task prior dataset to learn useful skill representation space, and supervising the task-specific policy using learned latent skills. These multi-task imitation approaches could be complementary to skill representation learning pipeline, ie., we can consider incorporating language and task ID supervision into our approach.
Skill-based Imitation Learning. We adopt skill-based imitation learning as the underlying frame- work for our method. The objective is to learn temporally abstract representations of sensory-motor data (termed skills) to enable more effective imitation learning. A long line of work has proposed learning skill representations by segmenting demonstrations into trajectory segments termed sub- trajectories. Among these include work on segmenting demonstrations into variable-length sub- trajectories, either an unsupervised manner [26, 27, 28, 29, 30, 31, 31] or relying on additional su- pervision [33, 34, 35, 36]. Recent work has shown promise for encoding fixed length sub-trajectories approaches. We adopt this setting due its relative simplicity and scalability. In contrast to these prior approaches (see Sec. 3), we enforce two key properties- a temporal predictability objective in our learned skill representation and retrieval-bate -base mechanism to improve task task-specific policy learning.
without any additional supervision [8, 9, 6, 7] using variational autoencoder-
3 Problem Formulation
Our goal is to leverage prior data to effectively learn novel target tasks in data-efficient manner. Formally, we consider target task as a Markov Decision Process M (S,A, r. Pu: ?) repre- senting the state space, action space, reward function, transition probability, initial state distribution, and discount factor. Our objective is to learn a policy that maximizes the discounted sum of rewards for the task. To learn the policy, we assume access to small offlin dataset Dtarget collected for the target task and large offline dataset 'D prior either from previous related tasks or task-agnostic interactions. These datasets consist of variable-length trajectories the form fon, Go 01, .., Or, } with 02 E O denoting the observations and a, E A denoting actions. We highlight that D prior and Prarget may have significant differences, i.e., the two datasets may come from different environ- ments, be collected by different human demonstrators, and demonstrate different tasks.
Skill-based Imitation Learning. We employ a skill-based imitation learning framework [6, 7] which consists of two stages: skill learning and policy learning. In the skill learn- ing phase, we learn a skill embedding space c I2d fixed-length sub-trajectories 7 = {oo, ag 01, OH- 0 in D priorr These skill embeddings serve as an abstract repre- sentation of the agent's behavior and can be invoked to solve range of downstream tasks. A num- ber of representation learning methods can be employed to learn the skill embeddings, spanning reconstruction- based methods and contrastive learning. the subsequent policy learning phase, we are given Drarget and our goal is to learn a policy π for the target task T. The policy now emits skill embeddings Z and during execution we decode 2 into sequence of actions {ao, a1...., - via a skill decoder model pa- To learn the policy, prior work has proposed parametric [6] and semi- parametric [7] polices that first parse segments of Dtarget into skills and subsequently learn to map observations in D Piarget to these skill embeddings. Note that icy- to distinguish the role of these two components. Skills represent thort-horizon behaviors that can be re-used across many tasks, while the policy solves a specific long-horizon target task.
distinct terms ferll-skill and pol-
4 Skill-based Imitation Learning with Retrieval
In this section, we describe our skill-based imitation learning approach that can leverage prior multi- task data to efficiently learn novel target tasks with small amount of task-specific demonstrations. As discussed in Sec. 3, this consists of two phases-a task-agnostic skill learning phase, where a latent skill space is learned using the prior data, and task-specific policy learning phase where task- specific data is used to learn policy using the skills as supervision. Compared to prior methods, our approach makes two key considerations--n we (1) ensure that the learned skill space is predictable
3

---

Skill Learning
Policy Learning
OH
of* Policy
0H
Skill encoder
Skill encoder
Skill encodir
Skill prior
Z
KI Z
2
Z
0b:II
Skill decoder
Temporal Distance Model
Skill decoder 00:8-1
0g.H7
d1
Figure 2: Model Overview. Our method consists of skill learning and policy learning phase. (Left) In the skill learning phase, learn latent skill representation of sub-trajectories via variational autoencoder. We include an additional temporal predictability term learn a more consistent latent representation. (Right) the policy learning phase, we train the policy to predict the latent skill given history of observations preceding the sub-trajectory To execute the policy, we decode the predicted latent using the skill decoder.
representation for downstream policy learning, and (2) improve the efficacy of task-specific policy learning by retrieving task-relevant datapoints from the prior dataset. See Fig. 2 for our model overview.
4.1 Learning a Predictable Representation of Skills
We learn skill representation by encoding sub-trajectories with variational autoencoder (VAE) [37], and we further introduce an auxiliary objective shape the representation. Denot- ing a given sub-trajectory т, we employ a long short-term memory (LSTM) [38] encoder dø that encodes into a Gaussian distribution over latent skills. Our decoder is an LSTM network ps that, for each timestep i, decodes a latent and the given observation og into the reconstructed action â, We additionally employ learned prior po encourage uub-trajectories with similar starting and ending observations to have similar latent representations [15]. Our VAE loss objective is then
H
CvAE(Ø, sb, 0) -Ezagg(=11)
Σ
log Po
8 www.:/o(llrllllllllooooooo oH)),
t=0
where B controls the effect of the KI. divergence term [39]'_
important to highlight that action reconstruction is not the sole objective of our skill learning model-learning a consistent and predictable representation of behavior is critical for downstream policy learning, as shown by recent work [40, 41]. While the KL divergence term is one step to- wards this objective (by encouraging skills be predictable given partial information from the sub- trajectories), in this work we introduce an additional temporal predictabiliry term that encourages the learned latent space predict the temporal difference between sub-trajectories. Specifically, given trojsub-trjjectories Ti and T2 from the same trajectory separated by timesteps, we learn a model m. to predict given the corresponding skill mean embeddings of the trajectories:
LTP(W; Φ) = (m ((((s(=)))) ()
p ((g(())))) -1)²
(2)
where μ denotes taking the mean of the distribution. We back-propagate LTp through the skill encoder model, allowing the this term to shape the learned skill representation. Note that this is just one way to encourage temporal predictability, other objectives are also readily compatible with our method, such as time-contrastive networks [42]. Our overall objective is weighted combination of the VAE and temporal predictability objectives:
Lsua (Ø, vb, 0,w) LvvE(d, v. + aCTPpw Φ).
In practice deterministic VAE decoder and compute the reconstruction loss using La distance.

---

Franka Kitchen: Prior Dala
CALVIN: Prior Data
Envoersae
Enverren 8
Bevtormnnt
CALVIN: Target Tasks
Selling
Franka Kitchen: Target Taak
Cleaning
Figure 3: Simulated Tasks, We perform extensive evaluations two simulation domains. (Left) Franka Kitchen: our target task involves specific permutation of four subtasks and consider two prior datasets: demonstrations involving all subtasks and demonstrations involving all subtasks except opening the microwave. (Right) CALVIN: we adopt the play dataset of Mees et al. [4] as our prior data and perform evaluations on two larget tasks: setting up the playroom environment and, conversely, cleaning up the environment.
Refer to Algorithm for detailed summary of our skill learning algorithm.
4.2 Retrieval-based Policy Learning
In the policy learning phase, we employ an LSTM policy that outputs the skill z to execute next. We train the policy a dataset poolicy {(o). ((qs(T:))), where т; is an H-length sub- trajectory, 2; is the mean encoding that sub-trajectory. and o', the frame-stacked history of F observations preceding the trajecraycc.ry. We train the policy to predict 2; from o using a standard behavioral cloning loss. During execution, we roll out the LSTM skill decoder Pv in a closed-loop manner, i.c., at each timestep observe or and exccute the next action a, u(pp (2, ot)). After rolling out the skill for H timesteps we repeat the process by sampling a new skill from the policy.
A common approach to skill-based policy learning is training the policy with all H H-length sub- trajectories in Dtarget- However, this limits the amount of supervision, especially when Ptarget is small. On the other hand, naïvely training on all sub-trajectories Ptarget and D prior can hurt performance [43] due to divergent and conflicting behaviors between the prior and target datasets. We thus introduce retrieval-basse mechanism train on sub- aub-trajectories in D prior that have high similarity with those Ptarget- While many similarity metrics are suitable, in this work we measure similarity with respect the skill embedding
sub-trajectories with similar skill embeddings demonstrate similar behaviors. First we obtain skill embeddings of randomly sampled sub-trajectories D Prrior and Drarget:
Zprioo (p(g)(T))) D priori
Ziarget (m((o(T;))) Diarget-
(4)
We then calculate the pairwise & distances between the prior and target dataset skill embeddings. i.e., D D[i [j] = Uperue Zungezl2z 12. Next, for each prior dataset skill embedding 2; Zprior. we find the closest corresponding target dataset skill, D min[i] min [i] [:]). Finally, retrieve the top-n sub-trajectories in 2 Pprior with the smallest distance argsort (D_ (__in) [:n], resulting in the retrieval dataset Dret- We train the policy using the aggregated set of sub-trate tub-trajectories in Dret and Drarget- We use a behavioral cloning loss split across two terms: one for Diarget. and one for Pret weighted by a factor 7 to control the effect of the retrieval data relative to the target dataset. At the same time training the policy we additionally fine-tune the skill model on Dtarget- We summarize the retrieval, policy learning, and skill fine-tuning steps in Algorithm 2.
5

---

Dataset
BC-RNN
BC-RNN (FT)
(RAM)
IQL.
(QDS(UDS)
FIST
SAILOR (ours)
Kitchen-Al All
9.0
1.
2.1
6.1
78.3 7.0
68.7 1..71.25
95.7 1.7
Kitchen-No Microwave
77.7 4 7.6
70.3 2:
83.0 0.8
95.0 2.2
CALVIN Setting Up
33.0 +: 2.2
72.8 5.4
26.0 28.0
7.7 1.7
16.3 2.6
3.7 4 2.4
77.3 3.1
CALVIN CALVIN-Cleaning Up
41.0 1.6
61.8.1 9.1
2:2
14.7
20.0
3.7
9.3 2: 5.0
88.0 1 5.1
Table I: Quantitative evaluation on two simulation domains. We evaluate our method SAILOR against a set of six baselines and report the mean task success rate and standard deviation over three seeds (exception: six seeds for BC-RNN (FT) due to high variance). Note: for the kitchen tasks we report one number for baselines that do not involve prior data. We see that SAILOR significantly outperforms the baselines all tasks.
5 Experiments
5.1 Simulated Experiment Setup
We perform empirical evaluations on two simulated robot manipulation domains (see Fig. 3):
Franka Kitchen [44]: A simulated kitchen environment involving different sub-tasks, such as open- ing cabinets, moving kettle, and turning on stove. This environment comes with large dataset of approximately 600 demonstrations performing various permutations of seven subtasks. In this dataset, subset of 18 demonstrations correspond to Drarget and demonstrate a specific permutation of subtasks: opening the microwave, followed by moving the kettle, flipping the light switch, and opening the sliding cabinet. We consider two prior datasets 'D prior (1) using all demonstrations except the ones corresponding to the target task (Ki (Kitchen-All); and (2) using all demonstrations except those that involve interacting with the microwave (Kitchen-be Mi Microwave). These prior datasets have 584 and 235 demonstrations, respectively.
CALVIN [4]: A simulated tabletop playroom environment accompanies by a large dataset of task- agnostic "play" data with 2.3M transitions. The play data encompass diverse behaviors, such as opening and closing drawers. turming on and off the lights, and picking, placing, and pushing blocks. We use all play data as 'D prior to solve two target tasks. The first target task involves setting up the playroom environment in multiple stages (CALVIN-Setting Up). Specifically, the robot must turn on the lights, and retrieve three blocks and place them on the table. The second target task in contrast involves cleaning up the playroom environment (CALVIN-Cleamie robot must open the drawer, place all three blocks into the drawer, close the drawer, and turn off the lights. For each task, we collect 30 demonstrations, which amounts to about half an hour of data collection.
Up). Specifically, the
The CALVIN domain is substantially more challenging than the Franka Kitchen domain, as the target tasks have a longer horizon and involve a greater number of objects. Also, in contrast to Franka Kitchen, the prior and target datasets in CALVIN are collected by different human demonstrators who exhibit different styles of teleoperation.
We learn vision-based policies for both domains. We refer readers to Appendix B for detailed discussion of our tasks and datasets, and Appendix C.5 for environment implementation details.
5.2 Quantitative Analysis
We evaluate our method SAILOR against state-of-the -hetttrrrtt imitation learning and offline reinforce- ment learning algorithms:
BC-RNN: behavioral cloning Ptarget without prior data. We adopt the ASTM-basee BC-RNN implementation in robomimic [1], which has shown superior performance over other behavioral
cloning approaches.
BC-RNN (FT): BC-RNN variant that leverages prior data. We first pre-train BC-RNN on D prior and subsequently fine-tune on D Ptarget- This baseline aims examine the effectiveness of supervised pre-training on interaction data for imitation learning.
BC-RNN (R3M): behavioral cloning on Piarget using a frozen R3M visual representation [21] pre-trained on the large-scale Ego4D video dataset [20]. This baseline intends to examine the effec- tiveness of using visual representations trained on natural images and videos.

---

Dataset
Ours
No TP
No Retrieval
All Retrieval
No Prior Data
CALVIN-Settng Up 77.3 + 3.1
68.0 + 3.7
65.0 - 2.2
64.3 + 10.8
50.7 6.5 60.7 3.1
CALVIN-Cleaning Up
88.0 1: 5.1
74.7 + 7.9
70.0 = 0.8
65.3 1: 7.3
Table 2: Ablation Results. We find that temporal predictability and retrieval are critical to skill-based im- itation learning. as without these components the agent struggles against a naive BC-RNN (FT) baseline. In addition, we validate that prior data plays a large role in the performance of our method.
IQL: Implicit Q-Learning [45], recent offline reinforcement learning method with state-of- of-the--rtttttttt performance on the D4RL dataset [46]; trained on Ptarget-
Implicit Q-Learning with Unlabeled Data Sharing [47, 48], which is a variant of IQL trained jointly on D prior and Diarget. where the transitions in D prior are labeled with the minimum reward (0 for our tasks). Singh et al. [47] and Yu et al. [48] show that this simple data augmentation procedure can effectively leverage prior data without additional rewards annotation.
IQL (UDS):
FIST: Few-shot Imitation with Skill Transition Models [7], an analogue of our method that employs a semi remi-parametric policy to select the skill to execute next. This baseline uses the same underlying skill model as our method but different policy learning scheme and does not involve retrieval.
Refer to Appendix C for additional implementation details on our model architecture, pseudocode, and evaluation protocols, and Appendix D for hyperparameter details. We report performance of all methods in Table 1. SAILOR greatly outperforms the baselines with an average task success rate of 89.0%. Notably it outperforms the most competitive baseline BC-RNN (FT) by 12.7%. BC- RNN performs poorly as it fails to learn an effective policy from small number of demonstrations. In comparison, BC-RNN (R3M) shows significant improvements on the Franka Kitchen tasks, but performs worse on the CALVIN tasks. We hypothesize that this is due to the limited generalization ability of the pre-trained visual representations. The offline reinforcement learning baselines show more promising results on the Franka kitchen tasks but struggle on the more challenging CALVIN tasks. Finally, FIST significantly under-performs our method. As FIST uses the same underlying skill model as our method, we attribute the limitations of FIST to its perammerattr policy.
5.3 Ablation Study
We perform an extensive ablation study to understand the effects various modeling choices on our method. First, we study the effect of the temporal predictability term in Eq. (2) on downstream task performance by removing it from Eq. (3) (No TP). Next we study the role of retrieval by training the policy solely on sub- uub-trajectories in Dtarget (No Retrieval). We also study the opposite case- training the policy on all of D Pprior and Drarget (A11 Retrieval). Finally,
study the role of prior data on our method by training the skill and policy solely on Diarget (No Prior Data). We also report ablations in Appendix A on the size of the prior, retrieval, and target datasets, in addition to the choice of retrieval method.
We present results in Table for the more challenging CALVIN tasks. First, find that both the temporal predictability objective and the retrieval mechanism have a significant impact on the final performance. It is worth noting that removing these components makes our model degenerate into the skill-based imitation learning setting of OPAL [6]. In fact, these two ablations perform worse than the naive BC-RNN (FT) baseline for the CALVIN- Cleaning Up task, indicating that both an effective skill representation and the retrieval mechanism play a critical role in skill-based imitation learning. The All Retrieval ablation also performs robot often was "distracted" and performed behaviors unrelated to the target task, likely due to the multimodal distributions in the prior data. Finally, the No Prior Data ablation validates the role of prior data in learning effective skills and policy. Comparing the No Retrieval ablation and the No Prior Data ablation, we see a 10 15% gap in performance, and this is attributed the fact that the skills in the No Retrieval ablation are additionally trained on the prior data. Despite the loss of performance in the No Prior Data ablation, we still see that it outperforms the BC-RNN baseline by significant margin. We attribute this to the temporal abstraction afforded by our skill-based learning framework. In sum, our ablation studies suggest that effective skill abstractions, coupled with mechanisms that effectively leverage the prior data, allow us to achieve strong results.
quuiiimllly--ullaattiivlllll we observe that the
7

---

Multi-Task Kitchen Environment
Target Tasks
Setting up Breakfast
Cooking Meal
Figure 4: Real World Tasks. On the left, we illustrate the set of objects use for collecting the play dataset. On the right shows two three target tasks, setting up breakfast and cooking.
5.4 Real World Experiments
Finally, we showease the efficacy of our method in the real world with kitchen environment in- volving eight food items, receptacles, a stove, and a serving area (see Fig. 4). We first collect play dataset of exploratory interactions involving the food items and receptacles. We consider three target tasks: (1) Real-Breakfast: setting up breakfast table by placing the bread, butter, and milk in the serving area; (2) Real-Cook: cooking a meal by placing the fish, sausage, and tomato into the frying pan: (3) Real Poaal-Cook-aart variant of the Real-Cook task involving placing the pan onto the stove. We collect 30 demonstrations; refer to Appendix B.3 for detailed descriptions on the tasks datascts. We evaluate SAILOR against the most competitive baseline, BC-RNN (FT) (see Appendix C.3 for our evaluation protocol). We find that while on Real-Breakfast both methods achieve a success rate of 76.7%, Real-Cook our method (FT) with a success rate of 73.3% vs. 23.3%, and similarly for significantly outperforms BC-RNN (76.7% vs. 46.7%) To the value of prior data we ran the No Prior ablation for the Real task, achieving 53.3% success rate compared to our 73.3%. Interestingly we see that the No Prior ablation largely outper- forms the BC-RNN (FT) baseline on this task (23.3%) which had access to additional prior data. Overall, we observe that BC-RNN (FT) often failed to correctly grasp objects. One hypothesis for this result is that pre-training stage biases the policy to learn the multi-modal behaviors in the prior dataset, preventing the policy from learning specialized target task behaviors during the fine-tuning phase.
Limitations
While our method shows significant promise, it leaves limitations that we hope to address in future work. First, acquiring large amounts of multi-task prior data is difficult and costly. To amortize the high cost, large prior multi-task datasets should be useful in a diverse range of downstream tasks, rather than a handful. In this work, we evaluate our method in a limited set of target tasks and leave it for future work to scale up the variety of tasks. We hope (and believe) that the need for large robotic datasets will be addressed in the coming years [10, 23]. Second, our method is more computationally expensive than the BC-RNN baseline [1], due to the higher number of losses and networks used. Third, our experiments focus on domains and datasets where the prior data and target tasks are reasonably close to each other. Notably our experiments do not evaluate generalization to unseen objects between the prior and target datasets. It would be interesting to investigate methods that are tolerant to much larger domain shifts between prior and target task data.
7 Conclusion
We present SAILOR, a skill-based imitation learning framework for robot manipulation. Our method uses prior data construct a latent space of predictable and consistent skill representations. It uses these latent skills as the temporal abstraction to learn policies for vision-based manipulation. Key to its effectiveness is our newly designed representation learning objectives and retrieval-based data augmentation procedure. We demonstrate that our method can solve horizoo manipulation tasks in simulation and on physical hardware. brings forth a data-efficient way of programming robots with new behaviors using small number of target task demonstrations. For future work, we plan to address the limitations we discussed in the previous section and investigate the effectiveness of this approach with various forms of prior data at different scales.
8

---

Acknowledgments
We would like to thank Jake Grigsby, Huihan Liu, and Zhenyu Jiang for providing feedback on this manuscript. We would also like to thank Yifeng Zhu for real robot infrastructure support. We acknowledge the support of the National Science Foundation (1955523, 2145283), the Office of Naval Research (N00014-22-1-2204), and Amazon.
References
[1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning. 2021.
[2] P. Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. Conference on Robot Learning, 2021.
[3] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
[4] O. Mees, L. Hermann, E. Rosete- Beatee and W. Burgard. Calvin - a benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Roboties and Automation Letters (RA-L., 7(3):7327-7334, 2022.
[5] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. In Robotics:
Science and Systems (RSS), 2022.
[6] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
[7] K. Hakhamaneshi, R. Zhao, A. Zhan, P. Abbeel, and M. Laskin. Hierarchical few-shot imi- tation with skill transition models. In International Conference on Learning Representations, 2021.
[8] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, 2020.
[9] K. Pertsch, Y. Lee, Y. Wu, and J. J. Lim. Demonstrationnguuider reinforcement learning with learned skills. In Conference on Robot Learning, 2021.
[10] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. Conference on Robot Learning, 2018.
[11] Wong, A. Tung, A. Kurenkov, A. Mandlekar, L. Fei-Fei, S. Savarese, and R. Martín-Martinn Error-aware imitation learning from telcoperation data for mobile manipulation. In Conference on Robot Learning, 2021.
[12] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep im- itation learning for complex manipulation tasks from virtual reality teleoperation. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[13] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Robotics: Science and Systems (RSS), 2018
[14] A. Mandlekar, F. Ramos, B. Boots, Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[15] C. Lynch, M. Khansari, T. Xiao, V. Kumar, Tompson, Levine, and Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2019.

---

[16] S. Dasari, F. Ebert. S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
[17] S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Systems (RSS), 2020.
[18] R. Goyal, S. E. Kahou, V. Michalski, J. Materzyúska, S. Westphal, H. Kim, V. Haenel, I. Fru- end, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The "something something" video database for learning and evaluating visual common sense. IEEE International Conference Computer Vision (ICCV), 2017.
[19] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, Munro, T. Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference Computer Vision (ECCV), 2018.
[20] K. Grauman, A. Westbury, E. Bymee Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu. C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do. M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Ge- breselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Souther- land, Y. Sugano, R. Tao, M. Vo. Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the World 3,000 Hours of Egocentric Video. In JEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2022.
[21] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022. URL https: /ttpss://aarxiii arxiv. arg/abs///0 12601.
[22] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control, 2022. URL https: / /arxiv. frrg/aba/2200 /t1///223...2.....22
[23] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing, 2021.
[24] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning. 2021.
[25] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. In Robotics: Science and Systems (RSS), 2021.
[26] G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot learning from demonstration by constructing skill trees. International Journal of Robotics Research, 2012.
[27] S. Nickum, S. Osentoski, G. Konidaris, and G. Barto. Learning and generalization of complex tasks from unstructured demonstrations. In IEEE/RSI International Conference on Intelligent Robots and Systems, 2012.
[28] Krishnan, R. Fox, I. Stoica, and K. Goldberg. DDCO: Discovery of deep continuous options for robot learning from demonstrations. In Conference on Robot Learning, 2017.
[29] T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In Inter- national Conference on Machine Learning, 2020.
[30] Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2020.
[31] T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and Battaglia. Compile: Compositional imitation learning and execution. In International Con- ference on Machine Learning, 2019.
10

---

[32] D. Tanneberg, K. Ploeger, E. Rueckert, and J. Peters. Skid raw: Skill discovery from raw trajectories. IEEE Robotics and Automation Letters (RA-L), 2021.
[33] K. Shiarlis, M. Wulfmeeer, S. Salter, S. Whiteson, and Posner. Taco: Learning task decompo- sition via temporal alignment for control. In International Conference on Machine Learning. 2018.
[34] Z. Su, O. Kroemer, G. E. Loeb, G. S. Sukhatme, and S. Schaal. Learning manipulation graphs from demonstrations using multimodal sensory signals. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[35] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-hon long-horizon robot manipulation. IEEE Robotics and Automation Letters (RA-L), 2022.
[36] Belkhale and D. Sadigh. Plato: Predicting latent affordances through object-centric play, 2022.
[37] D. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[38] Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 978):1735- 1780, 1997.
[39] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. 3-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017.
[40] A. Allshire, R. Martin-Martin, C. Lin, S. Manuel, S. Savarese, and A. Garg. Laser: Learning a latent action space for efficient reinforcement learning. In IEEE International Conference Robotics and Automation (ICRA), 2021.
[41] M. Yang, S. Levine, and O. Nachum. Trail: Near-optimal imitation learning with suboptimal data. In International Conference on Learning Representations, 2022.
[42] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[43] Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. In Advances in Neural Information Processing Systems, 2020.
[44] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, 2021.
[45] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
[46] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020. URL htttps://aax:: http:p:://aaxii..o//bbb///////// 2021s/2210.........
[47] A. Singh. A. Yu, J. Yang. J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. 2020.
[48] T. Yu, A. Kumar, Y. Chebotar, K. Hausman, C. Finn, and Levine. How leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, 2022.
[49] O. Khatib, Inertial properties in robotic manipulation: An object-level framework. Interna- tional Journal of Robotics Research, 1995.
[50] K. He, Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition, 2016.

---

[51] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Roboties and Automation (ICRA), 2016.
[52] A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizen nog-hhoizonn tasks from human demonstrations. In Robotics: Science and Systems (RSS), 2020.
[53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[54] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
12

---

Appendix
A Additional Experiments
A.1 Ablation on prior data
We perform a more fine-grained study on the role of prior data using the CALVIN domain. We compare to variants of our method using 25% or 50% of the available prior data to see whether the quantity of prior data plays a significant role on downstream policy performance. We also compare to variant of our method that only utilizes prior data collected from different environments than target task environments.
the context of our CALVIN domain, the prior data spans four environments (A, B, C. D), while the target task data only spans one environment (D). Thus for this ablation we only consider data from unseen environments (A, B, C) for our prior dataset. This ablation examines the robustness of our method under environmental mismatch between the prior data and target task data. We outline all results Table 3.
Dataset
prior data
25% prior data
50% prior data
ABC prior data
Ours
CALVIN-Setting Up SALVIN-Clcaning Up
53.3 = 4.0 60.7 = 3.1
71.3 + 0.5 76.0 = 2.9 82.3 2.1
76.7 6.5 77.3 + 10.1
77.3 + 3.1 88.0 +
80.3 + 3.3
Table Ablation on prior data.
First, we see that increasing the size of prior data yields greater downstream policy performance. There is significant performance gain from using prior data using 25% prior data, from which point increasing the amount of prior data leads to smaller gains. In addition, restricting the prior data to unseen environments still results in meaningful performance increase, which is promising sign that our method can operate even under controlled environmental distribution shifts between the prior and target task data.
A.2 Ablation on retrieval metric
In this work we use & distance in our latent skill space as the underlying distance measure for retrieval operation. We also consider performing retrieval based KL-divergence distances. ie. given two inference distributions and we compute their distance as the average forward and reverse KL divergence: d(g1. 92) 1 (Dxz(q (q9)l + Dxz(oellqı), This metric effectively incorporates both the mean and standard deviation of the inference distributions. We compare our standard retrieval procedure with the KL-based retrieval operation Table
Dataset
Ours
KL-based Retrieval
CALVIN-Setting Up CALVIN-Cleaning Up
77.3 + 3.1 88.0 = 5.1
79.3 + 5.7 80.7 + 0.9
Table Ablation on retrieval method.
We do not find significant difference between these two variants, suggesting that our method can work with alternative distance metries for retrieval.
A.3 Ablation on target dataset
We perform an ablation study on the size of the target dataset. We find for the CALVIN-Setting Up task that increasing the number of target task demonstrations from 30 to 100 yields an increase in success rate from 77.3% to 93.3% (see Table 5). Note that while it is promising that increasing the number of target task demonstrations yields an increase success rate, this comes at the expense of additional burden for human collecting demonstrations for the target task.
13

---

Dataset
30 demos (Ours)
100 demos
CALVIN-Sctting Up
77.3 + 3.1
93.3 = 4.5
Table 5: Ablation on target dataset.
A.4 Ablation on retrieval dataset
We perform a detailed ablation study on the quantity and quality of the retrieved data for policy learning. Recall that our retrieval procedure: (1) we first randomly sample N trajeccoooiie from prior dataset as possible retrieval candidates, (2) sort them according to their relevance to target task, and (3) select the top r% of candidates for retrieval. We study the following ablations, which use our standard settings of N=250,000 and r=10% unless otherwise stated:
No Retrieval: N=0. Ic. we retrieve no data
All Retrieval: N=sizeof(prior dataset), r=100%. Ic. we retrieve the entire prior dataset. For CALVIN this is 2.3M sub-trajectories
Random Retrieval: instead of sorting the retrieval candidates according relevance, randomly select 10% of the candidates. This is a test of data quality, to see whether the relevance of the retrieved sub-trajectories to the target task matters.
2 / 50 90 % Retrieval: retrieve r=2%, 50%, 90% of the N candidates. This is to test whether our setting of r=10% is good threshold for retrieval
Large Retrieval: N=sizeof(prior dataset). This ablation uses the same threshold r=10% as our method perform retrieval but considers all prior sub-trajectories as retrieval candidates and thus retrieves a significantly larger quantity of data.
Note that Ours, No Retrieval, and All Retrieval are from the original submission and we in- clude these results again for reference. We present results on the CALVIN Setting Up and Cleaning
Dataset
Ours
No Retrieval
Retrieval
Random Retrieval
2% Reteteral 50% Retrieval 90% Retrieval
Large Retrieval
CALYIN- Sening
05.-122. 700 1.8
82.0 3.7
75.7
24.3000
6.5
=
CALVIN Cleaning Up 98.0 1: 5.1
60.3 a 7.8
56.3 7.1
807 2.9
78.7
46.3 20.0
83.3
Table 6: Ablation on retrieval dataset.
Up tasks in Table 6. We make the following observations for CALVIN-Cleaning Up:
Data quality is important. The Random Retrieval retrieves the same quantity but lower quality of data Ours. The performance significantly degrades result. We the same trend from the 50 / 90% Retrieval experiments. Ic. as we increase the threshold for retrieval from 1=10% to 50% and 90% (and thus decrease the quality of data) see consistent and significant drop in performance.
Our standard setting r=10% optimal, striking the right balance between diversity and quality of data. Lower and higher thresholds (2%, 50%, 90%) perform worse.
Retrieving larger amounts of data does not have major impact performance. Large Retrieval achieves performance within the margin of error as Ours.
CALVIN-Setiing Up however offers a different analysis. For this task data quality does not appear to matter, as the Random retrieval, 2% / 50%, 90% Retrieval ablations all perform similarly to Ours within the margin of error. One possible explanation for this observation is that the Setting Up task involves a more diverse range of behaviors than Cleaning Up the Setting Up task involves manipulating all components of the environment whereas the Cleaning Up task involves a subset. Another potential hypothesis is that the prior data is more biased towards behaviors seen in the Setting Up task. Because many of the behaviors in the Cleaning Up task are mirror behaviors of the Setting Up task, this may result in unfavorable bias for the Cleaning Up task, necessitating
14

---

retrieval procedure to filter out irrelevant behaviors.
The implication of all of these results is that the importance of retrieval may be task and dataset dependent, with some tasks being especially sensitive to the choice of retrieved data.
15

---

B
Tasks and Datasets
B.1
Franka Kitchen
The Franka Kitchen domain consists of a simulated 9-DoF Franka robot operating in a kitchen environment comprising microwave, kettle, light switch, stove knobs, and sliding and hinge cabinet. In our experiments the agent operates the robot via joint torque control resulting in a 9-dimensional action space. For observations, the agent has access to proprioceptive information consisting of the 9-dimensional joint values of the robot, in addition to RGB images from a third- person view camera and an eye-in-hand camera.
Prior Data. This environment accompanied by approximately 600 human demonstrations each performing subset of four out of seven possible subtasks: opening the microwave, turning on the light switch, turning on the top burner, turning on the bottom burner, moving the kettle, opening the hinge cabinet, and opening the sliding cabinet. We consider two prior datasets D prior (1)
using all demonstrations except the ones corresponding to the target task (Kitchen-Alll and (2) using all demonstrations except those that involve interacting with the microwave "KichhenNol Microwave). These prior datasets have 584 and 235 demonstrations, respectively.
Target Task. We consider one target task demonstrating a specific permutation of subtasks: opening the microwave, followed by moving the kettle, flipping on the light switch, and opening the sliding cabinet. We define task success as whether the agent has completed all of these subtasks (in no particular order). For the target dataset Ptarget we obtain all demonstrations in the original dataset that perform this specific permutation of subtasks, resulting in 18 demonstrations. Note that this dataset is equivalent to the kitchen-conee cocpeen-comlleee complete--i ete-v0 dataset in the d4rl benchmark [46]. These demonstrations have an average length of 194 timesteps.
B.2 CALVIN
The CALVIN domain consists of simulated 7-DoF Franka robot operating in a playroom environ- ment comprising drawer, cubbies, two lights, and three blocks. The environment comes in four variants (see Figure 3), each with different textures, block sizes, and fixture locations. In our experi- ments the agent operates the robot via inverse kinematics control resulting in a 7-dimensional action space. For observations, the agent has access to proprioceptive information consisting the robot end effector pose and gripper state, in addition to RGB images from third shrdd-pesoon view camera and an eye-in-hand camera.
Prior Data. This environment is accompanied by large dataset of task-agnostic "play" data across all four environment variants and comprises 2.3M transitions. The play data encompass diverse behaviors, such as opening and closing drawers, turning on and off the lights, and picking, placing, and pushing blocks. We use all play data as DD Priior to solve two target tasks.
Target Tasks. We consider two target tasks:
CALVIN-Setting Up: the robot must turn on the lights, retrieve the pink block from the drawer, place on the table, and retrieve the red and blue blocks from the cubby area and place them on the table. We define task success as whether the agent has completed all of these subtasks (in no particular order). At environment resets the lights are always off, the pink block is randomly initialized inside the (closed) drawer, and the red and blue blocks are randomly initialized inside the cubby area with one block in the left region of the cubby and the other block in the right region of the cubby. For this task we collect 30 demonstrations, amounting to about half an hour of data collection. In these demonstrations, we first turn on the lights, then retrieve the pink block, then retrieve the first unoccluded block from the cubby area, then move the slider to retrieve the other block from the other side of the cubby area. These demonstrations have an average length of 584 timesteps.
CALVIN-Cleaning Up: the robot must open the drawer, place all three blocks into the drawer, close the drawer, and turn off the lights. We define task success whether the agent has completed all of these subtasks (in no particular order). At environment resets the lights are always on, the drawer closed, and the three blocks are randomly placed in left, center, and right regions of the table. For this task we collect 30 demonstrations, amounting to about half an hour of data collection. these demonstrations, we first open the drawer, then place the blocks by one into the drawer from right
16

---

to left, then close the drawer, and finally turn off the lights. These demonstrations have an average length of 572 timesteps.
B.3 Real World Kitchen
We designed real world kitchen environment to study the utility of our method physical hard- ware. Our kitchen environment comprises a Flexa toy kitchen", a set of toy food items", a number of serving items (placemat, plate, knife, fork), and small pot and pan that we purchased from local store. We use a 7-DoF Franka Emika Panda robot which is operated via Operational Space Control (OSC) [49]. We found OSC to be fitting choice, it offers task-space compliant behavior that makes for a more intuitive data collection experience. We restrict the OSC controller to the position and yaw of the end effector*, which combined with the gripper controller results in 5-dimensional action space. For observations, the agent has access to proprioceptive information consisting of the robot end effector pose and gripper state, in addition to RGB images from a third- third-person view camera and an eye-in- sye-in-hand camera.
Prior Data. We collect a large prior dataset of task-agnostic play behaviors involving the food items and the pot and pan. Overall our prior dataset involves 150 trajectories each with approximately 2,000 timesteps, resulting in approximately 300,000 total timesteps. For each trajectory initialize the scene by randomly sampling four out of eight food items (milk, bread, butter, sausage, fish, tomato, banana, cheese) and randomly placing these four items around the serving area. We also randomly initialize the pot and pan on the two front stove burners or occasionally place one on the table next to the serving area. We then randomly pick and place food items either on the table, the serving area, or the pot and pan. We also occasionally pick and place the pot or pan to the table or stove burners.
first
Target Tasks, We consider three target tasks:
Beak-Brea the objective of this task is to place the bread, butter, and milk from the table onto the serving area. These food items are initialized randomly in the vicinity of three possible locations on the table: the left, center, and right of the region preceding the serving area. We consider two possible permutations for the placement of object onto these three regions (in left- center-right format): butter-bread-milk, bread-butter-milk, and butter-milk-bread. The pots and pans are initialized on the front stove burners. We define task success as whether the robot has (in no particular order) placed the bread onto the plate, the butter to the left of the plate the placemat, and the milk to the right of the plate on the placemat. For this task we collect 30 demonstrations, amounting to about half an hour of data collection. In these demonstrations butter, and milk in order onto their corresponding goal locations. These demonstrations have an average length of 546 timesteps.
place the bread,
the objective of this task is to place the fish, sausage, and tomato from the table into the pan. These food items are initialized randomly in the vicinity of three possible locations on the table: the left, center, and right of the region preceding the serving area. We consider three possible permutations for the placement of object onto these three regions (in left-center-right format): fish- mausage-tomato, sausage- sausage-fish-tome fish-tomato, and on the front stove burners. We define task success whether the robot has (in no particular order) placed these three items into the pan. For this task we collect 30 demonstrations, amounting to about half an hour of data collection. In these demonstrations we place the food items from left to right (in order) into the pan. These demonstrations have an average length of 548 timesteps.
Real-Cook:
tomato-sausage The pots and pans are initialized
Real-Setup-Pant the objective of this task is to place the pan from the table onto the stove and subsequently place the fish and sausage into the pan. The pan is initialized randomly in the vicinity of the right region of the table preceding the serving area. The food items are initialized randomly in the vicinity of the left and center regions of the table preceding the serving area. We consider three possible permutations for the placement of the objects onto these three regions (in left-center-right format): bish-tomato-piie -ant and tomato- fish-pan. The pot is initialized on the front stove burners. We define task success whether the robot has (in no particular order) placed the pan onto the stove
Phttps://er amazon. Mooa/MelisaaDoug---- Food-Groups Saruups-HandPaaee Hand-Painteed ht//////////// BO000BBBMA
Wee did not find the roll and pitch actuation to be necessary for our real world tasks and we opted for simpler action space.
17

---

and the two food items into the pan. For this task we collect 30 demonstrations, amounting to about half hour of data collection.
18

---

C Implementation Details
C.1 Model Architecture
We elaborate further our method architecture outlined in Figure 2. Our implementation is based on top of robomimic", a recent open source codebase with extensive benchmarking results across a number of imitation learning algorithms. We adopted the same neural modules (same RNN back- bone, VAE, visual perception encoders) for our algorithm, and in fact our BC-RNN baseline uses the exact implementation from robomimic.
Our model specifically consists of five neural network modules: four networks for the skill model comprising RNN encoder, an RNN decoder, feedforward VAE prior", and feedforward tem- poral prediction network; and one RNN network for the policy.
Observation Encoder. Four out of the five modules described above take observation inputs (among other potential inputs), and each of these modules is equipped with an observation encoder to pro- cess these observations. The observation encoder specifically consists of ResNet- 18 backbones [50] to encode the third-person image and eye-in-hand image, and a multi-layer perceptron (MLP) for all remaining low-dimensional observational inputs. Note that we pre-process the ResNet inputs with random cropping and post-process the outputs with a Spatial Softmax [51] pooling layer. After pro- cessing the image and low-dimensional observation imputs form one unified observation encoding. Note that our RNN encoder, RNN decoder, and RNN policy process sequence observations individually using the observation encoder and then processes these encoded observations into one unified representation with a recurrent neural network.
concatenate the resulting outputs to
Skill Model. The skill model is Variational Autoencoder that encodes uub-trajectories into a latent skill representation and decodes information back into the actions of sub-trajectories. The skill encoder and decoder are RNNs with a 2-layer LSTM followed by a 2-layer MLP, while the VAE prior and temporal prediction network are 2-layer MLPs.
Policy. The policy is a 2-layer LSTM network that maps a history of F observations into latent skill z. We also condition the policy on dataset id {0.1} indicate whether the policy is optimized on the target dataset or the retrieval dataset, to prevent potential interference between the target and retrieved data (see Algorithm 2 for additional details). Note that we can extend our policy to incorporate fine- mne-graieed goal information by conditioning on additional context information such as goal images or language goals [44, 52, 23].
C.2 Training
Our algorithm consists of two phases. In the first phase we pre-train our skill model on sub- trajectories in 2 prior (see Algorithm I for further details?). In the subsequent phase we are given the target dataset Ptarget and proceed to learning the policy and fine-tuning the skill model. Before we perform policy learning, we first retrieve sub-trajectories in Pprior that have similar embeddings those in Ptarget. We aggregate these retrieved embeddings into our retrieval dataset D Pret- We then proceed to train the policy jointly on embeddings from Ptarget and Dret- At the same time we continue to fine-tune the skill model with sub-trajectories sampled from both D Pprior and D Ptarget- We summarize these steps in Algorithm 2.
We sample fixed-length sub- sub-trajectories uniformly at random to train our model, following recent skill-based imitation learning works [6, 9, 7]. More specifically, for each dataset we concatenate all trajectories into one continuous stream of data and uniformly sample sub-trajectories from this stream. Note that this can result in sampling overlapping sub-trajectories. For training the policy we additionally train the frame stack of observations preceding the sampled sub-trajectory. There are some edge cases, such as when the sub-trajectory intersects with the next trajectory and when
'tttps://github com/ARISE--T wwm/ARRSE-Iiiiiatiiveeeee Inwiiatiieerroommtttit...teeeeee
'we also utilize feedforward deterministic inverse dynamics model but we found that it does not lead to significant change in downstream policy learning results
in our code we also have a slowness term to ensure that two nearby sub- sub-trajectories have similar skill em- beddings. We did not find this feature to have noticeable impact on downstream policy performance and therefore omit it from the algorithm pseudocode for simplicity.
19

---

the Trame-stack intersects with the previous trajectory. We deal with these cases by padding all data from the offending consecutive trajectory with the first / last observation of the current trajectory.
Algorithm Skill pretraining
Input: prior dataset 2 prior
1: while not done do
2: Sample {(or, ((or,ae))ee I= H-1 prior
3: Sample temporal offset d [-50,50 50] and obtain 7' shifted d timesteps from т
4:
Mr. E. (T(
// Encode 7
5:
Pt', Ex. t (o(T')
// Encode
6:
2 2~ N(,, Enr)
// Sample latent
7:
à t (e,
// Decode actions through RNN
8:
d s (1(r [[ye)
// Predict temporal distance
9:
LVAE + lia - + D oH)) // Compute VAE Loss
10:
LrC + a (d d)²
// Compute TC Loss
SSKII LVAE Lre
12:
update d, db, 0, w Lsan via gradient descent
13:
end while
Algorithm 2 Policy learning and skill fine-tuning
Input: prior dataset D prior- target dataset Dtarget. pre- pre-trained skill encoder qá
1: // Obtain retrieval dataset
2: Zprior + (u((s(())) (T))) D prior
// Sample and encode N sub-trajs from D prior
3: Zurget + {p(9s ((o(g((T)))) (T)))) Ptarget
// Sample and encode M sub-trajs from Dtarget
4: Dij = -1 Zprioell
// Compute all pairwise encoding distances
5: D D_min; min, (D,)) // Find closest target sub-traj for each prior sub-traj
6: K + argsort (D min) [:n] // Compute list of indices in Zurget with minimal distance
7: Drett :lo7.: Zprind BRek // Retrieve skill embeddings and their preceding observations
8:
9: Traii policy
while not done do
11: Sample (0fs,2) Ptarget
I/ target dataset sub-traj encoding and frame stack // retrieval dataset sub-traj encoding and frame stack // predict skill for target dataset sub-traj
12:
Sample (0')))²) Dret
& + (ofa,iid 0)
14:
3' + T(f T'o id 1)
// predict skill for retrieval dataset sub-traj
Lpolicy - (2 182-2 =)² + -2'-
// Compute Policy Loss
16:
update on LPllie via gradient descent
fine-tune Lskill on sub- sub-trajectories sampled from D Prior and Dtarget see Algorithm 1
18: end while
C.3 Evaluation
To perform policy rollout, we first obtain a skill = =0 id = 0) from the policy. We execute this skill with our closed-loop skill decoder Pv (o,,2) for H timesteps and we subsequently repeat the process by obtaining new skill from the policy. Note that we do not preempt skill execution; we execute all timesetps until completion We terminate the episode either when the agent has successfully solved the task if the agent has exceeded the time budget for the rollout. We assess each episode based on whether the agent successfully solved the task in the allotted time budget. While other metric also exist (time to complete task), we chose binary success for its popularity and relative simplicity. We elaborate further on our evaluation protocol:
'We believe this is reasonable choice, (1) the closed-loop skill decoder can react to current environment con- ditions during skill execution, and (2) the policy is still operating high frequency and can react accordingly
20

---

Simulation Experiments: We evaluate the success rate across 3 seeds (unless otherwise noted) and report the average and standard deviation across all seeds. To evaluate a seed, we perform 100 policy rollouts every checkpoints and record the success rate for each checkpoint. We then record the success rate for the seed as the highest success rate across all checkpoints evaluated for that seed".
Real World Experiments; Due to the challenges of real-world evaluation we only evaluate I seed for each baseline. To evaluate an experiment, we perform an initial evaluation of different policy checkpoints, evaluating each checkpoint for only a few trials. Upon choosing the most promising checkpoint we perform 30 rollouts and report the success rate over these rollouts.
C.4 Baselines
All baselines are implemented the robomimic codebase for fair comparison. We briefly elaborate on these implementations follows:
BC-RNN: We use the default implementation BC-RNN in robomimic and we use identical hy- perparameters as those reported in the robomimic study paper [1].
BC-RNN (FT): We use an identical architecture and identical hyperparameters as BC-RNN. We first train the baseline on D prior and subsequently fine-tune on Dtarget via a second stage of training. We also experimented with jointly training a task-conditioned BC-RNN policy in D prior and Dtarget we found that yielded very poor performance due to the multi-modality of actions in the prior data.
BC-RNN (R3M): We use an identical architecture and identical hyperparameters as BC-RNN but with a pretrained R3M visual representation. We specifically replace the weights of our ResNet- 18 networks with the pretrained ResNet-18 weights from R3M We follow the same practice from the R3M paper and we freeze the pretrained ResNet weights during downstream imitation learning.
IQL: We base our implementation off of the publicly available PyTorch implementation of IQL".
IQL (UDS): We make small modification to our IQL. implementation. For each batch that we sample from Dtarget. we also sample an equivalent-size batch from 'D Prioor with the rewards set to 0. We then perform gradient updates the aggregated data from both of these batches.
FIST: use the same underlying skill model as our method but a semi-parametric policy place of our parametric neural network policy. We use an identical scheme for the semi parametrriii policy as the FIST paper [7].
C,5
Environment Implementation
C.5.1 Gripper Logic
We elaborate on the gripper logic in our environments. The gripper state is either the position of the gripper fingers (Franka Kitchen, Real World Kitchen) or the opening width of the gripper (CALVIN). The gripper action is continuous 1-D variable, and we interpret this either opening (if < 0) or closing (if > 0). The gripper is controlled via position control. When the agent specifies closing action the position target of the controller is set to close the gripper fingers all the way (and for opening the target is set to open the gripper fingers all the way). There are limits on the force and velocity of the fingers in order ensure gripper stability.
this is the same evaluation protocol used in [1] Phttpp///gsshib..cc/aaaabbbrrere
21

---

D Hyperparameters
We adopted similar set of hyperparameters as the BC-RNN baseline from robomimic-we used the same LSTM settings, batch size, and RNN policy history length of F = 10. We did experiment
with different choices of sub-trajectory lengths for the skill model (H {1, 5, 10, 25}) and found that H = 10 performs optimally. Longer horizons may be helpful in some settings, however we hypothesize that RNN-based architectures lack the capacity to accurately predict actions over sig- nificantly longer horizons. It would be interesting to investigate if the optimal H changes under Transformer-bass [53] architecture.
Hyper-parameter
Value
Skill encoder: LSTM hidden units
1000
Skill encoder: MLP hidden sizes
1024, 1024
Skill decoder: LSTT hidden units
1000
Skill decoder: MLP hidden sizes
1024, 1024
Skill prior: hidden sizes
1024, 1024
TC: hidden sizes
128, 128
Policy: # LSTM hidden units
1000
Sub-trajectory length H Skill latent dimension
10
64
Skill KL weight 3
le-5
TC weight o Retrieval weight 7
1e-6
0.15 for Real tasks, else 1.0
# observation history frames F
10
Batch size
16
Optimizer
Adam [54]
Learning rate: skill VAE
5e 4
Learning rate: TP
le- 4
Learning rate: Policy
1e-3
Retrieval: max # D prior samples N
300,000 for Real tasks, else 250,000
Retrieval: max # Dtarget samples M % of samples chosen for retrieval
2,500
5% for Real tasks, else 10%
Table 7: Hyperparameters for our method.
22

---

Method
Epochs
Evaluation checkpoints freq
BC-RNN
800
40
Franka Kitchen: 300
BC-RNN (FT): phase
CALVIN: 600
Real tasks: 300
Franka Kitchen: 400
20
BC-RNN (FT): phase 2
CALVIN: 400
20
Real tasks: 600
50
BC-RNN (R3M)
800
40
IQL
800
40
IQL (UDS)
800
40
Franka Kitchen: 300
FIST: phase
CALVIN: 600
FIST: phase 2
200
10
Franka Kitchen: 300
Ours: phase
CALVIN: 600
Real tasks: 200
Ours: phase 2
200
10
Table 8: Training and Evaluation Hyperparameters.
23

---

Task
Image Size
Rollout Length
Franka Kitchen
x
280
CALVIN: Setting Up
84 x 84
1000
CALVIN: Cleaning Up
x
1000
Real Breakfast
128 X 128
1500
Real Cook
128 x 128
1500
Table 9: Task Hyperparameters.
24


Reviews:
[{"review_id": "pF_kOuDHaMp", "reviewer": "Reviewer_Qe4M", "summary": "This paper addresses the problem of imitation learning. Specifically, this work assumes access to a prior dataset that contains task-agnostic play data or demonstrations solving different tasks and extracts temporally-extended skills from it. Then, a small sef of demonstrations for the target task are provided for learning the target tasks. To this end, the paper proposes a framework that is built upon existing skill-based imitation learning methods, while introducing two techniques: (1) when learning skills, additionally optimizes a temporal consistency term that encourages the learned skill space, and (2) when learning policy, leverages the prior demonstrations which are similar to the target demonstrations. The experiments on two robot manipulation domains (Franka Kitchen and CALVIN) show that the proposed framework outperforms baselines, including BC, IQL, and FIST. Ablation studies justify the effectiveness of the two proposed techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Paper strengths and contributions\n**Motivation and intuition**\nThe motivation for leveraging prior datasets to accelerate target task learning is convincing.\n\n**Technical contribution**\n- Encouraging the learned skill space to capture temporal difference seems effective and this paper presents a reasonable implementation to achieve it.\n- Augmenting the target dataset using similar demonstrations from the prior dataset seems simple yet helpful.\n\n**Clarity**\nThe overall writing is clear. The authors utilize figures well to illustrate the ideas.\n\n**Ablation study**\nAblation studies are comprehensive. The proposed framework incorporates two techniques and the provided ablation studies are helpful for analyzing the effectiveness of each of them.\n\n**Experimental results**\n- The presentation of the experimental results is clear.\n- The experimental results on two robot manipulation domains (Franka Kitchen and CALVIN) show that the proposed framework outperforms baselines, including BC, IQL, and FIST.\n\n**Reproducibility**\nGiven the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. \n\n## Paper weaknesses and questions\n\n**Navigation and locomotion domains**\nThis work evaluates the proposed framework in two robot manipulation domains. It would be more convincing if the experiments were also conducted in navigation or locomotion domains. I believe the proposed temporal consistency term would yield an even larger improvement in navigation domains.\n\n**Implicitly predicting time step difference**\nThis work proposes to predict the time step difference between two sub-trajectories for encouraging temporal consistency. I wonder if implicitly learning this (such as performing contrastive learning where the model select which sub-trajectories are closer in terms of temporal difference, see \"Time-Contrastive Networks: Self-Supervised Learning from Video\" by Sermanet et al.) would also yield improvement.\n\n**Kitchen-No Microwave vs. Kitchen-All**\nI wonder why FIST and IQL (UDS) perform better on Kitchen-No Microwave, which is supposed to be harder compared to Kitchen-All.\n\n**Similarity between the prior dataset and the target dataset**\nThis work assumes that the prior dataset and the target dataset are similar enough so that learning from some of the prior demonstrations is helpful. I found this assumption a little too strong.\n\n**Similarity between trajectories**\nThis work proposes to use $L_2$ in the learned skill space to measure the similarity between two trajectories. At first glance, it does not look like a good idea. Yet, the improved performance backs it up. I feel I would get more convinced if prior demonstrations with high and low similarity are qualitatively presented.\n\n**$\\mu$ in eq(2)**\nI was a bit confused reading Eq. (2) since $\\mu$ in Eq. (2) is not introduced when it is first mentioned there. It is only mentioned in L157.\n\n====== After rebuttal ======\n\nI thank the authors for the response and the revision, which addresses many of my concerns. I have carefully read other reviews and decided to keep my original rating (weak accept).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this work studies an important and active research direction (i.e. skill-based learning in RL) and proposes two interesting reasonable techniques. The experimental results justify the effectiveness of the proposed techniques. Yet, I have a few concerns (stated above) such as the limited scope of the evaluation (only in robot manipulation), I am leaning toward accepting this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Described in the Strengths And Weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "ZEop9zpUqY", "reviewer": "Reviewer_bYiA", "summary": "This work proposes an imitation learning framework that extracts temporally-extended sensorimotor skills from prior data and subsequently\nlearns a policy for the target task with respect to these learned skills.\nThe contributions of this manuscript are the temporal consistency term in the VAE loss function, a\nrepresentational learning objective to enable more predictable and consistent skill representations,and the data retrieval based on the ensuing encoder,\nto increase the scope of supervision for the policy.\nThere are both simulated and real robot results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe contributions of this manuscript are the temporal consistency term\nin the VAE, and the data retrieval based on the ensuing encoder.\n\nExperiments involve simulation and hardware, and compare the method to\nother baselines.\nGood set of ablation studies, which ties in nicely with the baseline\n comparison (like against BC-RNN without D_prior, as the authors'\n method without D_prior still performs better)\n\nData retrieval based on skill matching is interesting\n\nISSUES and CONCERNS\n\nThe real robot tasks could be much more compelling.\nThe tasks seem limited to pick and place. It would be easy to\nuse traditional methods to achieve the same results, so why bother with\nthe complexity and risk of learning approaches?\nFrom a perceptual point of view, for the real robot there is a set of\nfixed objects - no variation in size, shape, or appearance. It would have\nbeen better to do the experiments with real vegetables and fruit, which\nhave natural variation.\nThere is generalization in terms of adjusting to object and goal locations.\nThere is no evidence for generalization in terms of objects.\n77% performance on these tasks is not impressive.\nIt is not surprising that the ABC prior data vs.\nD task ablation works well (around\nline 433) as all the tasks are similar.\nWhat about using the utensils to actually manipulate something else,\npouring the milk,\nactually using a real stove (turning knobs and pressing buttons),\nspreading butter on toast or something else for the breakfast task,\ncutting the bread, sausage, or fish, or \npeeling the banana?\n\nThis paper presents a simplistic view of what a skill is.\n1) Only position, yaw, and gripper state are controlled in the real robot\nexperiments. This is a reflection of the limited nature of the tasks\nexplored in this paper.\n2) Skills are limited to trajectories (open loop sequence of actions) only, and trajectory-only skills\ndidn't get robotics very far over the past decades. \nThat is why robotics  adopted:\n - force control, impedance control, and other control laws\n   (including nonlinear control laws)\n - timing and transition rules for finite state machines\n - error handling\nIn situations with rigid objects and rigid environments, something must be\ncompliant or infinite (large) forces result. That means either some\nmechanical compliance in the robot or some form of force control.\nI don't see how the proposed system would be extended to handle these\naspects of skills.\n\nFor many tasks a solution has to be a policy u=pi(x). An open loop sequence of\ntasks cannot solve unstable tasks, for example.\nThis is a mistake made in good old fashioned AI.\nIt is true that demonstrations or any instance of a task execution\nare sequences of actions, but this does not imply that a controller\nthat just emits previously successful sequences of actions will succeed.\nConsider riding a bicycle. A bicycle rider can't just play back the\nsteering and balance motions used in a previous ride, they will quickly\nfall over.\n\nI had to go to the supplementary document to figure out what the\nactions were. It turned out they were different in the 3 cases:\n452: agent operates the robot via joint torque control resulting in a\n9-dimensional action space\n(joint torque control is probably a bad choice)\n474: agent operates the robot via inverse kinematics control resulting\nin a 7-dimensional action space.\n507: OSC, position and yaw of the end effector, gripper state.\n\nIt is not clear what the gripper state is (open/close, or finger\npositions).\nHow is gripper closing controlled? Do the gripper fingers\ngo to a fixed position, or is a closing force specified?\nIf positions are controlled, what happens if a position is too small\nfor the object being gripped? The gripper outputs a maximum force?\nAll of this should be clearly explained in the paper itself.\n\nIt is not clear how goals are specified to the system? Desired final\npositions for objects? An optimization criterion? Some measure of\nsimilarity to a demonstration?\n\nThe network architectures chosen were not justified or compared to\nalternative architectures.\n\nThe paper does not make clear which trajectory segments (sub-trajectories)\nare used from D_target or D_prior. Are all possible trajectory segments\nused, in which case adjacent segments of length H share H-1 values\n(which is implied by the use of the term \"all\" in line 160 and 162)?\nOr are trajectory segments distinct, with no overlap between adjacent\nsegments?\nOr are trajectory segments initiated or terminated at significant events\nlike making or breaking contacts?\nPrior work (6, 8) is described as \"parsing\" the demonstrations (line 122),\nbut how does the parse/segmentation work in this paper?\n\nIn line 163 similar sub-trajectories are retrieved. If sub-trajectories\noverlap, this will choose many from the same demonstration. Is that good\nor bad?\n\nHow is H picked? I would imagine it would have a huge possible range\n(some skills are quite short, while others (like steering a car) last\na long time (as long as one is driving the car)).\n\nIt is not obvious to this reader that predicting temporal differences\nis useful, but let's see what happens in the ablations section.\nOne could always decode the latent representations producing trajectory\nsegments and then explicitly correlate the trajectory segments.\nHow is that a constraint on the latent representation, if the\nencoder/decoder pair already work well?\n\nIs F, the number of observations in the argument to the policy, a constant?\nHow it chosen? Wouldn't it make more sense to map observations into a\nobservation-latent-space (this is what state estimators like Kalman\nfilters do), and use the latent variable (the \"estimated state\") as\nthe first component of the training points as well as the index into\nthe policy? Perhaps the output of the perception network is such an\nobservation latent space.\n[I see this is addressed starting at line 547 in the supplementary materials.\nPerhaps it should be discussed in the paper.]\n\nHow is it decided where to start the readout of a trajectory? Always\nat the start?\n\nHow is it decided when a new/different skill should be executed?\nWhen the readout of the trajectory is finished (which is implied by\nthe paper's reference to \"H timesteps\" on line 159)?\nThere is no preemption of the current trajectory if things do\nnot go as expected? This leads to robots doing stupid things like trying\nto close an automatic door when the person going through is having trouble\nwith their luggage, and the door starts swatting them.\n\nIn the discussion of \"retrieval-based\", what is retrieved?\nThe actual trajectory segment? The latent representation?\n\nThe policy is trained using instances (input: o_fs, output: tau).\nThe retrieval-based policy learning focuses on similarities of tau.\nWhy not focus on similarities of o_fs? It would seem to this reader\nthat for a policy similar inputs should lead to similar outputs\n(except at a policy discontinuity), but I would not expect that\nsimilar outputs are caused by similar inputs (Many (most?) policies\nare many to one mappings, in that the dimension of the state or\nobservation space is usually much larger than the dimension of the\naction space, and by replacing actions with skills, the dimension\nof the policy output space is reduced further.)\n\nI don't understand the purpose of the binary input bit saying whether\nthe training point is from the prior or target demonstrations.\n\nHow are initial state and goal conditioning of trajectories done (how\nare the trajectories morphed to start and stop in the right places)?\nHow are grasping trajectories modified to be appropriate for an object.\n\nThe real robot video is impressive. I wouldn't bother presenting the\nsimulation results, and just focus on the real results.  You give 9\nlines to real results, and more than 10 times that to simulation\nresults. Do the comparisons and ablations on the real robot!  Much\nmore convincing!  Though simulation is helpful to speed up\ndevelopment and evaluation, the reviewer believes that\nit is ultimately the real robot experiment that demonstrates the value\nof the authors' method. The reviewer would suggest the authors put\nmore emphasis on the real robot experiment and conduct quantitative\nanalyses similar to the ones completed in simulation on the real robot\nsetup. This change could make the readers better appreciate the value\nof the authors' method and its performance.\n\nThe authors have used only \"success rate\" as the performance metric in\nthe real robot experiment.  However, for the authors' method to be\npractical in the real world, the reviewer believes that performance\nmetrics are not limited to success rate---other, even more important,\nperformance metrics exist.  Since the task in the real robot\nexperiment is setting up a breakfast table and cooking a meal, one of\nthe relevant performance metrics could be the total time for\ncompleting the entire meal preparation process.  Obviously, if such a\nrobot cooks a meal in a real household, the users would tend to prefer\nthe robot the finish the meal within a reasonable and predictable\namount of time. In this regard, the reviewer would suggest the authors\nto put more emphasis on practical performance metrics for the task\nchosen for the real robot experiment, instead of a single \"success\nrate\".\n\nHow do we know whether the skills failed (the output action sequence\nis not good enough)? It appears the simulators simulate physics as joint\ntorques are generated, but there is little description of them.\nIn lines 577-584 the words \"we evaluate\" are used, but I have no\nidea what those words actually mean. What is a checkpoint? Who chose\nthem? How are they evaluated?\n\nIt’s not clear exactly how this paper fits into the broader context of\nexisting research in the field. A major shortcoming is the discussion\nof prior approaches for skill learning - the authors seem to only be\nreferencing works that use variational auto-encoders.  While the\nproposed method does demonstrate relatively good performance in the\nevaluation domains, the discussion of the results does not provide a\nlot of intuition about why other baselines failed to perform as\nwell. I think this paper would be much improved by more explicitly\ndescribing how this work compares to prior works and the\nbaselines. Specifically, what mechanisms can we attribute to the\nsuccess of the method?\n\nThe distinction between a skill and a policy is confusing. The terms\nskill and policy are used interchangeably in many contexts. I think by\nskills the authors are referring to the latent embedding.\n\nThe authors claim “A major limitation of existing approaches is that\nthey only focus on using the prior data for skill learning but not\npolicy learning.” This needs to be better supported. Among works that\ndon’t make a distinction between “skill” learning (i.e. learning\nlatent embeddings) and policy learning, prior data has been used to\nupdate the\npolicy. (http://proceedings.mlr.press/v139/wulfmeier21a/wulfmeier21a.pdf\nis just one example)\n\nFigure 2: I think the components of the system should be better\njustified. Why make a distinction between skill learning and policy\nlearning? Why not just learn low level policies that are “skills” and\na high-level policy that selects the skills? What is the advantage of\ncontinuous skill embeddings vs. having a discrete high-level policy\nthat learns to select skills?\n\nSection 5.1: Quality of the prior and target datasets - Because\nimitation learning is used and there is no explicit representation of\ntask goals or reward, the quality of learned policy is dependent on\nthe quality of the data in the prior and target datasets. It would be\nhelpful to have more detail about how each dataset affects the learned\npolicy. For instance, I would imagine that it is more important for\nthe target dataset to have high quality demonstrations, since the\nsamples from the prior dataset are chosen to be close to the\ntarget. What would happen if the target had poor demonstrations and\nbut the prior had higher quality demonstrations and vice versa?\n\nWhat is the relationship between the number of sub-trajectories\nsampled from the prior dataset and policy performance? What would\nhappen if you thresholded the distance instead of selecting a fixed\nnumber? If the prior was poor quality or not particularly relevant to\nthe target, it would probably be more beneficial to use threshold for\nselecting sub-trajectories. Depending on the data, it may be better to\nnot take any samples from the prior.\n\nLine 225: What is the form of the non-parametric policy used in the\nFIST baseline? And what would the performance of this method be if\ndata retrieval was used with the non-parametric policy?\n\nHow could knowledge of task goal/reward be incorporated into this\nframework? It seems like indexing the most similar trajectories would\nnot always be sufficient to filter out bad examples. There can be poor\ndemonstrations that are closer to the target than other samples, but\nstill would not result in reaching the goal state. Relabeling the\nsub-trajectories with rewards for the target task is a common approach\nused in multi-task reinforcement learning. Is there a benefit to\nframing this as an imitation learning problem without using task\nreward signals?\n\nEvery single task needs re-learning: there's no obvious way of\nchanging the order of parts of the task for example: if the agent has\nlearnt to do A-->B-->C, it has to re-learn to do C-->B-->A. (that\nsaid, if I open a cabinet drawer, I can take a fork then a knife, or\ntake a knife then a fork without a problem. But if I learn to play a\nmusic sheet, even just a few bars, I can't directly play it in\nreverse. What is the fundamental difference? and if I learn to play it\nin reverse, there's probably going to be catastrophic forgetting for\nthe normal play)\n\nThe temporal consistency for skill embedding seems only suitable for\nrestricted domains with highly repetitive and ordered action\nsequences. Not really suitable for improvisation and\nproblem-solving. Discuss.\n\nThe distinction between a skill and a policy is confusing. The terms\nskill and policy are used interchangeably in many contexts. I think by\nskills the authors are referring to the latent embedding.\n\nComments on the experiments:\n- for the baselines: any training on D_target alone is unfair as\n  only the target task dataset is used. That said, It makes more sense\n  after the ablation study when comparing to the \"non-retrieval\"\n  study.\n- What is meant by \"struggles to find appropriate goal observations to reach\"\\\n ?\n- Why does FIST perform better for no-microwave?\n- Could the authors provide a comment on why BC-RNN (FT) performs so poorly o\\\nn hardware experiments?\n- There is no real inherent generalizability, the training and\n  execution domains are the same. The no-microwave in D_prior was\n  interesting though, showing that the encoder still made enough sense\n  of this new skill to supervise the policy learning. However, other\n  methods also seem pretty robust against no-microwave.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would like to see more challenging tasks done on the real robot.\n\nPerformance on the tasks in simulation (77%) was not good.\n\nI would like to see a richer notion of what a skill is, ideally a policy mapping low level states to low level actions.\n\nSince information on whether trajectory segments was not clear, it is difficult to assess the role of temporal consistency.\n\nI would focus on real robot results.\n\nPOST REBUTTAL COMMENT\n\nI am still in favor of accepting this paper.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Do a wider range of tasks on the real robot, beyond pick and place.\n\nImprove performance on in the simulation and on the real robot.\n\nLEFTOVERS FROM MAIN REVIEW (MOVED HERE BECAUSE OF STUPID CHARACTER LIMIT)\n\nSmall comments on paper:\n- sub-trajectories -> trajectory segments - throughout the paper\n- 41: pace -> space\n- 73: to a specific types of tasks -> to a specific type of tasks\n- 93: complimentary -> complementary\n- 94: Jargon alert: you are assuming your reader knows what VAE means.\n    - spell out (as in line 134) and provide a reference to make your\n      paper more accessible.\n- 136, 153: Jargon! LSTM has not been explained, spelled out, or had a\n  reference provided.\n- Equation 2: you might explicitly says that mu() is E() or mean(),\nor just use the standard notation E()\n- 171: train the policy the aggregate -> train the policy with the aggregate\n"}}, {"review_id": "OjlHclWQWor", "reviewer": "Reviewer_XCRj", "summary": "This paper presented Skill-Augmented Imitation Learning with prior Retrieval\n(SAILOR), for improving task learning using skill learning from prior data and a small amount of target task data. The two key additions of this paper are: latent skill learning with an auxiliary task that encourages skills that are temporally similar to be close in latent skill space, and data retrieval that compares (in the latent space) the skills of the prior data to the target task dataset and limits the sampling of skills during downstream policy learning to those close to demonstrations on the target task. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well presented, clear, and easy to follow. The contributions are minor practical enhancements of existing approaches, however, the tasks are well selected to show a significant improvement over existing methods. The comparisons seem relevant, and an ablation study shows the effect of each component of the method. Finally, the real-world experiments round off this paper nicely.\n\nI think the main thing that is missing is a deeper evaluation of the two contributions. For example, it would be nice to show that regions in the latent space are indeed close in temporal space, and that behaviours retrieved are indeed close to behaviours from the target task dataset. The ablation study does enough to show these contributions are effective, though it would be a nice addition to see the effect of these methods practically, and also beneficial to the community that may adopt these approaches. This could be as simple as showing samples in each case and discussing the edge cases.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As mentioned, a deeper analysis of the effectiveness of each of the contributions would enhance the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "No figures are referenced in the text. It is not known when the figures back up what is written, which breaks up readability. \n\nThe symbols are confusing in parts. \n- “p” is used as transition probably, initial state distribution, skill decoder, and learned prior.\n- Figure 2 could be clearer. Is the “temporal consistency” block the same as “m_ω”? Is “phi” the skill encoder reported as “q_phi” in the paper? Is “psi” the same as “p_psi”? \n- It might be worth changing the font in figure 2 to match what is in the text to improve first-glance readability. \n\nThe reference list often shows the arxiv URL, but not the publication venue if the paper was accepted. For example: “LASER: Learning a Latent Action Space for Efficient Reinforcement Learning” was accepted at ICRA 2021. At first glance the reader may think many of the references are not from peer-reviewed sources. \n\nSpelling / Grammar / Typos:\n\n“to a specific types of tasks”\n\n“These includes task-conditioned”\n\n“our goal is learn a policy”\n\n“representations of behavior is critical”\n\n“train the policy the aggregated set”\n\n“prior dataset that similar”\n\nVery minor, (to improve readability):\n“In contrast to other skill-based imitation learning (see\nSec. 3) approaches,” -> “In contrast to other skill-based imitation learning approaches (see\nSec. 3),”\n\n“Yet despite their appeal, after experimenting with one such approach, R3M [28], we found that it can sometimes hinder downstream performance.”  -> as this was encountered later in this paper, maybe change the wording to “for the tasks explored in this paper we found that it can sometimes hinder downstream performance.”\n"}}, {"review_id": "8aJl1AtA3ow", "reviewer": "Reviewer_m8W5", "summary": "The work introduces a new method for skill-based imitation learning which can exploit the availability of prior data. It proposes a two-step method. In the first step a \"latent skill spaced\" is learned using a variational objective augmented with a temporal consistency term which enforces the predictability of the temporal distance of sub-trajectories. Second, a policy is learned by behavioral cloning on skill level using both data from the target task at hand as well as a subset of the prior data which is built based on the distances of embedded skills. \nThe resulting method is evaluated on two simulated and one real-world experiment and ablations are performed on one of the settings. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the paper proposes an interesting approach to an important problem in robotics. The main algorithmic contributions, adding the temporal consistency loss and distance-based retrieval of relevant prior data for policy learning are novel and, despite their rather heuristic nature, seem to be effective as shown by the ablation. \n\nIn general. the experimental evaluation shows the method's capabilities in several challenging tasks and indicates the method's potential impact. While the presented experiments are sufficient, for an even better assessment of this impact further ablations on the sizes of prior, retrieval, and target data-set would be useful. \n\nWhile the main ideas are clear and the paper is generally easy to follow, I think there is some rigor missing in the methods section, where the objective is simply stated without context and several details have to be pieced together from the supplement (for more details see issue section)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes an interesting, novel idea and sufficiently demonstrates the potential impact on a set of challenging experiments. While the key ideas are easy to follow and clear, the paper could be improved with more rigor in the method section ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I believe the paper would heavily benefit from more rigor and details in the method section. To be clear, I don't think any of this is \"wrong\" or unreasonable but figuring that out and piecing it together should not be left to the reader but be more clearly stated:\n\n- VAE loss objective (Equation 1): What exactly is lower bounded ($\\sum_{t=0}^{H-1} log p(a_t|o_t)$, what are the independence assumptions/graphical models underlying the generative and inference model and why $-\\mathcal{L}_{VAE}$ is a proper lower bound.\n- From the supplement, it seems a modified version of the bound (MSE instead of gaussian ll and scaled KL term) is actually used (Algorithm 1), again a detail that should be mentioned. \n- Is the policy deterministic or stochastic and which loss is used for training? While that information is available in the appendix, I again think it should be stated here for clarity. In particular, the loss, which seems to be down-weighted for the \"retrieval\" data in the real-world experiment (i.e. there is a factor beta in line 15 of Algorithm 2 in the supplement, which according to table 5 is considerably lower than 1 for the real world data). This seems like an important detail and a method to address the shift between prior and task data. \n\nQuestions / further remarks: \n- Why is the distance between the means used for ranking during retrieval and not something that considers the standard deviation/variance of the inference distribution? \n- Is there any further intuition/ analysis it is sufficient to enforce predictability of the temporal difference (especially as $m_\\omega$ seems to be a rather large network for such a presumably simple task) to get the shown performance gains? Some analysis how the learned representations differ would be helpful \n\nTypos: \n- \"on\" missing in line 171 \n- punctuation missing after equations. "}}]

Author rebuttal:

Decision: Accept (poster)