XSkill: Cross Embodiment Skill Discovery
Authors: Mengda Xu
Zhenjia Xu
Cheng Chi
Manuela Veloso
Shuran Song
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Representation Learning
Cross-Embodiements

Abstract: Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior.  However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross-embodiment representation called skill prototypes purely from unlabeled human and robot manipulation videos, 2) transfers the skill representation to robot actions using conditional diffusion policy, and finally, 3) composes the learned skill to accomplish unseen tasks specified by a human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in a more general and scalable imitation learning framework.

XSkill: Cross Embodiment Skill Discovery
Mengda Xu 1.2, Zhenjia Xu Cheng Chi Manuela Veloso 2.3, Shuran Song
Department of Computer Science, Columbia University
J.P. Morgan Al Research School of Computer Science, Carnegie Mellon University (emeritus)
Abstract:
Human demonstration videos are widely available data source for robot learning and intuitive interface for expressing desired behavior. However, directly extracting reusable robot manipulation skills from unstructured human videos is challenging due to the big embodiment difference and unobserved action parameters. To bridge this embodiment gap, this paper introduces XSkill, an imitation learning framework that 1) discovers a cross- eooos-eebodimmet representa- tion called skill prototypes purely from unlabeled human and robot manipulation videos. transfers the skill representation to robot actions using conditional dif- fusion policy, and finally, 3) composes the learned skill accomplish unseen tasks specified by human prompt video. Our experiments in simulation and real-world environments show that the discovered skill prototypes facilitate both skill transfer and composition for unseen tasks, resulting in more general and scalable imi- tation learning framework. The benchmark, code, and qualitative results are on project website.
Keywords: Manipulation, Representation Learning, Cross-Embodiements
(a) Training skill discovery with cross-embodiment videos
(b) Inference on novel tasks
Human video on new task
Skill Identification
XSkill Space
0o0
Unaligned
Skill Alignment
videos
Skill
Policy
Robot Action
p(als.z)
Current State
Figure 1: Cross Embodiment Skill Discovery. XSkill first learns cross-embodiment skill representation space (XSkill Space the left). During inference, given human demonstration of unseen tasks, XSkill first identifies the human skills by projecting the video demonstration onto learned cross-embodiment skill representation space. The identified skills are then executed by the skill-conditioned visuomotor policy.
1 Introduction
successful imitation learning algorithm from human demonstration is enabled by three critical capabilities: 1) Discover, decomposing the demonstrated task into set of sub-tasks and identifying the common and reusable skills required to accomplish these sub-tasks. 2) Transfer, mapping each of the observed skills to its own embodiment, which is different from that of the demonstrator, 3) Compose, performing novel compositions of the learned skills to accomplish new tasks.
This paper addresses these critical capabilities by decomposing and identifying appropriate "skills" from human demonstration that they transferable robots and composable to perform tasks. We refer to the task "Cross-Embodimen Skill Discovery" and introduce method
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

XSkill for this task. At its core, XSkill learns a shared embedding space for the robot and human skills through self-supervised learning [1, 2, 3, 4, 5J. The algorithm extracts features for unaligned human and robot video sequences, where video clips share similar action effects (i.e., similar skill) should result in closer feature distance. To encourage across-embodiment alignment, we introduce set of learnable skill prototypes through feature clustering. These prototypes act as representative anchors in the continuous embedding space. By force to share the same set of prototypes, could effectively align the skill representations between embodiments.
With the identified cross-embodiment skill prototypes, the robot can then learn a skill-conditioned visuomotor policy that transfers each identified skill to the robot's action space. During inference, the algorithm can one-shot generalize to new tasks using the learned skills, where the new task is defined by a single human demonstration (i.e., prompt video). With the proposed skill alignment transformer, the algorithm robustly align skills in the human video to the robot visual observa- tion, despite the embodiment difference and unexpected execution failures.
Our approach improves upon the direct imitation learning method [6] by decomposing the complex long-horizon tasks into a set of reusable skills (i.c., low-level visuomotor policies), which is much easier to learn and generalizable to new tasks through composition. Meanwhile, our approach differs from existing work on single-embodiment skill discovery [7, 8, 9], which solely relies on on-robot demonstration data. By learning cross- cuss-embbodiment skill prototypes, framework use direct human demonstration, which is more cost-effective and scalable, even for non-expert demonstrators.
In summary, our contributions are as follows:
We formulate the task of cross-embodiment skill discovery, a useful and essential building block for imitation learning. Together with the new cross- emsodimebod dataset in simulation and the real world, we hope to inspire future exploration in this area.
Introducing the first attempt toward this task XSkill that consists of three novel components: 1) A self-supervised representation learning algorithm that discovers a set of share skill prototypes from the unlabeled robot and human videos. 2) A skill-conditioned diffusion policy that trans- lates the observed human demonstration into robot actions. 3) A skill composition framework (with skill alignment transformer) to robustly detect, align and compose the learned skills to accomplish new tasks from a single human demonstration.
Our experiments in simulation and real-world environments show that the discovered skill proto- types facilitate both skill transfer and composition for unseen tasks, resulting in more general and scalable imitation learning framework. The dataset and code will be publicly available.
2 Related Work
Robot Skill Discovery. A large body of works have been proposed for discovering robot skill via option frameworks [10, 7, 11, 12, 13, 14] through the lens of mutual information [15, 8, 16, 17, 18, 19, 20]. Most of these works require interacting with the environment and yield high sample complexity. To ease the sample complexity, the other line of works [9, 21, 22, 23, 24] have explored discover skill directly from robot demonstration data. The majority of those prior works require physical state or robot action trajectories. BUDS [25] eases this requirement by discovering skills through raw RGB data. Unlike these prior works which discover skills only for a single embodiment, XSkill explores skill discovery in a cross-embodiment setting.
Imitation learning. Learning robot behavior from demonstration data is a longstanding chal- lenge [6]. Prior works on imitation learning have shown promising results on real-world robot manipulation task through explicit policy [26, 27, 28, 29, 30, 31, 32, 33], implicit policy [34] or diffusion model [35, 36, 37]. Our works utilize the hierarchical imitation learning framework [38, 39, 40, 41, 42] to transfer the discovered skills through skill-conditioned diffusion p35icy[35]. XSkill can be categorized as one-shot imitation learning [43]. Most of the prior works imitate from the same-embodiment demonstration [44] or from different embodiment [45, 46] but require task

---

Human & Robot Videos
v
temporal
Softmax
Augmentation
Predicted probability
Learnable Skill
Sample
Temporal Skill Encoder
...
Prototypes (Linear Layer)
Cross
(Transformer)
prototype
Entropy
Augmentation
Sinkhorn
z'
Clustering
temporal
Target
XSkill Discover: At each training iteration, batch of video are sampled from the same embodiment dataset. Each video of is augmented into two versions and encoded using temporal encader fiempocat. The learnable skill prototypes fpeceapps are implemented a normalized linear layer without bias. Both fremporal and Sprocaygee are trained jointly to minimize the CorssEntorpy loss between the predicted and target the probability of skill prototypes. Sinkhorn regularization is applied to the target probability. ensuring all prototypes are used for each batch (same embodiment), thereby encouraging prototype sharing across embodiments.
Figure 2:
label during the training. In contrast, XSkill does not require any task label or correspondence between embodiments during training to accomplish one-shot imitation from human demonstration.
Learning from human video. A number of works [47, 48, 49, 50, 51] have studied leveraging human videos to learn robotic policy. One approach is construct reward functions from human video through domain translation [48, 52, 53, 54, 55, 56), video classifier [50, 57], or state represen- tation learning [58, 49, 59]. Despite showing promising results in learning from cross-embodiment demonstration, most these works require reinforcement learning (RL) to learn policy based on the constructed reward functions, which is expensive to deploy in the real world. In contrast to the majority of these works, our method does not involve RL the loop and focuses on one-shot im- itation from human videos. Bahl et al. [60] proposed initialize policy through human prior but still requires interaction with the environment to improve the policy. Our work is related to Yu al. [51, 61], which explored one-shot imitation through meta-learning. Unlike meta-learning approach, our method does not require any task pairing information during the training. More recently, Mim- iePlay [62] proposed leveraging human videos to learn a cross-embodiment plan latent space. While MimicPlay aims to minimize robot demo collection, our emphasis is on learning a cross- eoosdiembod representation that allows us to reduce the reliance on robot demonstrations during inference.
Approach
The XSkill framework consists of three phases: Discover $3.1, Transfer $3.2, and Compose $3.3 that uses three different data sources. In the discover phase, the algorithm has access to a human demonstration dataset D' and robot teleoperation dataset D to discover a cross-embodiment skill representation space Z. This space, along with K common learnable skill prototypes, is learned through self-supervised learning. Both datasets are unsegmented and unaligned and each video in the dataset performs a subset of N skills. In the transfer phase, the algorithm uses the robot teleoperation dataset Dr learn the skill-conditioned visuomotor policy P(a)s,=), where and includes both robot proprioception and visual observation o. In the Compose phase, the algorithm takes input single human prompt video "prompt for new task that requires an unseen composition of skills complete. From this video prompt, the algorithm first identifies the order of skills used in the prompt and then composes the skills using the learned policy P(als, 2).
Z
3.1 Discover: Learning Shared Skill Prototypes
As the first step, XSkill aims discover skills in self-supervised manner such that the learned visual representation of the same skills executed by different embodiments can be close in the cross-embodiment skill representation space Z. Off-the-shelf vision representations are often insuf- ficient since they are often sensitive to the visual appearance of the agent or environment. Instead, we want the learned skill representation to focus on the underlying skills being performed. To achieve this goal, XSkill introduces two key ideas:
Learning set of shared skill prototypes through soft-assignment clustering. These discrete prototypes act representative anchors in the continuous embedding space. By forcing the use of shared prototypes, effectively align skill representations between embodiments.
3

---

Regularizing the training process using Sinkhorn-Knope clustering [63, 1] within single- embodiment batches. Together, they ensure all prototypes are used for each batch (all from the same embodiment). This design avoids the degeneration case where different embodiment maps to different prototypes, thereby ensuring prototype sharing.
Skill representations. XSkill extracts skill representations using human and robot videos from D and Dr. mapping them into shared representation space Z. To mitigate variations in exe- cution speed across different embodiments, we sample M frames uniformly from each video V, and construct video clips { ]=0 using a moving window of length L. Then, we extract the skill representation 2ij = fiempprai from each video clip with a temporal skill encoder consisting of a vision backbone and a transformer encoder [64]. We append a learnable representation token [65, 66] into the sequence to better capture the motion across frames.
Skills as Prototypes. Once the skill representations are obtained from demonstration videos, XSkill maps representations from all embodiments to a set of K skill prototypes {cx] where each is a learnable vector. The skill prototypes are implemented as a normalized linear layer fponotype without bias. This mapping is accomplished through a supervseee learning framework similar to SWAV [1]. The XSkill starts by augmenting the video clip Maj using a randomly selected transforma- tion before feeding into fremporal (e.g., random crop). Subsequently, XSkill projects the normalized onto the set of learnable skill prototypes fcx |k= The probability Pij of skills being executed in the given video clip Uiy is predicted by applying the Softmax function. The target distribution giy is obtained from the other augmented version of the same video clip. The target probability, instead of applying the Softmax function over projection, is obtained by rumning on- line clustering Sinkhorn- Sinkhorn-Knopp algorithm, which describe later in the paper. Both fiemporal and fposssype are trained jointly to minimize the CorssEntropy loss between the predicted Pij and target
Asg representation TT=;īlz
B
M
K
qu skill prototypes distributions: Epeototype
(k) log (k) where B is batch size. gig
Learning Aligned Skill Representation. To ensure that the skill representation focuses on underly- ing skills rather than embodiment and is aligned across embodiments, XSkill employs a combination of data sampling and entropy regularization during the clustering process in training. In each train- ing iteration, XSkill samples video clips from the same embodiment and constructs a batch. This batch then fed into the framework shown in Fig. 2, where clustering performed on the fea- tures from the same embodiment, disregarding the embodiment differences. A significant challenge arises as the skill embedding space might be segmented by embodiment, with skill representations for each embodiment occupying distinct regions in the embedding space. To address this issue, goal is to enable the skill representation for each embodiment to fully utilize the entire embedding space. By allowing different embodiments to share each region in the space, the clustering algorithm is compelled to group representations based on the effect of the skill, resulting in an aligned skill representation space. We approach this as an entropy-regularinn efficiently solved using the Sinkhorn clustering algorithm. Further details and pseudocode for the Sinkhorn-Knopp can be found in the appendix.
clustering problem, which can be
Time Contrastive Learning. XSkill utilizes a time contrastive loss [67, 68] in order to encap- sulate the temporal effects of skills within video demonstrations. It posits that skill prototype probabilities should be similar for video clips closer in time and dissimilar for those farther apart. This achieved by establishing a positive window Wp and a negative window Wn- For given clip Vir at time x in video V,, positive sample Viy is chosen within wp from time x, and negative sample Diz is selected outside w,, XSkill minimizes the following InfoNCE loss [69]] B
exp(S(pia Lien E log exp (S(ppe:PPuu// -xx (// wherein S is the measure of similarity, which is implemented as dot product in XSkill and Tien is the temperature. Here, Pize Piys Pix. represent the skill prototype probabilities for clips Ver, Vigs and Viz, respectively.
3.2 Transfer: Skill conditioned imitation learning
To transfer skill representations into concrete robot actions, train Akill-oonditione' visuomotor policy using imitation learning, In theory, any imitation learning policy can be used with the XSkill

---

Robot Observation
Human Prompt Video for Unseen Task
(+) Input
Observation Encoder
Temporal Encoder (Pretrained)
Current Skill e,
Sequence of Skills from
(b)
Skill Alignment
Observation Feature dy
video
Skill Alignment Transformer
Current State 9 w/ Proprioccption
Skill-Conditioned Diffusion Policy p(a,)s, 4
Robot Action a,
Imitation
Poliey
Figure 3: Transfer Composition: During inference, human demonstration a new task is given, XSkill first extracts sequence of skills, which can be viewed as a high-level task plan. However, this plan is not immediately aligned with robot execution speed due to the embodiment gap. Therefore we need to align the plan based the robot's current observation, which achieved by the Skill Alignment Transformer. The inferred skills are then passed into skill-conditioned diffusion policy to get the robot's actions.
framework. practice, we prefer to use diffusion policy as it achieves Maat-off-the-at results in many existing benchmarks. More specifically, our approach builds upon diffusion policy [35] which Denoising Diffusion Probabilistic Models (DDPMs) [70] to represent the multimodal action distributions from human teleoperation demonstration. Diffusion policy has been shown to be stable to train and work well with a small amount of data, both are essential for our tasks.
Our diffusion-basel imitation learning policy p(at|*t; =x) is trained with robot teleoperation dataset D", where a, denotes an action-sed sequnnee {at, d++L} } of length starting from state 8t. The diffusion policy takes skill representation 2, and state 8, which includes robot proprioception and visual observation or input and produces action-sequence ag. The skill representation 2, is computed with trained fiempural using V {or, 0.+1..., 0t+L we described in $3.1.
3.3 Compose: Performing unseen task from one-shot human prompt video
Once skills have been discovered and transferred into robot manipulations through imitation learn- ing, our object is to compose the skills to solve unseen tasks based on human prompt video which contains demonstration by a human on how to complete an unseen task. To do so, XSkill first maps a human prompt video with length Prompp into the cross-embodiment skill representation space Z using learned fumppral. This generates a sequence of skill representation for the demonstrated task, denoted {22) t= which is essentially task execution plan. The robot complete the task by sequentially executing the skills in the plan by querying the skill-conditioned diffusion policy.
However, directly following the skill sequence for execution often results in a fragile system that sensitive to unexpected failures or speed mismatch. For example, if fails to open a light, needs retry the skill succeed. If the robot sequentially executes the will proceed execute the next skill without correcting the error. Therefore improve the system's robustness, introduce Skill Alignment Transformer (SAT) that is described below.
Skill Alignment Transformer (SAT). The Skill Alignment Transformer, denoted as Φ(2 (o,, Z), aligns the robot with the intended skill execution based on its current state. The key idea that by the skill sequence ž, the robot can determine which part of the skills has already been executed and infer the most likely skill be executed next. The cluster structure created by discrete prototypes aids in facilitating skill identification by SAT during inference time. This alignment process allows the robot synchronize its task execution with the demonstrated task in the human prompt, thereby minimizing discrepancies arising from variations in execution speed between robots and humans and offering robustness against execution failures.
analyzing the current state and comparing
As illustrated in Fig. 3, SAT regards each skill in the skill sequence غ as a skill token. employs a state encoder, Samte-nncoder to convert the robot's current visual observation into a state token. The skill and state tokens with position encoding, are then passed into transformer encoder predict the skill that needs to be executed next. Within the transformer, the state token can attend to each
5

---

skill token to determine whether the skill has been executed given the current state information. For example, if the light is already open in the current state, the skill to open the light should not be considered the next skill to be executed.
To train SAT (0, 2), we sample a full trajectory from the robot teleoperation dataset and extract the skill sequence 2. This is achieved by passing robot trajectory video clips {vr {vr) where ve {oe, O.+1..., 1. through the temporal skill encoder, yielding {2r { The length of the sampled trajectory is represented by T. Next, a time index is chosen randomly within the range [0, T Our system predicts & using the skill sequence å and visual observation Or as inputs to SAT, We optimize the model by minimizing the mean square error between & and the actual Et.
Evaluation
Environment. We test XSkill both simulated and real-world environments:
Franka Kitchen: is simulated kitchen environment [71] that includes 7 sub-tasks and is accom- panied by 580 robot demonstration trajectories. To create a cross- cross-embodiment demonstration, we construct a sphere agent that is visually significantly different from the original robot. To further increase the domain gap, we sub-sample sphere agent demonstrations to emulate the execution speed differences. During the inference, the robot must complete an unseen composition of sub- tasks after viewing a prompt video from the sphere agent demonstration.
Realworld Kitchen: is new benchmark we introduce to evaluate algorithm performance on physical robot hardware. The dataset contains four sub-tasks, namely, opening the oven, grasping a cloth, closing a drawer, and turning on a light. We have recorded 175 human demonstrations and 175 teleoperation demonstrations. Each demonstration completes three sub-tasks in a randomly determined order. During inference, the robot is required to complete an unseen composition of either three or four sub-tasks after observing a prompt video taken from a human demonstration.
Baselines. We compare XSkill with the following baselines:
GCD Policy: Instead of using skill-conditioned policy, we compare to a goal-conditioned diffu- sion policy [st 9t). where the goal image 9, is the image in prompt video H steps after the current time f after alignment. The alignment is done using nearest-neighhor matching between robot observation and prompt video in the embedding space, where the embedding is trained jointly with the policy.
GCD Policy w. TCN: Same the GCD Policy above but replacing the video encoder with pre-trained Cone-Conrasttive Network (TCN)[67).
XSkill w. NN-composition: XSkill removing skill aligmment transformer. Instead, find the alignment using the nearest neighbor image between the robot observation and the prompt video.
The image embedding extracted using the same encoder fiemporal as XSkill.
XSkill w.o proto. loss: XSkill removing prototype loss Eprotoype in the discover phase.
Implementation Details. We set the number of prototypes K as 128 in the simulated environment and 32 prototypes in the real-world environment. The video clip length L and uniform sample frames M are set as 8 and 100 for both simulated and real-world kitchens. The ablation study on K, time contrastive loss, and more implementation details can be found in the supplementary material.
Evaluation protocol. During inference, the robot is required to accomplish the sub-tasks in
(Sim) Franka Kitchen
Real Kitchen Env
the same order demonstrated in the prompt video. The performance of XSkill and all base- line methods is evaluated based on both sub- task completion and order of completion. the robot executes an undemonstrated sub-task, the episode ends. The evaluation metric is denoted
Figure 4: Evaluation Environments.
as number of sub-tasks completed In the simulation, each method is trained using three distinct seeds and number of lotal sub-tasks
tested under 32 unique initial environment conditions during the inference. In the real-world kitchen, each task is assessed 10 times under varying initial environment conditions during the inference.

---

Turn on Light
Close Drawer
Human and Robot execute the same task;
Turn on Light
Cirasp Cloth
Onen Oven
Open Oven
Grasp Cloth
Human Video
14,80s
+
Piohahliity
Robot
Huming
Robot
(a) I-SNE Visualization Skill Space
(b) Skill Prototype Distribution Over Time
Figure 5: XSkill embedding, (a) We utilize t-SNE visualization showcase the alignment of skill representa- tions among various embodiments when contact with the same object. (b) We present projected prototypes for both humans and robots executing identical tasks. XSkill achicves efficient alignment of representations, not just during physical contact, but also during the transition between manipulating different objects.
4.1 Key Findings
XSkill can learn cross-embodiment skill representation. [XSkill] learns cross-embodimett skill representation by effectively extracting reusable skills from demonstrations including both in- contact manipulation with various objects and also the transitions between them. In Fig. 5(a), we visualize these learned skill representations using t-SNE, where the skills executed by the human and robot to manipulate the same object are grouped together and clearly separated from the others. Additionally, we visualize the projected prototypes for human and robot completion of the same task in Fig. 5(b). Despite differences in execution speed (humans execute x 2 faster), [XSkill] can decompose the task into meaningful and aligned skill prototypes for both in-contact manipulations and transitions between them. Consequently, the performance of [XSkill] with cross-embodiment prompts only drops around 5%, compared to using the same embodiment prompt (Tab. 2 seen tasks).
XSkill can generalize the imitation pol- icy to unseen tasks. [XSkill] achieves 70.2% and 60% success (Tab. & on unseen tasks with cross-embodiment prompts in simulated and real-world en- vironments, which outperforms all base- lines. As shown in Fig. 6 (a), [XSkill] capable of decomposing unseen tasks into sequences of previously seen skill abstractions which can be executable by the learned skill- conditional imitation policy. As result, [XSkill] enables one-shot imitation learning through skill decomposition and re-composition. The performance of [XSkill] slightly drops in the real world due novel transition dynamics presented in prompts and scarcity of collected robot data. For instance, the robot struggles to complete tasks involving grasping the cloth followed by closing the drawer, since no such transition dynamics are present in the collected robot teleoperation dataset. In summary, [XSkill] demonstrates promising task generalization, but its efficacy is still limited by the diversity the robot teleoperation data.
Table Simulation Result (%)
Same
Cross Embodiment
Avg
Execution speed
x 1 x 1.3
1.5
GCD Policy
91.4
0.00
0.00
0.00
22.8
GCD Policy TCN
2.50
3.55
2.00
1.25
2.32
XSkill w. NN-compose
93.7
61.2
23.4
15.2
48.4
XSkill w.o proto. loss
80.1
56.3
12.5
3.75
38.2
XSkill
95.8
89.4
83.7
70.2
84.8
Table 2: Real-world Result (%)
Skill prototypes are essential for cross- embodiment learning. To assess the im-
3 Subtasks
4 Subtasks Cross
Avg
Same
Cross
portance of shared skill prototypes, we
Seen Unseen Seen Unseen
Unseen
compare our approach with [XSkill w.o
GCD policy 68.3
53.3
0.00
0.00
0.00
24.3
proto. loss] in simulation. [XSkill] out-
GCD w. TCN
25.0
22.2
26.7
23.3
15.6
22.6
performs the baseline significantly with
XSkill
86.7
80.0
81.7
76.7
60.0
77.0
cross-embodiment prompts. Further, we observe that the performance of [XSkill w.o proto. loss]
deteriorates rapidly when the cross-embodiment agent operates at a higher speed. These results sug-
gest that skill prototypes not only facilitate the learning of ceaphology-invarian representations but also avoid lcarning speed-sensitive one.
7

---

Close Drawer
Approacc
Uern
Graup Gleth
Apprach
Open
Human
Skill
Identification
(b) Skill
Timeline
19.25
Aligument
(e) Perturbation
(c)
Skill
Prediction
(d) One-shot Imitation on Novel Task
(f) Replan after Perturbation
Figure 6: Execution novel task and robustness to perturbation. (a) XSkill analyzes human video of timestep (represented by distinct colors). (c) The robot leverages this analysis to predict the appropriate skill based on the current observation and subsequently executes the corre- sponding skill. Skill aligmment (b) is critical to handle execution speed difference caused by cross emhodiment. (e) With appropriate skill conditions at each step, the robot achieves one-shot imitation of the novel task. (e) Deliberately introducing a perturbation, human manually turns off the light. The robot accurately predicts the necessary skills adaptively replans the execution successfully reach the goal state once again. Please check out the project website for videos
a novel task, identifying skills
SAT can align skills based on task progress. [XSkill] outperforms [XSkill NN-composition] by more than 50% with cross eross-embodiment prompt. The performance of NN- NN-composition significantly declines when the cross-embodiment agent executes at a much faster speed (Tab. 1). We down- sample cross-embodiment demonstrations and also prompt videos with different ratio. For instance, a ratio of x1.5 emulates that human execute x1.5 faster than the robot the real world. This can be attributed to two main factors. Firstly, relying solely on visual representation is prone to distraction due to morphological differences. Second, when the demonstration speed in the prompt videos is much faster than the robot's execution speed, certain states might not be captured in the prompt video. As result, the retrieved skills can become inaccurate and unstable. In contrast, as illustrated in Fig. 6, the SAT enhances the robustness of [XSkill] to the demonstration speed and enables adaptive adjustment of skills based on the current state of the robot and the task progress.
Sequential input benefits cross-embodiment learning. Unlike (GCD Policy w. TCN], which relies on TCN encoding for single images, [XSkill] utilizes sequential image input generate representa- tions. In the rros-embodiment scenario, [GCD Policy TCNJ outperforms vanilla [GCD Policy], indicating partial bridging of the embodiment gap through TCN embedding. However, [GCD Pol- icy w. TCNJ cannot follow the sub-tasks completion order in the prompt video, resulting in lower evaluation score. This suggests that TCN embedding does not completely align the representations.
4.2 Limitation and Future Work
One limitation of XSkill is the requirement of specifying the number of prototypes as an input to the algorithm. While our ablation study demonstrates that XSkill is not highly sensitive to this number, fine-tuning hyperparameters may necessary for optimal performance based on the dataset and its use. In addition, XSkill doesn't 't demand labeled correspondence between human and robot datasets. However, current benchmark only comprises videos from the same laboratory camera setup. Future rescarch could investigate the applicability of our approach in more diverse camera setups and environments, leveraging readily available YouTube videos and multi-environman datasets [72].
5 Conclusion
We introduce XSkill for the task of cross-embodiment skill discovery. This framework extracts common manipulation skills from unstructured human, robot videos in a way that is transferrable to robots and composable to perform new tasks. Extensive experiments in simulation and real- world demonstrate that XSkill improves upon the direct behavior cloning method especially complex long-horizon tasks. Moreover, by leveraging cross-embodiment skill prototypes, XSkill can directly leverage non-expert human demonstration for new tasks definition, making the framework much more cost-effective and scalable.

---

Acknowledgments
We would like thank Zeyi Liu, Huy Ha, Mandi Zhao, Samir Yitzhak Gadre and Dominik Bauer for their helpful feedback and fruitful discussions.
Mengda Xu's work supported by JPMorgan Chase & Co. This paper was prepared for informa- tion purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("JP Morgan"), and is not product of the Research Department of JP Morgan, JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as invest- ment research or investment advice, recommendation, offer solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful. This work was supported in part by NSF Award #21 #2143601, #2037101, and #2132519. We would like to thank Google for the UR5 robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] M. Caron, I. Misra, Mairal, Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-99244, 2020.
[2] Y. M. Asano, Rupprecht, and A. Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.05371, 2019.
[3] M. A. Bautista, A. Sanakoyeu, E. Tikhoncheva, and B. Ommer. Cliquecnn: Deep unsupervised exemplar learning. Advances in Neural Information Processing Systems, 29, 2016.
[4] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference computer vision (ECCV), pages 132-149, 2018.
[5] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical representations. 2021.
[6] B. Argall, S. Chernova, M. M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics Auton. Syst., 57:469 483, 2009.
[7] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. In International Con- ference on Learning Representations, 2020. URL https::/p/// openrevieneeeeeeeeeeeeeeeee net /formm?
BlggiiiW
[8] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. ArXiv, abss/1802.06070 2018.
[9] A. Sharma, S. S. Gu, S. Levine, Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. ArXiv, bbss/1907.01677 2019.
[10] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112:181-211, 1999.
[11] R. Fox, S. Krishnan, I. Stoica, and K. Goldberg. Multi-level discovery of deep options. ArXiv, abs/1703.00894 2017.
[12] G. D. Konidaris and A. G. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In NIPS, 2009.

---

[13] V. C. V. Kumar, S. Ha, and C. K. Liu. Expanding motor skills using relay networks. In Conference on Robot Learning, 2018.
[14] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms, 2018.
[15] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. ArXiv, abs/1611.07507, 2016.
[16] K. Hausman, T. Springenberg, Z. Wang, N. M. O. Heess, and M. A. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
[17] Shankar and Gupta. Learning robot skills with temporal variational inference. In Pro- ceedings of (ICML) International Conference on Machine Learning, pages 8624 8633, July 2020.
(18) I.. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient ploration via state marginal matching, 2020. URL https: id=HklaleHPPS.
[19] H. Liu and P. Abbeel. Aps: Active pretraining with successor features. In International Con- ference on Machine Learning, 2021.
[20] M. Laskin, H. Liu, X. B. Peng, D. Yarats, A. Rajeswaran, and P. Abbeel. Un- supervised reinforcement learning with contrastive intrinsic control. 35:34478-34491, 2022. URL fft1://6r22222000000000222222222
[21] D. Tanneberg, K. Ploeger, E, Rueckert, and J. Peters. SKID RAW: skill discovery from raw trajectories. IEEE Robotics and Automation Letters, 6(3):4696-47000 4703, 2021. doi:10. 109/LA. 2021.3068891.
[22] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference Learning Representations, 2021. URL https::t///o /ttp::////eeeeeeeeeeeee net. wtt.:oortrrt/
forum? wwt.:e//oorrmi/eeesv------- d==sss-woseess
[23] K. Pertsch, Y. Lee, and J. J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
[24] M. Xu, M. Veloso, and S. Song. ASPire: Adaptive skill priors for reinforcement learning. In Advances in Neural Information Processing Systems, 2022. URL https: //tp://w/pp.....ee//e/ net/ forum? d= r 028 9wAUa.
[25] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for lori-norizon robot manipulation. IEEE Robotics and Automation Letters, 7:4126-4133 2021.
[26] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkamni L. Fei-Fei, S. Savarese, Y. Zhu, and R. Mart' in-Mart' in. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021.
[27] R. Florence, L. Manuelli, and R. Tedrake, Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5:492 4992 2019.
[28] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task ma- nipulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference Robotics and Automation (ICRA), pages 3758-3763 2017.
10

---

[29] A. Zeng, P. R. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, 2020.
[30] Zhang, Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1-8, 2017.
[31] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with stone. In Thirty-Sixhh Conference on Neural Information Processing Systems, 2022. URL wwwhthtt///// http://ppeeevvveeeeeeeee ew wtte//eeoorreeee wtwp:m/mddaaararrrroo
[32] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. VIOLA: Object-centric imitation learning for vision- based robot manipulation. In 6th Anmual Conference on Robot Learning, 2022. URL https:
//openreview wtt.eeewweeeeeeeeeeeeeeeeeeeeeee forumm www.rromidddBBBe htt.:mmmmdddhhhhfPPeeee..
[33] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, Hsu, Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Ju- lian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-
I: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2204.0169)) 2022.
[34] Florence, C. Lynch, A. Zeng. O. Ramirez, A. Wahid, L. Downs, A. Wong. J. Lee, I. Mor- datch, and Tompson, Implicit behavioral cloning. Conference on Robot Learning (CoRL), 2021.
[35] C. Chi, S. Feng, Y. Du, 7. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[36] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https: http::://operrvview net/
[37] M. Reuss, M. X. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies. ArXiv, abs/2304.02532, 2023.
[38] C. Lynch, M. Khansari. T. Xiao, V. Kumar, Tompson, S. Levine, and Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2019.
[39] H. M. Le, N. Jiang, A. Agarwal, M. Dudík, Y. Yue, and H. D. I. au2. Hierarchical imitation and reinforcement learning, 2018
[40] A. Mandlekar, F. Ramos, B. Boots, I. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforce- ment without interaction at scale for learning control from offline robot manipulation data. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414 2019.
[41] A. Mandlekar, D. Xu, Martin-Martín, Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations, ArXiv, abs/2003.06085, 2020.
[42] K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and Posner. Taco: Learning task decompo- sition via temporal alignment for control. In International Conference Machine Learning, 2018.
11

---

[43] Y. Duan, M. Andrychowicz, B. C. Stadie, J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba. One-shot imitation learning. In NIPS, 2017.
[44] Z. Mandi, F. Liu, Lee, and P. Abbeel. Towards generalizable one-shot visual imitation learning. 2022 International Conference Robotics and Automation (ICRA), pages 2434- 2444, 2021.
[45] C. Finn, Yu, т. Zhang, Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. ArXiv, abs/1709.04905, 2017.
[46] S. Dasari and A. K. Gupta. Transformers for one-shot visual imitation. ArXiv, abs/201 2020.
[47] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik, Masked visual pre-training for motor control. ArXiv, abs/2203.06173, 2022.
[48] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation, 2018.
[49] K. Zakka, A. Zeng, P. R. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross- embodiment inverse reinforcement learning. In Conference on Robot Learning. 2021.
[50] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research,
2021.
[51] T. Yu, C. Finn, A. Xie, Dasari, T. Zhang, P. Abbeel, and S. Levine. One-shot imitation from observing humans via domain-adaptive meta-learning, 2018
[52] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos, 2020.
[53] Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller, 2019.
[54] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg. Learning by watching: Physical imitation manipulation skills from human videos. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834, 2021.
[55] K. Schmeckpeper, O. Rybkin, K. Danillidis, S. Levine, and C. Finn. Reinforcement learning with videos: Combining offline observations with interaction. CoRR, abs/2011.06500 2020.
URL ht https:://// //aaixiivv..//..///aaaa//aaaavva wwwrxvivor//...... prgg/bs// aaas//011... 201110011..0 06507.
[56] Sermanet, K. Xu, and S. Levine. Unsupervised percettaal rewards for imitation learning. CoRR, abs/1612.06699, 2016. URL
/1l2a. 06699.
[57] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos. ArXiv, 1tss/2003.168) 2021.
[58] M. Sieb, X. Zhou, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In Conference on Robot Learning, 2019.
[59] Kumar, J. Zamora, Hansen, Jangir, and Wang. Graph inverse reinforcement learning from diverse videos. In Conference Robot Learning, 2022.
[60] S. Bahl, A. Gupta, and D. Pathak. 01122077.04400. 2022.
Human-to-robot imitation in the wild. ArXiv,
[61] T. Yu, P. Abbeel, S. Levine, and C. Finn. One-shot hierarchical imitation learning of compound visuomotor tasks, 2018.
12

---

[62] C. Wang, L. J. Fan, J. Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mim- ieplay: Long-horizon imitation learning by watching human play. ArXiv, abs/2302.12422 2023.
[63] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[64] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, E.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[65] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[66] Kostrikov, D. Yarats, and Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[67] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video, 2018.
[68] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022.
[69] A. v. d. Oord, Y. and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.0378, 2018.
[70] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1239, 2020.
[71] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, 2019.
[72] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets, 2021.
Appendix
A.1
Generalization to unseen transition
To evaluate the generalization capacity of XSkill, we conducted experiments within the simulation environment to assess its performance with unseen transitions. We designed two levels of task difficulty by excluding transitions used in the inference task from the training dataset. In Level 1, 25% the transitions were removed, and in Level 2, 30% were removed. Notably, the main paper's experiment in the simulation was already conducted in the scenario involving the removal of 25% of transitions. The complete experiment results are summarized in Table Al.
Table Al: Generalization Study Result (%)
Level I
Level 2 Embodiment
Same
Cross Embodiment
Overall
Same x1
Cross
Overall
Execution speed
X
x 1
x 1.5
x
x
XSkill K=128
95.8
89.4
70.2
85.1
62.5
55.0
47.5
55.0
XSkill K=256
98.3
86.6
76.7
87.2
91.3
81.3
61.3
77.9
XSkill K=512
97.5
90.7
71.8
86.7
87.5
84.4
67.5
79.8
The results of these experiments demonstrate XSkill's capacity to generalize to new tasks involv- ing previously unseen skill compositions and transitions. XSkill achieved success rates of 90.7%
13

---

and 84.4% in Level I and Level tasks, respectively, when using the same- speed cross-embodiment prompt, and 71.8% and 67.5% for the cross-embodiment prompt x 1.5 speed. Further, we observe that increasing the number of prototypes K enhanced the generalization potential. While various K choices demonstrated comparable performance in Level tasks, larger values of K (256, 512) significantly outperformed smaller ones the more challenging Level 2 tasks, which lacked 50% of the inference task transitions from the training data. We hypothesize that augmenting K en- hances the granularity of the representation space, thereby facilitating better generalization through interpolation.
A.2 Ablation Study
A.2.1 Number of skill prototypes
We performed an ablation study to assess the impact of the number of skill prototypes (K) in our XSkill framework within the simulated Franka Kitchen environment. We tested K values of 32, 128, 256, and 512, with the results for K 128 reported in our main paper. The outcome of this ablation study can be found in Tab. A2.
We observed that increasing the number of skill prototypes (K) to 256 or 512 did not degrade the performance of XSkill. However, reducing K did impact performance adversely. We hypothesize that a smaller K value (i.e., 32) may limit the representation capacity of the skill space, as all skill representations are enforced to map around one of the skill prototypes. This could potentially force distinct skills map around the same prototype. resulting diminished manipulation performance. On the contrary, larger K value doesn't hinder performance; in fact, increasing K might augments the granularity of the representation space, allowing unique skills to have distinct representations within this space.
These results suggest that the performance of our framework is not significantly affected by the choice of K. We believe this is due to the fact that the projected skill prototypes are not directly inputted into the imitation learning policy P(at St, zt) and SAT (zz(,, ot). Instead, we utilize the continuous skill representation prior projection. This choice allows for greater flexibility and granularity in representing skills, making the specific choice of K less crucial. Therefore, while fine-tuning K may still be necessary for optimal results in certain environments (e.g., a simulated Franka Kitchen with seven sub-tasks requiring large K as opposed to real-world kitchen with four sub-tasks where a K value of 32 may suffice), our framework demonstrates robustness against variations in the number of skill prototypes.
Table A2: Ablation: Number of K (%)
Same X 1
Cross
Embodiment
Execution speed
x I
x 1.5
XSkill K =32
91.6
67.5
48.7
XSkill K 128
95.8
89.4
70.2
XSkill K =2256
98.3
86.6
76.7
XSkill K =512
97.5
90.7
71.8
A.2.2 Time contrastive loss
In order to demonstrate the significance of time contrastive loss, we conduct a comparative study with [XSkill w.o TC loss] using simulations. The results, presented in Tab. A3, clearly show a significant drop in performance for [XSkill w.o TC loss] compared to [XSkill], even under the same- embodiment setting. This empirical evidence underscores the vital role of time contrastive loss in enabling our representation effectively capture the temporal effects of skills. Consequently, the leamed skill representation with time contrastive loss is beneficial for downstream manipulation tasks.
14

---

Table A3: Ablation: Time contrastive loss (%)
Same
Cross Embodiment
Avg
Exccution speed
x
x
1
x 1.3 x 1.5
/
XSkill w.o TC loss
3.75
2.25
2.25
1,25
2.38
XSkill
95.8
89,4
83.7
70.2
84.8
A.3 Additional Experiment Results
We present the performance results of XSkill for each task using cross-embodiment prompts during inference for the real-world environment in Table A4. When combined with the additional study on generalization in Appendix A1, primary limitation for achieving generalization in real-world environments becomes apparent: the diversity present in robot teleoperation data.
Our observations indicate that XSkill is capable of generalizing to previously unseen transitions in simulations due to the sufficient and diverse nature of the collected data. This allows for effective interpolation and generalization. However, it's important to note that the data collected for the sub- task Drawer in the real-world environment lacks multi-modal properties, as illustrated in Figure in the main paper. As a consequence, XSkill struggles to extend its capabilities to solve unseen transitions involving the Drawer sub-task.
Table A4: XSkill Inference Task Per Task Results (%)
Inference Task
Cross Embodiment
Oven, Draw, Cloth
80.0
Draw, Cloth, Oven
73.3
Oven, Light, Cloth, Draw
75.0
Draw, Cloth, Light, Oven
90.0
Draw, Oven, Cloth, Light
25.0
Draw, Light, Cloth, Oven
50.0
A.4 Implementation Details
We have presented summary of the three phases, namely Discover, Transfer, and Compose, in pseudocode. The pscudocode for each phase is provided in Algorithm 1, 2, and 3, respectively.
Algorithm Cross-embodiment Skill Discovery
1: Input: K: Number of skill prototypes. Dh: Human demonstration dataset. D Robot teleop-
eration dataset
2: Require: T: Random augmentation operation. Matrix multiplication
3: Require: /temporat: Temporal skill encoder. C C'ece,..... [[i,..., cx]:
Skill prototypes
4: while not converge do
5: Sample a batch of video clips from D or D
6: UA T(v) and un ((v)
Compute augmentations of u:
7:
femporal (VA) and = fiemporal (UB)
> Compute skill representations
8:
SA mm(22, 2A, C) and sg mm 2B. C)
b Compute projection
9:
Softmax and рв Softmax(
b Predict skill prototypes probability
qA Sinkhorn(ss) and QB Sinkhorn(sA)
D Compute target probability
11:
Epotto (CrossEntrropy(ee qA) CrossEntropy Sample positive and negative video clips for UA: y
meg qn))
& Compute prototype loss
12:
> Or for 2b
Compute associated skill prototypes probability v .PA
Follow line to line
14:
Lim = InfoNc((P.. (P pos .PA peg)
Compute time contrastive loss
Latisooeery Eprato + Lion
Compute skill discovery loss Update models
16:
Update Siemporal and C
end while
15

---

Algorithm 2 Cross-embodiment Skill Transfer
1: Input: DD: Robot teleoperation dataset.
2: Require: /temporat: Learned temporal skill encoder (freeze).
3: Require: Φ: Skill Alignment Transformer (SAT), p: Imitation learning policy
4: while not converge do
5: O, a, s.rop ~ DD"
Sample robot trajectory with length T Identify skills in video and form skill execution plan
6:
Skilllddetifiiatiionn)
7:
~ [0.7]
Sample a time index Predict the skill need be executed at time Compute SAT loss Predict the actions based on identified skill & at time Compute behavior cloning loss
8:
& Φ(or. 2)
9:
SSAr MSE(2, &
at = plo st. prop 21)
11:
Lhe MSE( ae. at)
12:
Liranafer = LSAT + Lbc.
13: Update & and p
Update models based on transfer loss
14: end while
Algorithm 3 Cross-embodiment Skill Compose (Inference)
Input: 7 pprompe: Human prompt video
2: Require: 6: Learned Skill Alignment Transformer (SAT). p: Learned Imitation learning policy
3: Skill Identification(( prompt)
>- Identify skills video and form skill execution plan
4: while not success or episode not end do
5: St propp env.get.obs()
Get observation and robot proprioception from environment
6: & o (O,,
Predict the skill need be executed at time t > Predict the actions based on predicted skill >> execute actions
7: å = pl ot, st
prop ;4)
8:
Execute the å, in environment.
9:
end while
A.4.1 Sinkhorn-Knopp Algorithm
XSkill employs the Sinkhorn algorithm to solve an entropy-regularie soft-assignment clustering procedure. As outlined in our main paper, our objective enhance coos-embbdiimnt skill repre- sentation learning. We strive to ensure that each embodiment fully leverages the entire embedding space. Given that the skill prototypes serve the cluster centroids, they essentially act as anchors in the representation space. We can efficiently realize this aim by uniformly soft-assigning all samples to every prototype.
Our goal to project a batch of skill representation matrix (ci....,cg. cK). where the columns of the matrix are c1.... C.. CK- The intended code
[= onto the skill prototypes
Q " Q8] or the target skill prototype probability should retain similarity with the projection while maintaining specific entropy level. This can be expressed as optimal transport problem with entropy regularization [63, 1]:
max TR TQ CT + (Q) QEQ
The solution [63, 1] is given by:
C E
Q' Diagg() exp
Diag(v),
where u and v are normalization vectors, which can be iteratively computed using the Sinkhorn- Knopp algorithm.
Bolkhorn-Knopp receives projection C as the input and iteratively modifies the matrix to satisfy the entropy regularization by producing double stochastic matrix. Through our experiments, we'r we've observed that three iterations are sufficient. The Pytorch-like pseudocode for the Sinkhorn-Kan Knopp can be found in pseudocode lisiting 1. The target skill probability for 2; be obtained from the th column from the output Q*
16

---

A.4.2 Temporal Skill Encoder & Prototypes Layer
The temporal skill encoder Stemporal consists of a vision backbone and a transformer encoder. To effi- ciently process a large batch images, we employ a straightffrwwrd 3-layer CNN network followed by an MLP layer as our vision backbone. This network can be trained on single NVIDIA 3090. Each image in the input video clip is first augmented by randomly selected operation from of image transformations, including random resize crop, color jitter, grayscale, and Gaussian blur. The augmented video clip is then passed into the vision backbone, and the resulting features flat- tened into 51 512-dimensional feature vectors. The transformer encoder, on the other hand, comprises 8 stacked layers of transformer encoder layers, with each layer employing 4 heads. The dimension of the feedforward network set to 512.
The prototype layer Sproecypp is implemented its weights with every training iteration. We freeze its weights for the first 3 training iterations to stabilize the training process. For the TCN loss, in practice, we replace the skill prototype probability with its unnormalized version C/C (before applying the Softmax function). We noticed that the Softmax function saturates the gradient, leading to unstable training.
single linear layer without bias, and we normalize
The additional hyperparameters are summarized in Table A5 and Table A6 for simulated and real-
world kitchen environments, respectively.
Table A5: Simulated Kitchen Skill Discovery Hyperparameter
Hyperparameter
Value
Video Clip length
8
Sampling Frames T
100
Sinkhorn iterations
3
Sinkhorn epsilon
0.03
Prototype loss coef
0.5
Prototype loss temperature
0.1
TCN loss coef
1
TCN positive window Wpp
4
TCN negative window W.
12
TCN negative samples
16
TCN temperature
0.1
Batch Size
16
Training iteration
100
Learning rate
le-4
Optimizer
ADAM
A.4.3 Skill Alignment Transformer
The Skill Alignment Transformer (SAT) comprises state encoder, denoted as fstate-encader and transformer encoder. The state encoder implemented as standard Resnet18. The transformer encoder consists of 16 stacked layers of transformer encoder layers, each employing 4 heads. and the feedforward network has a dimension of 512. As depicted in Section 3.3 of the paper, a set of skill representations { extracted from the sample trajectory т, and passed into SAT as skill tokens. For practical purposes, XSkill adopts uniform sampling approach, selecting NsAT prototypes from the skill list. This approach is motivated by two primary reasons. First, skills are typically executed over extended periods, and we only require information about the start and end times, as well as the time allocated to each skill. Uniform sampling preserves this necessary information while reducing redundant prototypes in the list. Second, human demonstrations may occur significantly faster pace than the robot's execution, leading to variations in the length of the skill list. This discrepancy can hinder the learning algorithm's performance during inference. By uniformly sampling fixed number of frames from the set, the learning algorithm operates under consistent conditions in both learning and inference stages. NS is set to approximately half of the
17

---

Table A6: Realworld Kitchen Skill Discovery Hyperparameter
Hyperparameter
Value
Video Clip length I
8
Sampling Frames
100
Sinkhorn iterations
3
Sinkhorn epsilon
0.03
Prototype loss coef
0.5
Prototype Softmax temperature
0.1
TCN loss coef
1
TCN positive window wp
6
TCN negative window wh
16
TCN negative samples
16
TCN temperature Tun
0.1
Batch Size
20
Training iteration
500
Learning rate
le-4
Optimizer
ADAM
average length of frames in robot demonstrations. During inference, if the length of the extracted skill list is less than NSAT, XSkill uniformly up-samples the skill list. We include representation token after the skill token and the state token to summarize the prediction information. The latent representation of the representation token is then passed into multi-layer perceptron (MLP) to predict the desired skill 2. We set NSAT 100 in the simulated kitchen environment and NSAT 200 the realworld robot trajectories significantly longer than those in simulation.
A.4.4 Diffusion Policy
We use the original code base from Chi et al. [35] and adapt same the configuration for both the simulated and realworld environment. We refer the reader to the paper for details.
Listing I: Pseudocode for Sinkhorn
PyTorch-like pseudocode for Sinkhorn -Knop
Sinkhorn -Koopp
def sinkhorn (scores, eps=0.05, niters=3).
Q = exp scores / e
Q /= sQ)
K, B = shape
u.
zeros (K), ones (K) / K, ones (B) / B
for in range niters):
sum(Q, dim=l)
Q == (r u). unsqueeze (1)
Q (c sum (Q, dim 00)= unsqueeze (0)
return (Q su, dim=0, keepdim=True T
A.5 Environment& Collections
We begin with a formal description of the three distinct data sources. 1). Human demonstration dataset: Herein, 7 = fop..., 07, }. where o, denotes the RGB visual observation at the time t. Within each trajectory, a subset of skills { is sampled from a skill distribution p(Z) contain- ing N unique skills and human performs
random sequence. 2). Robot teleoperation data: This dataset comprises teleoperated robot trajectories 7.' {(o0, grrpp aa)).., (OT, 57, where grrop a correspond to robot proprioception data and end-effector action at time respec- tively. We utilize the symbol for sf" throughout the main paper. Analogous to the human
prop .T.))
18

---

demonstration dataset, each trajectory incorporates a subset of skills = sampled from the skill distribution p( which the robot executes in a random sequence. 3). Human prompt video: This single trajectory of human video 'prompt = {oo. ... Uprampe demonstrate unseen composition of skills { p taken from the skill distribution p(Z). We represent the RGB video trajectories that include only the RGB visual observation {oo, or, } for both human and robot in the main paper as V; for the sake of simplicity.
A.6 Simulation
In order to produce cross-embodiment dataset, we modified the initial Franka Kitchen setup, in- troducing sphere agent distinctly visual from the original Franka robot. The sphere agent demon- stration dataset was generated by substituting the Franka robot arm with the sphere agent and re- rendering all 600 Franka demonstrations. The images for both Franka robot and sphere agent demon- stration are in a resolution of 384x384. Each trajectory in this dataset features the robot completing four sub-tasks in randomized sequence. The demonstration from both embodiments was divided into training set and prompt set, with the latter containing trajectories involving unseen com- binations of sub-tasks. This requires the robot to complete tasks namely, opening the microwave, moving the kettle, switching the light, and sliding the cabinet in order.
For skill discovery, we downsampled the demonstration videos to a resolution 112x1 and ran- domly applied color jitter, random cropping. Gaussian blur, and grayscale to the input video clips. For diffusion policy training, the environment observation incorporates 112 x 1 112 RGB image and 9-dimensional joint position (include gripper). We used stack of two consecutive steps of the observation input for the policy.
A.7 Realworld
We conducted data collection for our cross-embodiment dataset in real-world kitchen environment using a URS robot station. The UR5 robot is equipped with WSG50 gripper and a 3D printed soft finger. It operates by accepting end-effector space position commands at a rate of 125Hz. In the robot station, we have installed two Realsense D415 cameras that capture 720p RGB videos at 30 frames per second. One camera is mounted on the wrist, while the other provides side view.
Our dataset consists of demonstrations involving both human and robot teleoperation for four spe- cific sub-tasks: opening the oven, grasping cloth, closing the drawer, and turning on the light. To introduce variability, the initial locations of the oven and the pose of the cloth are different for each trajectory. Each demonstration trajectory involves the completion of three sub-tasks in a random order.
For training, created seven distinct tasks for each embodiment and collected 25 trajectories for each task. The robot teleoperation demonstrations were recorded using 3Dconnexion SpaceMouse at rate of 10Hz. For the inference task, we created two unseen tasks with three sub-tasks each, and four unseen tasks with four sub-tasks each. For tasks with three sub-tasks, we recorded both human and robot demonstrations as prompt videos, while for tasks with four sub-tasks, we recorded human demonstrations only. The details of task collections are illustrated in Tab. A7
During skill discovery, exclusively utilized videos recorded from the side camera and down- sampled them to 160x120 at 10fps. Similar to before, we applied random transformations such as random crop, Gaussian blur, and grayscale to the input video clips. For diffusion policy training, we used visual inputs from both cameras, downscaled to 320x240. The input to the diffusion policy included a 6-dimensional end effector pose, 1-dimensional gripper width, and two visual inputs from both cameras. We only considered one step of observation as the policy input. Position control was selected the diffusion-police action space, encompassing the 6-dimensional end effector pose and I-dimensional gripper width. During training, we applied random crop with shape of 260x288, while during inference, we utilized a center crop with the same shape.
19

---

Table A7: Training & Inference Task
Tasks
Human(seconds)
Robot(seconds)
Overlapping Training Task
Draw, Light, Oven Light, Cloth, Oven
12.92 + 1.19
29.12 2.03
15.23 0.92
32.76 2.42
Draw, Light, Cloth
15.73 + 1.22
26.83 2.28
Draw, Cloth, Light
17.21+1.05
31.71 + 3.74
Oven, Draw, Cloth
11,62+1.58
Human exclusive Training Task
Cloth, Oven, Light
12.6940.86
Cloth, Light, Oven
13.3740.67
Light, Oven, Draw
32.41+3.04
Robot exclusive Training Task
Oven, Light, Cloth
26.7542.56
Light, Draw, Cloth
27.10+1.95
Oven, Draw, Cloth
14.4
45.7
Draw, Cloth, Oven
12.6
41.4
Inference Task
Oven, light, Cloth, Draw
20.5
/
Draw, Cloth, Light, Oven
20.9
Draw, Oven. Cloth, Light
Draw, Light, Cloth, Oven
19.2
/
20


Reviews:
[{"review_id": "hrxlevt5-rP", "reviewer": "Reviewer_fc5Y", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an approach for imitating human demonstrations by using cross-embodiment skill representations. The approach learns skill representations from segments of video clustered through self-supervised augmentations. Alignment across human and robot skills is encouraged by maximizing the entropy of the marginal skill distributions predicted within each embodiment. To imitate human videos, a transformer is trained to predict a next skill from the human video to perform at a given state. A skill-conditioned policy is learned to execute these skills sequentially. Experimental results in simulation and a real-world setting show the robot can imitate novel combinations of skills given with human video. The pipeline outperforms ablations and is robust to perturbations and changes in execution speed across embodiments.", "summary_of_recommendation": "The paper presents a complex but well-justified approach to imitating human videos through hierarchical skill decomposition. Unlike past work, the approach is able to discover skills that transfer across embodiments without supervision or explicit alignment. Experiments ablate the components of the method and show reliable performance in simulation and the real world with robustness to speed and re-planning on perturbation. Limitations include using a small set of skills in both settings and some minor issues with the clarity and justification of the approach.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why is the time contrastive loss needed for the approach / do any experiments show it is helpful?\n- The prediction of skills for each frame could be clarified with pseudocode or a diagram. The learned policy appears to be conditioned on predicted skill rather than the nearest prototype, which seems more natural if the skill predictions are justified as classifying between prototypes. Was this choice ablated?\n- What does the variation between prototypes within the same cluster of Figure 5 represent?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work shows impressive performance at imitating human video by extracting and composing cross-embodiment skills. The method and  results are clearly presented. The authors discuss why each component of the method is needed and how they build on past work.\n\nStrengths:\n- Results show good performance at one-shot imitation of novel combinations of skills from human video.\n- Videos show good performance at imitation and replanning with perturbations\n- Approach is robust to changes in execution speed across embodiments due to SAT component\n- Writing is clear and video on website explains method well\n- Figure 5 shows the structure of extracted skill decompositions well\n- The approach extracts skill decompositions and aligns them across robot and human video without supervision\n- Policy and skill learning only needs offline data\n\nWeaknesses:\n- Number of skills must be specified in advance\n- Limited number of concrete skills in the evaluation environments: 7 in simulation and 4 in the real world\n- There are many components in the pipeline for this approach, not all of which are ablated / justified (see below)", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Zr62KyG0JR", "reviewer": "Reviewer_1ysS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new system for imitation learning from human videos. The input during training time is the demonstration video of both the human and the robot performing same set of tasks (with temporal compositions), and during run time the system takes one single video of human demonstration (possibly with unseen temporal composition) to specify task sequence order, and a continuous stream of robot observation for controller feedback (using a Diffusion policy). The core innovation of the paper, is to learn the correspondence between robot and human demo video by mapping them to a same set of “skill prototypes”. The network loss encourages a batch of human or robot video clips to map to as many prototypes as possible, indirectly encouraging clips from human and robot containing similar temporal (i.e. task) information to map to the same prototype.", "summary_of_recommendation": "I lean towards acceptance given the effort the authors put into building such a system, and running thorough evaluations for it. On the other hand, I hold reservations on what benefit are we actually gaining, in terms of ability to generalize etc., with such large number of total learned skill prototypes.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The technical components of the paper seems solid and well executed. The authors put substantial effort in building such a complex system, and evaluate the system on both synthesized and real tasks. \n\nIn terms of the core idea of “skill discovery”, I’m not too sure if calling them “skills” is a bit of over-claim, and in the end if they are better (more generalizable, robust) than learning a shared latent continuous encoding of human and robot videos. In detail:\n1. There are 512 skill prototypes and the paper is “evenly assigning” samples in each batch to all prototypes — most of the skills learned therefore seem to only represent short segments in the video without interpretable meaning.\n2. If most of these 512 skills are not interpretable anyways, one might wonder if one should build an easier system with continuous (rather than discrete as in the paper) latent representations. There is plenty of literature learning correspondence between images/video clips between two domains, which can potentially map human and robot video windows to a shared latent space. The policy then might directly operate on current latent (mapped from robot video window) and near-future latent from human video. While asking a new baseline is out of scope, I might appreciate some discussions in the paper on system design alternatives.\n\nOther minor concerns:\n1. The discussion of related works is not entirely fair. To my understanding, though it’s true that MimicPlay [61] does not take human prompt video as input, the focus is also quite different with [61] trying to minimize robot demo collection which is much more tedious than human demos. This paper seems to use same amount of human and robot demo data, which I assume means more robot demo time than [61]\n2. The paper honestly describes that when the human prompt video contains unseen transitions between temporal tasks, the system would fail. Then what exactly is the generalization ability of this learned representation? I don’t feel like we can confidently call a task sequence “unseen” if both the primitive tasks (e.g. press button) and the transitions between tasks need to be “seen”.\n3. Even combing the main text and appendix, the paper is not self-contained in some places. Quick examples: what are the augmentations used? What is “iteratively modified to produce a double stochastic matrix” and why this can be intuitively viewed as “entropy regularization”?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "KjsKHfaeu7", "reviewer": "Reviewer_9Eu4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes XSkill - an imitation learning method for cross-embodiment settings. XSkill recognizes the skill sequences from a human demonstration video as a prompt, then conditions the skill sequences on a low-level diffusion policy to reproduce the task. At training phase, XSkill samples two aligned human and robot videos from a dataset, maps them into a latent space, predicts the primitive skill probability vectors from the observed latent vector, then finally matches the skill probability vectors with Sinkhorn Clustering targets by a CrossEntropy objective. Additionally, a low-level diffusion policy is trained with the conditioned skill latent representation to execute the low-level skills. At inference phase, a human video is prompted together with the current robot observation, then XSkill predicts the skill sequences and the current skill to condition on the low-level diffusion policy for task execution. The method's efficacy is shown in various real-world kitchen tasks.", "summary_of_recommendation": "Overall, the method has some originality and good experiments showing its efficacy. However, with the current form of paper quality and presentation, I recommend rejection.\n\n------ Post Rebuttal ------\n\nThe authors addressed all my concerns. I decided to raise the score to Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Overall, I encourage the authors to significantly refine the papers, as it is very unclear in their current form.\n\n1. An overall pseudo code for training and inference of XSkill would be appreciated.\n2. More elaborations on the Sinkhorn Clustering motivation and how it is used to compute target skill probability from batch videos are needed.\n3. What are the motivations for using a low-level diffusion policy in this case? \n4. With the described training process, it seems XSkill will overfit the objects being manipulated in the videos, hence bypassing the correspondence problem between morphologies. Is this really a desirable property? If so, in the imitation learning case, we can just learn the affordance of objects being manipulated in keyframes and no need for video data. Otherwise, evidence for capturing morphology correspondence must be provided, e.g., a rotating human hand with a corresponding rotating robot end-effector.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "++: Strength, --: Weakness\n\n**Novelty & Significance**\n\n++ Although bearing similarities with previous works on latent space encoding for cross-domain imitation learning, XSkill is interesting as it discovers and recognizes the primitive skills across morphologies, which is a powerful concept for cross-domain imitation learning for long-horizon tasks.\n\n++ Skill Alignment Transformer is a good idea for re-planning on task-level.\n\n++ The real-world kitchen dataset will be appreciated by the community if released.\n\n-- It seems XSkill still needs datasets collected from teleoperations. Similar to telekinesis teaching, these datasets are known to be very hard to collect large samples. This will hinder the applicability of XSkill.\n\n-- Other limitations are mentioned by the authors.\n\n**Quality & Clarity**\n\nThe presentation flow of the paper is fair to follow. However, there are some confusing technical.\n\n++ Strong experiment setups and adequate baselines.\n\n-- It is very confusing on how the human & robot videos representing the same task are sampled and constrained to be closed in the latent space. Fig. 1 depicts aligned videos are input for learning latent space with the same task and different embodiments. However, on line 95, both human and robot datasets are said to be unaligned.\n\n-- Fig 2. does not help in the understanding of training XSkill. What is “augutation”? Why are there two versions of video batch for computing the skill probability vectors?\n\n-- It is not clear how the Sinkhorn Clustering works in this case. On line 144, what are C and Z, as it is not introduced? Are they input as costs or marginal probabilities? It is very confusing on the intuition at lines 146-148. Why does utilizing as many as possible skill prototypes promote prototype alignments in different embodiments? Overall, there is also no clear motivation why Sinkhorn Clustering is applied in this case.\n\n-- The motivation for using a low-level diffusion policy in this setting is unclear. Why does multi-modality in myopic policy benefit in executing tasks? It seems myopic diffusion policies might jump between modes and thereby deteriorate task success.\n\n-- Metrics on overall task success are needed, as this is what really matters.\n\n-- Some typos exist. For example, “CorssEntorpy” on line 134 in Fig. 2 caption.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduced an imitation learning algorithm that can perform tasks based on a single human demonstration (prompt video). The key technique is constructing a latent skill representation space (XSkill space) shared across human and robot embodiments. This skill space is learned through self-supervised objectives, and the skill representation is used to condition a low-level diffusion policy to generate the motor actions. This paper received mixed initial reviews. The reviewers raised some concerns, including the presentation clarity, comparisons to existing work like MimicPlay, the use of discrete skill prototypes, etc. The authors did a good job addressing these issues in the discussion phase. As a result, Reviewer 9Eu4 updates their rating to Weak Accept, making the final recommendations from the reviewers all positive: two Weak Accepts and one Strong Accept. Although some of the concepts behind this work have been explored in prior work, the AC believes that this work has presented a convincing system with the ability to understand and execute tasks based on human prompt videos. Taking into account the reviews, the rebuttal, and post-rebuttal discussions, the AC recommends accepting this paper at CoRL.
Decision: Accept (Poster)