Translating Robot Skills: Learning Unsupervised Skill Correspondences Across Robots
Authors: Tanmay Shankar
Yixin Lin
Aravind Rajeswaran
Vikash Kumar
Stuart Anderson
Jean Oh
Venue: ICLR 2022 Submitted (2022)
Keywords: Robot Skills
Unsupervised Correspondences
Unsupervised Learning
Alignment
Density Matching
Skill Learning
Robot Learning
Transfer Learning
Skill Transfer

Abstract: In this paper, we explore how we can endow robots with the ability to learn correspondences between their own skills, and those of morphologically different robots in different domains, in an entirely unsupervised manner. We make the insight that different morphological robots use similar task strategies to solve similar tasks. Based on this insight, we frame learning skill correspondences as a problem of matching distributions of sequences of skills across robots. We then present an unsupervised objective that encourages a learnt skill translation model to match these distributions across domains, inspired by recent advances in unsupervised machine translation.  Our approach is able to learn semantically meaningful correspondences between skills across 3 robot domain pairs despite being completely unsupervised. Further, the learnt correspondences enable the transfer of task strategies across robots and domains. 
We present dynamic visualizations of our results at https://sites.google.com/view/translatingrobotskills/home. 

Under review as a conference paper at ICLR 2022
TRANSLATING ROBOT SKILLS: LEARNING UNSUPER- VISED SKILL CORRESPONDENCES ACROSS ROBOTS
Anonymous authors Paper under double-blind review
ABSTRACT
We explore how we can endow robots with the ability to learn correspondences between its own skills, and those of morphologically different robots different domains, in an entirely unsupervised manner. Our insight and premise is that mor- phologically different robots use similar strategies (high-level skills sequences) solve similar tasks. Based this insight, we frame learning skill correspondences as a problem of matching distributions of sequences of skills across robots. We then present unsupervised objective that encourages learnt skill translation model to match these distributions across domains inspired by recent advances unsupervised machine translation. Our approach is able to learn semantically mean- ingful correspondences between skills across robot domain pairs despite being completely unsupervised. Further, the learnt correspondences enable the transfer of task strategies across robots and domains. Dynamic visualization of our results can be found here: https://sites.goote wwwpssoooooooooooooiooeeeeeeoeee wwtp////wwwewtttannooooooo//////
INTRODUCTION
Humans have remarkable ability to efficiently learn to perform tasks by watching others demonstrate similar tasks. For example, children quickly learn the skills needed to play a new sport by watching their parents perform skills such kicking a ball. Notably, they are able to learn from these visual demonstrations despite significant differences between themselves and the demonstrator, including visual perspectives, environments, kinematic and dynamic properties, and morphologies. This ability may be attributed to two factors; firstly, humans have well-developed basic motor skills that we can execute with little effort. Second, the demonstrator uses, and understand a corresponding set skills that we can execute ourselves (Meltzoff & Moore, 1977).
recognize the sequence of skills (or the high-level strategy)
In this paper, we explore how we can endow robots with this ability i.c., to adopt task strategies from morphologically different robot demonstrators, and then execute corresponding skills solve similar tasks. Key to solving this problem is for the robot to identify how its owns skills correspond those of the demonstrator, which tremendously powerful. First, allows the robot to adopt the demonstrator's strategies for solving various tasks. Second, by adopting and adapting these
Boxter Left
Sawyer
Baxter Right
Figure 1: Sample skill correspondences learnt by unsupervised approach, across three different morphologi-
cal robots. We visualize reaching skill the Baxter left hand (top), translated to the Sawyer (middle) and the Baxter right hand (bottom) respectively.

---

Under review as a conference paper at ICLR 2022
strategies for itself, the robot can efficiently learn to solve a variety of tasks previously outside its repertoire. Finally, allows us to understand the task strategies used by various robots in a unified manner, enabling robots to learn from data collected with heterogeneous collection of tasks and robot morphologies. Skills provide a natural framework to facilitate such concise knowledge transfer across robots compared to low-level robot controls; skills inherently abstract away low-level details that may differ across the demonstrator and the learner, and instead focus on the commonalities between them, such as the task strategy. For example, skills such as reaching and placing on robot manipulators abstract away differences in morphologies or configuration, and are thus well grounded across robots, while there may not be obvious correspondence in their low-level actions.
How can one acquire correspondences between skills? Stated more formally, given morphologically different robots, a set of unlabelled demonstrations of each robot solving a variety of tasks, and a method for extracting skills from those demonstrations, how can one learn correspondences between the skills of the different robots? Learning such correspondences is straightforward when supervised pairs of skills or trajectories are available. However, collecting and annotating such data is time- consuming and tedious, and requires a high degree of human expertise, particularly at the scale necessary to successfully learn correspondences across diverse set of skills. In contrast, we explore whether we can learn such skill correspondences in an unsupervised manner, i.e., without access to supervised skill or trajectory pairs. Learning unsupervised skill correspondences potentially allows for learning a broad scope of correspondences from the diverse unlabelled and unsegmented data newly available in roboties (Sharma et al.. 2018: Mandlekar et al., 2018). Unfortunately. learning unsupervised correspondences is a difficult problem to solve- without supervision, it is difficult to guide learning towards the right correspondences. While this may be alleviated some extent by incorporating unsupervised constraints (Zhu et al., 2017: Ganin et al., 2016; Zhou et al., 2019), there are many spurious correspondences that satisfy these constraints.
How can one then go about learning such skill correspondences in an unsupervised manner? We propose leveraging the two following insights do so. Our first insight is that different morphological robots use similar task strategies (in terms of sequences of skills) to solve similar tasks; in other words, the sequences of skills executed by different robots to solve similar tasks ought to belong to similar distributions. We observe that this is generally true; for example, robot pouring out a cup of tea would likely first reach for the kettle, grasp it, move it appropriately over the cup, and then begin to pour the tea, irrespective of its exact morphology. Our second insight is that learning skill correspondences without access to supervised data closely mirrors unsupervised machine translation (UMT), where the objective is to learn a translation between representations in different languages (such as between word embeddings across languages), without access to parallel data (Conneau et al., 2017; Lample et al., 2018). Inspired by these two insights, we derive an unsupervised objective to guide our learning towards meaningful skill correspondences.
We approach the problem of learning unsupervised skill correspondences by learning a translation model to map from the skill space one "source" robot to the skill space on another "target" robot. We construct an unsupervised objective that encourages the translation model to respect our first insight- i.e., to preserve the sequences of skills observed across both the (translated source) and target robots. To do this, we take inspiration from our second insight, and specifically from Zhou et al. (2019), and construct explicit probability density models over the sequences of skills observed on the source and target robots, and then train the translation model to match these distributions.
We evaluate the ability of our approach to learn unsupervised skill correspondences across three different robots, the Sawyer robot, and the left and right hands of the Baxter robot. Our approach is able to learn semantically meaningful correspondences across each of the 3 pairs of robots, despite being completely unsupervised, as depicted in fig. 1. The learnt correspondences facilitate transferring task strategies across across domains, as we demonstrate set of downstream tasks. Our results are visualized at https::/sites.googee
2 RELATED WORK
Skill Learning: Eysenbach et al. (2019); Sharma et al. (2020) both address unsuperivsed skill learning from interaction data, by constructing information theoretic approaches. Fox et al. (2017); Krishman et al. (2017); Shankar et al. (2020); Sharma et al. (2018); Shankar & Gupta (2020); Kipf al. (2019); 9); Gregor et al. (2019); Kim et al. (2019) instead learn skills or abstractions from unlabelled
2

---

Under review as a conference paper at ICLR 2022
demonstration data by performing latent variable inference. These frameworks all learn skills in the context of a single domain, while we seek to learn correspondences of skills across domains.
Unsupervised Correspondence Learning: Our problem bears close resemblance with unsuper- vised machine translation (Conneau et al., 2017; Zhou et al., 2019), and unpaired image translation (Zhu et al., 2017; Park al., 2020). Specifically, they all share the notion of learning correspondences across representations learnt from unpaired, "monolingual" data:
Unsupervised Correspondence in Machine Translation: Conneau al. (2017) leveraged domain- adversarial training (Ganin et al., 2016) 6) to align word embeddings of two languages. Sennrich et al. (2016); Lample al. (2018) use the idea of back mack-translation constraint the learnt translation models across languages. Zhou et al. (2019) learn bilingual word embeddings by matching explicit density functions over the word embedding spaces across languages.
Unsupervised Correspondence Image Translation: The vision community has similarly ex- plored the unpaired "mage-to-image translation setting (Zhu et al., 2017; Park et al., 2020), using cycle-consistency losses (Zhu et al., 2017) or contrastive losses (Park et al., 2020). Unsupervised Correspondence in Video: Bansal et al. (2018); Wang et al. (2019) extend Cycle- GAN (Zhu et al., 2017) to the video domain, by incorporating temporal consistency losses.
Domain Transfer in Robotics and Graphies: The robotics and graphics communities have taken interest in cross-domain transfer of policies in recent years:
Policy Transfer with Paired Data: Gupta et al. (2017); Sermanet et al. learn morphology and viewpoint invariant feature spaces for policy transfer respectively, but require paired data to do so.
Policy Transfer via Modularity: Hejna et al. (2020); Devin et al. (2017); Sharma et al. (2019) address morphological transfer by modularity in their policies. Hejna et al. (2020); Sharma et al. (2019) adopt modularity in a hierarchical sense, but leverage common grounding of subgoals across morphologies to perform transfer. We do not assume access to such common grounding.
State based Policy Transfer: Liu al. (2020): Schroecker & Isbell (2017); Ammar et al. (2015) address cross-morphology transfer by state based imitation learning.
Motion Retargetting: The graphics community has addressed transferring behaviors across mor- phologically different characters (Hecker et al., 2008; Aberman et al., 2020; Villegas et al., 2018; Massi- al., 2017), using carefully handerafted kinematic models. These works transfer behaviors by imitating joint positions, and performing inverse kinematics to retrieve the full character state, which is often infeasible for widely different morphological characters.
Unsupervised Action Correspondence: Zhang et al. (2021); Kim et al. (2020); Smith et al. (2020) address learning low-level state and action correspondences from unpaired, unaligned interaction and demonstration data respectively. While similar in spirit, our work argues that learning high- level skill correspondence instead is a more natural choice, as mentioned in the introduction.
Applicability to learning skill correspondences: While successful in their respective problem domains, existing approaches are not trivially applicable to our problem. The adversarial training used in most of these approaches is notoriously difficult to train, and is prone to the mode dropping problem (Li & Malik, 2019). They require strong constraints such as pixel-wise identity losses (Zhu et al., 2017; Zhang et al., 2021), or inherent similarity of the spaces to be aligned, such as word embeddings across languages (Conneau et al., 2017; Zhou al., 2019). Learnt skill representations do not possess this property in general. Shortcomings notwithstanding, these approaches provide insight into how we may pursue learning unsupervised skill correspondences.
joint-wise
APPROACH
3.1 PRE-REQUISITES
We begin by first reviewing an important prerequisite the skill learning pipeline of Shankar Gupta (2020), which provides us a learnt representation of robotic skills given an unlabelled robot demonstration dataset. Their method first represents robotic skills as continuous latent variables z, and introduce a Temporal Variational Inference (TVI) to infer these skills or latent variables. TVI trains
variational encoder q(2(T) that takes as input a robot trajectory 7 = {s1. a1 Q1....S--1. an-14 sn), and outputs sequence of skill encodings
{21, 22, 2..2,,, J. Note that these skill encodings repeat over time for the duration of a given skill. TVI also trains a latent conditioned policy x (als, 2) that takes as input robot state 8, and the chosen skill encoding 2, and predicts the low-level action that the robot
3

---

Under review as a conference paper at ICLR 2022
Learnt Skill Representation
Translated Skill Space
Skill Tuple Dersity
Source Morphology
Density
Estimation
Translation Model
Eso Esalaatin Latin Sample Lkkllhood
Targes Morpho ogy
Sampling
Do
Sampling
Density Estimation
Learnt Skill Representation
Skill Tuple Density
Density Estimation
Evaluating Sample Likelihood
Sampling
Translation
Figure 2: Overview of our approach. We translate the original learnt skill space to the target domain. We then construct explicit density estimates over the original target and translated source skill-tuple spaces. We train translation model to maximize the likelihood of randomly sampled translated source and original target skill-tuples under these densities respectively, affording meaningful saill-corespoondenees across both robots.
should execute. We direct the reader Shankar & Gupta (2020) for more thorough description of their skill learning approach.
3.2 PROBLEM SETTING
Consider two robots with potentially differing morphologies and environments (or "domains") represented source and target domains, M2 and M, respectively. Associated with each domain an unlabelled and unsegmented dataset of demonstrations of the robot performing various tasks represented Dg and D, respectively. Dg and D, are collected independently, on an intersecting (but not identical) set of tasks. This is equivalent to the setting in machine translation without parallel data, with access only to monolingual corpora.
We also assume access to a skill-leaming pipeline, that can take such an unlabelled demonstration dataset D of robot M performing various tasks, and learns representation Z of skills given robot. We specifically use T'VI (Shankar & Gupta, 2020), but we believe our approach is compatible with any unsupervised skill-learning approach that affords continuous representation space. We train the skill-learning pipeline independently on each of the domain datasets Dg and D,. This affords us skill encoders 9s and 9r- and latent conditioned policies π. and कy for each domain. We may then encode trajectories in both domains Tg E D, and E D, into their constituent skills |Ty]
[>M T1=1 and { { [(=1 via 9s and 9, respectively. We represent the space of skills learnt on the entire dataset in each of the two domains as Z.g and E, respectively.
Our goal is to learn semantically meaningful correspondences between skills in the source domain, Zg E Zy. and those in the target domain, 2, E F.,, that helps solve tasks in the target domain M,- We specify these correspondences by learning a "translation" function T Z, + Z,,. that maps skills in the source domain Zg. translated skills in the target domain Z This mirrors the word translation models present in Conneau et al. (2017); Zhou et al. (2019). Learning meaningful cillelcoeresonneeces thus amounts learning good translation model T. We can also translate the entire source space Zg to the target domain, to retrieve the translated source space, 2 gg-
3.3 USING SEQUENTIAL INFORMATION OF SKILLS
Our first insight that different morphological robots follow similar strategies to address similar tasks in other words, sequences of skills executed by two different robots ought to belong to similar distributions. is important to consider sequences of skills here rather than skills in isolation, since the ordering of skills gives us additional information to bias our learning towards the right correspondences. Constraining correspondences based on individual skills alone could lead to incorrect results. For example, learning that "reaching" skills in one domain correspond to "grasping" skills in the other domain evidently incorrect, and is an error that sequential information could bias learning away from, since grasping skills are systematically executed after reaching skills.
4

---

Under review as a conference paper at ICLR 2022
However, processing entire sequences of skills is computationally infeasible for arbitrarily long trajectories. Instead, we draw inspiration from the simple yet powerful bigram models in the NLP community, and consider consecutive pairs, or tuples of skills. While less expressive than the full skill-sequences, these tuples retain enough information of the ordering of skills to guide the correspondence learning towards the right solution, and are far more computationally tractable. We thus construct a skill-tuple space, which represents the various transitions between skills observed in a given domain. Each consecutive pair of skills (z², 1++1) in the original space Σ of given domain is represented as single point z' in the skill-tuple space X of that domain. For initial and terminal skills ===0 and zd=|r|, we pad these tuples, and treat the preceeding and succeeding skill encodings as appropriately dimensioned 0''s respectively.
The skill-tuple spaces for source and target domains are represented as Xg and X respectively, and Nr
are implemented as a set of skill-tuples from each domain. i.c. X { ) N/ : = {x,,,) i=1 The procedure to construct these spaces is presented in the sub-routine of algorithm 1. We also construct translated skill-tuple space, x.- that is simply the translation of all skills present in g.
Le. _r_r { 2 i=1- Translating skill-tuple z'= (2 2) from source to target is done by translating the individual skills of and 25+1 to the target domain, .T the translated skill-tuple x S-T (2 =
and " and constructing
3.4 DISTRIBUTIONAL PERSPECTIVE LEARNING TRANSLATION FUNCTIONS
We would like our translation model to exhibit two properties first, to translate source skills 2 such that they belong to the distribution of target skills, and second, capture all modes of skills that exist in the target domain. Dropping modes of skills limits the set of skills the target robot can use, reducing its capabilities. GANs (Goodfellow et al., 2014) and domain-adversann et al., 2016) satify the first property since they optimize for how realistic the generated samples are (or how indistinguishable source and target domain features are (Ganin al., 2016)), but often drop modes of the "real" data (Li & Malik, 2019).
approaches (Ganin
Instead, we approach this problem from an explicit distribution perspective, and maintain explicit probability density estimates over the translated source and target skill-tuple spaces, X and X,, p( and p(z.,.) respectively. Following our insight that sequences of skills on different robots ought to belong to similar distributions, we would like these distributions p(xg_.)) and p(z,.) to match. We can optimize our translation model to enforce this, by maximizing the likelihood of translated skill-tuples Tig -r under the target skill-tuple distribution p(ı,,, ), and the likelihood of target skill-tuples T under the translated skill-tuple distribution p(xg_T) This is similar in spirit to optimizing both the forward and reverse KL between two distributions. We formalize this objective and provide intuition for it below.
Consider constructing an explicit probability density function pr, (x),, over the target domain skill- tuple space X,.. We follow Zhou et al. (2019)'s choice of representing p, (I)) using a Gaussian Mixture Model (GMM), owing to its expressive power and efficiency in low-dimensional spaces, but note that a wide variety of explicit density estimators may be used here. Here, P2 (x) is a GMM with N₂ Gaussian kernels, each centered at target skill-tuple a'z, X,:
NT p(a p( p(
Pr (x)
Σ
Σ p(at)r(,))
(1)
Zhou et al. (2019) use kernel weights p(z',,) proportional to the frequency of the word the kernel represents. In our setting, each kernel is simply one of the skill-tuples in the continuous target skill space X,, motivating our choice of equal mixture weights over all kernels (i.e. skill-tuples), i.e. p(a".. = 1/N,. The mixture components p p(z)a'r.)) are specified by a fixed variance Gaussian centered at 2' " i.e. p(x|=',, ) Naxx) 2 o²), where specifies the standard deviation of the Gaussian kernel. We can train a translation model to translate skills in such a way that the translated skill tuples xg-- belong to the distribution of target skill-tuples p,, (z), or that they are realistic with respect to the target distribution. We do this by maximizing the "forward" likelihood C, of the translated skill-tuples zg.rr under the target density p. (z)::
Cj Eng pp(p))(()))) pppx))=agnr ~ To ( (og,
(2)
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Translating Robot Skills
Input: Dy. 9s: tr Rg: Fr
Require demonstration datasets, trained skill encoders & decoders for source & target domains Output: T--r( 112g
Output translation model
1: Initialize T T (2g)
b Initialize translation model
2: x < BUILDSKILL SUUIISSSIITUUPLLSSSACCSSSSU
P Construct source skill-tuple space P. Construct target skill-tuple space
3: X, f BUULSSEIL.TUPESSPPCCCCPrrr
4: (x) + -i p(. 1x)
D Update target GMM density via eq. (1)
5:
for ir [1, 11,2,.., 1., Niterationn do
6:
x + ( (gg)
> Update translated-source skill-tuple space
7:
Ps TT (x) - -isl p( 12 s.
D. Update translated-source GMM density via eq. (3) b Get batch of source and target skill-tuples b Translate source skill-tuple to target domain b Evaluate forward objective Evaluate backward objective Evaluate full objective Update translation model with gradient ascent
8:
Ig Xg. Tip ~ Xy
9:
Ig I ~ Tg
L + log PT (T
Lb logps Ps (Ip)
12:
Lt C Lh.
13:
Update T via VL
Sub-routine: Build Skill-Tuple Space
Input: Domain M
Output: Skill-Tuple space X,,
14: function BULLDSKLLLTUPLLLSAACECODDO M)
15:
X,, + -0
> Initialize empty skill-tuple space
16:
for E (1.2...., Noldd do
17:
Taas ~ DR
Retrieve batch of trajectories from domain dataset 9sr (. (Tara) Encode trajectories sequence of skills via domain encoder
{2 1||
18:
19:
{a J==l +- {(2 20 24 24)4
11/111
>> Assemble skill tuples
20:
X, M t X U fot 111
M1=
b Add skill-tuples to skill-tuple space
21:
return X
To encourage the second mode-covering property in the translation model, we can also optimize the likelihood of the target skill-tuples under the translated source skill-tuple space. To do so, first construct an explicit probability density function Ps-r (r) over the translated source skill-tuple space, xg_.. that is equivalent to Pr (z). Ps_ (I) also similarly represented as a GMM, with Ng... equally weighted Gaussian kernels, each centered at translated skill-tuple x' s'a X g-y
NowT
N
Panr (x)
Σ 1=1
p( p(
p(
1=1
As before, the conditional p(x|=' Na -mii 0²). We may then optimize the "backward" likelihood La of a set of target skill-tuples x under P P (z):
Cb -p log Pour (2,))
(4)
By combining these two objectives may construct our final objective L for training the translation model, by simply combining these two objectives, L = Ls Lb:
L = Eeg ~p(pg). Tan (112g) log Pr (xg . Eu, mp(z,) log Ps sr (IrT)
(5)
Parsing the objective: In contrast with adversarial training methods, which require alternating training phases and stability tricks such as gradient penalties, our objective amounts to simple maximum likelihood (albeit in two directions). It is hence simpler, more stable, and quicker to Senverge.IItuttivee Ey captures how "realistic" each translated skill-tuple looks with respect to the target skill-tuple tuple distribution, while La captures how well the translated skills cover the modes of the target skill-tuple space. We present our full approach for learning unsupervised skill correspondences in algorithm algorithm 1, and pictorial representation of it in fig. 2.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We evaluate our approach's ability to learn skills correspondences across the following robots the Sawyer robot, the Baxter left hand, and the Baxter right hand. Our choices of robots are dictated by the availability of demonstration datasets for these respective robots. We consider of the possible 6 unique domain pairs Baxter Right-Hand to Baxter Left-Hand (Bax-R to Bax-L), Baxter Left-Hand to Sawyer (Bax-L to Saw), Baxter Right- Hand to Sawyer (Bax-R to Saw).
Robots:
Datasets:
For the Sawyer robot, we make use of the Roboturk dataset (Mandlekar et al., 2018), which consists of roughly 2000 demonstrations across different tasks. For the Baxter robot, we utilize the MIME dataset (Sharma et al., 2018). We partition the MIME dataset into two disjoint sets with solely left-handed and right-handed trajectories respectively. Each single-handed dataset also roughly contains 2000 demonstrations across 16 tasks. We treat each single-handed dataset as the respective dataset for the Baxter left-hand robot and the Baxter right-hand robot.
We learn seileeeeprssnntaiionn for each of the 3 robots independently from their respective datasets, using TVI (Shankar & Gupta, 2020). We use a 16-dimensional skill- representation space for each robot, that is learnt from robot joint-states and joint velocities. We then freeze these learnt skill representations for all of our experiments. We follow the preprocessing steps and training parameters specified by (Shankar Gupta, 2020). We also manually annotate 50 skills in each domain with one of 6 semantic labels of what type of skills they were. We emphasize that our approach is not trained with these labels, but rather is only evaluated against them.
Skill Representation:
4.1 LEARNING MEANINGFUL SKILL CORRESPONDENCES
The first question we would like answer is "Can our unsupervised method learn meaningful skill-correspondencss between skill spaces?" We first present qualitative results towards this.
Translating Individual Skills and Entire Trajectories: We present two sets of visualizations: individual skills and their translations, as well as entire trajectories (or sequences of skills) and their translations. We first visualize a set of source domain trajectories Tg. obtained by rolling out their skill encodings (=))L) via the source domain policy gT. We then translate these sequences
of skill encodings to the target domain, i.e. {=/_T>/' 1 and visualize rollouts of the target policy Tr conditioned on these translated skills. We present visualizations of individual skills and their translations, as well as visualizations of entire trajectories and their translations in fig. 1 and at wtw.://www....e///ei//aaoaaonnnn
Analysing Qualitative Skill Translations: Despite being learned in a purely unsupervised manner, we observe our translation model is able learn good semantically meaningful coarse skill cor- respondences between semantic clusters of skills that emerge in the original spaces, across all 3 domain pairs. For example, from fig. I, we see source domain reaching skills correspond to target domain reaching skills across all 3 domain pairs, do placing and pushing / sliding skills to the left or to the right, and returning skills to their rest configuration. However, our model is unable to guarantee learning perfect finer correspondences (such between twisting grasping skills), or between variations of skills that belong to the same semantic cluster in the original skill spaces (such as reaching with different arm shapes, as in fig. 1). We emphasize that our method is unsupervised, and instead exploits similar contexts of skills across domains to learn correspondences; these results result. Finer skills and intra-cluster variations of such skills
are understandable and expected are often executed in similar contexts, such as after reaching or before placing skills, making them difficult to disambiguate. We note that despite this, our model often does translate finer skills (such as twisting and grasping) correctly, suggesting its potential for improvement via techniques such as iterative refinement (Conneau et al., 2017), or via additional inductive biases.
Using the learnt correspondences, we are able to transfer the source domain trajectory's 's overall structure, and sequence of skills executed, remarkably well to the target domain. Our model does so despite variations in the precise shape of the robot arms, or start and end positions of these respective skills. This provides further evidence that the learnt correspondences are indeed semantically meaningful, and additionally suggests these learnt correspondences would be useful in transferring task strategies across domains. We believe this is a powerful result, especially so considering approach does in completely unsupervised manner.
7

---

Under review as a conference paper at ICLR 2022
Table 1: Evaluating Skill-Tuple Distribution Matching via approach against various baselines, across all domain pairs. Baselines are adapted from (1): Ganin al. (2016). (2): Zhu al. (2017), (3): Liu et al. (2020). Lower is better for all metrics except the label accuracy.
Domain Pair
Chamfer Distance
Cycle Error
Label Accuracy
Supervised Z Error
Approach:
Lb
DomAd [1]
38.1
43.4
20.1
17.4
10.3%
15.2
Bax-R
Cycle GAN [2]
32.1
39.8
17.0
9.2
14.2%
14.9
Bax-I.
State Im [3]
45.3
49.1
31.0
24.8
32.8%
12.3
Ours
14.3
20.1
8.2
10.2
73.8%
8.1
DomAd [1]
50.9
60.1
24.2
21.3
9.8%
14.9
Bax-L
Cycle GAN [2]
38.5
53.7
17.1
8.8
12.0%
15.4
to Saw
State Im [3]
66.5
73.1
38.9
22.4
19.3 %
36.8
Ours
16.8
33,9
7.8
68.7%
9.0
DomAd [1]
54.8
80.2
21.2
24.8
11.1%
19.1
Bax-R to Saw
Cycle GAN [2]
47.3
78.5
16.7
10.3
16.8%
20.1
State Im [3]
72.5
85.4
33.9
27.7
20.1%
23.0
Ours
15.4
36.5
7.2
10.1
64.6%
12.1
Quantitative Evaluation of Skill- Skill-Tuple Distribution Matching: To quantitatively measure how well our approach can match the distributions of skill-tuples across robots, evaluate the following unsupervised metrics. We first evaluate the forward and backward GMM densities Ly and Ls. We also compute the Chamfer distance (Fan et al., 2017) between the skill-tuple spaces, which represents the nearest neighbor distances across two point sets. Together, these three metrics specify how close the skill-tuple distributions across domains are. We also evaluate the following supervised metries, using the manually annotated semantic labels associated with 50 skills in each domain. We reiterate these labels are only used for evaluation purposes, and are unseen train time. First, we evaluate how accurately the learnt correspondences capture semantics of skills across domains, by measuring how well the semantic labels associated with set of skills in the source domain match with those in the target domain upon translation, reported as the label accuracy in table 1. We also evaluate supervised Z error i.c. the average distance in latent space between the translated source domain skills, and the nearest target domain skills with the same semantic label.
We compare our approach against the following alignment baselines, and present the results table 1. training (Ganin et al., 2016), where match skill-tuple
Our first baseline is distributions by training the translation model to fool a discriminator network trained to identify domains given skill-tuples. We also compare against Cycle-GAN (Zhu et al., 2017), where train two translation models between skills from source and target domains to be cycle-consistent with another, while also optimizing domain-adversarial style objective (Ganin et al., 2016). Finally, we also compare against a State-based Imitation approach (Liu et al., 2020), where we transfer trajectories across domains by copying end-effector states across robots, using inverse kinematics to retrieve the closest feasible joint state.
Analysis of Qualitative Results of Skill-Tuple Distribution Matching: From table 1, we observe that our approach is notable able to learn correspondences that achieves a much higher semantic label transfer accuracy, consistent with our approach learning good coarse correspondences. The baseline approaches in contrast, are only able to achieve random-level label transfer accuracy, indicating their inability learn correct correspondences. For example, the domain adversarial baseline ends up mapping several different types of skills in the target domain to single modes of skills in the target domain, across all domain pairs. The Cycle-GAN baseline somewhat mitigates this approach, but takes shortcut in the learning and simply places single source skill nearby every mode of target skill. In contrast, our approach explicitly optimizes for distribution matching, and achieves significantly better unsupervised distribution matching metrics than the implicit baseline approaches. The state based imitation approach requires careful engineering to match end-effector states, and even often fails to find feasible joint configurations while imitating trajectories.
correspondeils
8

---

Under review as a conference paper at ICLR 2022
Table 2: Evaluating Task-Strategy Transfer: Evaluating average rewards 10 episodes obtained via approach with and without finetuning against a hierarchical RL baseline, adapted from Kulkarni et al. (2016), across 3 tasks. Source domain results are shared across approaches, and are trained via Kulkarni et al. (2016).
Domain Pair
Downstream Task
Approach:
Reach (Source)
Reach (Target)
Push (Source)
Push (Target)
Slide (Source)
Slide (Target)
HRL [1]
12.2
12.6
10.7
Bax-R
Ours (Zero Shot)
16.4
19.2
30.1
17.2
21.3
13.4
to Bax-I.
Ours (Fine-tune)
20.1
19.1
18.9
HRL [1]
13.2
11.3
17.2
Bax-L
Ours (Zero Shot)
14.6
18.6
33.4
34.5
20.4
24.6
to Saw
Ours (Fine-tune)
19.1
37.1
38.1
HRL [1]
15.2
12.6
10.7
Bax-R to Saw
Ours (Zero Shot)
16.4
20.8
30.1
28.5
21.3
29.1
Ours (Fine-tune)
29.7
32.5
30.5
4.2 TRANSFERRING TASK STRATEGIES ACROSS DOMAINS
our learnt correspondences capture our first insight, i.e. that different robots use similar skill transfer task strategies across these robots
sequences address similar tasks, it follows that by translating the skill sequence that specifies the task strategy. We evaluate how well the learnt correspondences help transfer task strategies across robots follows.
We consider a set of tasks adapted from Mandlekar et al. (2018), described in the appendix. We then train high-level policies (Kulkarni al., 2016) in the source domain to predict sequences of skills (i.e., the task strategy) to execute on each of these tasks. We then evaluate how well the translation of this task strategy performs on the same tasks in the target domain, both without fine-tuning (i.c., in zero-shot manner). and after fine-tuning for 50 episodes. We evalu- ate this transfer against a hierarchical RI. baseline, i.e. high-level policy trained the target domain over 500 episodes, and report our results in table 2. We also provide visualizations of the rollouts from the original and translated strategies in https://sites..oogg http://ittss.ooole.cooo www.gool...ovvvi//errr.a.oaa hite/rranslaiiggroooskklllss
Task Strategy Transfer Setup:
Analysing results task strategy transfer: We observe that the task strategies translated by our method are able to achieve appreciable task performance across domain pairs and tasks, even without fine-tuning these strategies. Without line-tuning, we that the translated task strategies follow a semantically reasonable sequence of skills given the target task, achieving an appreciable task reward, but often reach slightly differing goal states than desired. Given translation model only observes skill encodings, and no additional state information, this is expected. Upon fine-tuning these translated task strategies in the target domain, we observe a consistent increase in task performance, since fine-tuning allows for picking slightly different (but semantically similar) skills that reach more appropriate goal locations, etc. In contrast, hierarchical RL on the target domain is often slow to converge, and must randomly explore appropriate skill sequences. Even with fine-tuning fine-tuning our approach needs roughly 10 times less training episodes to superior performance than that of the hierarchical baseline. These results suggest that our approach does indeed learn correspondences that facilitate transferring task strategies across robots.
CONCLUSION
We introduce purely unsupervised approach learn skill correspondences across different morpho- logical robots from different domains. Our approach leamss semantically meaningful correspondences across robot domain pairs, and helps transfer task-strategies across domains, without the need for any fine-tuning, despite being completely unsupervised. We believe our approach could enable learning of correspondences across more general temporal abstractions, such as between skills and language instructions, or between skills and human video demonstrations. We hope this work helps enable robots learn from data from heterogenous robots in different domains, and helps further the state of robot learning from demonstration.

---

Under review as a conference paper at ICLR 2022
REFERENCES
M. Abdul-Massih, I. Yoo, and B. Benes. Motion style retargeting to characters with different mor- phologies. Computer Graphics Forum, 36(6) 99, 2017. doi: http:///oi..rrg/10.111111 12860. URL https:://onnnneliiaattt/e//
wwllww.weeey.......... com/ doi /acii/aa///ae http://111111..ff11.
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung Daniel Cohen-Or, and Baoquan Chen. Skeleton-aware networks for deep motion retargeting. ACM Trans. Graph., 39(4), July 2020. ISSN 0730-0301.
doi: 10.1 041144//33666993332240
URL https://dbi.org////
114 1145/3386569
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E. Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. pp. 2504-2510, 2015.
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan gan: Unsupervised video retargeting. 2018.
Alexis Conneau, Guillaume Lample, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. CoRR, abs/1710.04087,, 2017. URL org/abs////// / 1710. 04087.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. pp. 217692176, 2017. doi: 10. 0009ICRRA20017.78992550 URL https: //w//////////o//// wowooooorgg1001 wtw.//////.....//......11111111. ICRA. 2017. 7989250.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. 2019. URL https: htt.hhtpsss/oppeeiiiiiieeo /ttp///ppp//veeeeeeeeeeeeeeeeeet nwt.t//wwweeee.ee..e.......ee. net//orum? forum? id=SJx63 RRF..
Haoqiang Fan, Hao Su, and Leonidas Guibas. A point set generation network for 3d object reconstruction from single image. July 2017.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv:1703. 1ee.1703..8220 08294, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Paseal Germain, Hugo Larochelle, François Lavio- lette, Mario March, and Victor Lempitsky. Domain-adverse advessriall training of neural networks. Journal of Machine Learning Research, 17(59); 2016. URL http: /mmrr. org prggppapess//1// 15 239 html.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. pp. 20.22622-200 2680, 2014.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. In International Conference on Learning Representations, 2019. URL htttps://operrevier. net wttp://emmmddd111eememmmm
Abhishek Gupta, Coline Devin, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning in- variant feature spaces to transfer skills with reinforcement learning. 2017. URL https: // //openreview. net forumm higiiddddgggll..eee.
Chris Hecker, Bemnd Raabe, Ryan W. Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. Real-time motion retargeting to highly varied user-created morphologies. 2008. doi: 10.1145/139954.1333 00155//399904.306266 URL https:/ /oiiooiooo////////oo org/10 1145/1399504 1165504..366622 1360626.
Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphological transfer. pp. 4159-4171, 2020.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefamo Ermon. Domain adaptive imitation learning. arXiv preprint arXiv: 1910.00105, 2020.
10

---

Under review as a conference paper at ICLR 2022
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstrac-
tion. In Advances in Neural Information Processing Systems 32, pp. 11566
11575. Curran Associates, Inc., 2019. URL
wwweersnnisse cc/ cttpaper/ppp//////a/a/// paper/
1.32-variations
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv:1412.6980 2014.
arXiv preprint
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzakee Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In ICML, 2019.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous 2017.
options for robot learning from demonstrations. arXiv preprint
Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi, and Joshua B. Tenenbaum. Hier- archical deep reinforcement learning: Integrating temporal abstraction and intrinsic motiva- tion. In Proceedings of the 30th International Conference on Neural Information Processing Systems. NIPS' 16, pp. 3682-3690, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Guillaume Lample. Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. 2018. URL https: /tttpss///peneevieee. net/ forum? d=
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation, 2019. URL https://
openreview net/ Poruumii-yyyussAq1.. rddtti rdxHFDr:
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- imitation learning. 2020. URL https:
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, 2018
Andrew N. Meltzoff and M. Keith Moore. Imitation of facial and manual gestures by human neonates. Science, 198(4312):75-78, 1977. doi: 10.1 198.:/ccienee....31111111
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun- Jun-Yan Zhu. Contrastive learning for conditional image synthesis. 2020.
Yannick Schroecker and Charles L Isbell. State aware imitation learning. 2017.
30.
URL
wwangg.neuuiii neurips. iss. cc/ http:c/ae////211//11.... Pap.et.7---pp.p......
08e6bea8e
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.003477 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. pp. 86-96, August 2016. doi: 10.18653/v1/P16-1 1/P0//009... URL https:
/ //aclannholooyy.... wrw.:g/..rrrpp.... wtw.///ppp1000000
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. Proceedings of International Conference in Robotics and Automation (ICRA).
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational infer- ence. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings Machine Learning Research, pp. 8624 8633. PMLR, 13-18 Jul 2020. URL https: https:://roccet ngs ml w.w.r..p.e///ee.........oe v11 19//999//// http://////aaaaaaaaaaaa hh0nkar00b htmm
11

---

Under review as a conference paper at ICLR 2022
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2020. URL htttps://ppereevee..oo net Eorum? werrrnmmiidikgioOoo-
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. 2020. URL htttt::s//openreveeww net forrmmrrmm iid= HJęLZR4KvH.
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In CoRL, 2018.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. sual imitation learning via decoupled hierarchical controller. pp. 2593-2603, neurips wwtppppee//1.///aaee.eo /
Third-peree person vi-
URL sttps://proceede d 770ccff00cccc555222eeeeeee -bbsttiraaaeeaaararaaaaat....aae html.
2019.
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. AVID: learning multi-
stage tasks via pixel-level translation of human videos. 2020. doi: 10.15607/RSS.2020) XVI.00 10. 10......11///////......
URL https: //ts:://do/ htt./oi.oooo
2020.022220222 XVI 024.
Ruben Villegas, Jimei Yang. Duygu Ceylan, and Honglak Lee. Neural kinematic networks for unsupervised motion retargetting. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8639- 8648. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00901. URL http: htttt:::///peeaccessss thecvf..e.... com/ wotmm//otteetttrrrtccooee (al__2018hhttt 8/hmm// 20app_eerthpppp.ttttmmmm html.
Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the
cycle-consistency of time. pp. 2566-2576, 2019. URL
doi: 10.1009/CVPR.201.00222
thecvf httpr//...cooooceecccctecee Seeering.ocrrpppeeeeeeeeeeeeertn 201 9 1al.9_apee.htte html.
Qiang Zhang, Tete Xiao, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross- domain correspondence for control with dynamics cycle-consistency. 2021. URL https:
//openreview.wrt www.eeeiwwwweeeeeeeeeeem
Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. Density matching for bilingual word embedding. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, USA, June 2019. URL attps://arxiv. owtpiii.orr//bb.a..
0204..2343.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using sycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 NETWORK ARCHITECTURES
We parameterize the various functions involved, namely the source and target domain encoders and lew-level policies and respectively, and the learnt translation model Tsr neural networks, with following specific architectures.
1. Variational encoders q: In each domain, the variational encoder q is parameterized as a 4 layer LSTM network, with a hidden size of 48 and ReLu activation layers. We further use an input layer from the appropriate input dimensions prior the LSTM, and two output layers to predict Gaussian mean and variance of the skill encodings predicted by the variational encoders.
2. Low-level policies т: In each domain, the low level policies π are parameterized as 3 layer LSTM networks, with hidden sizes of 48 and ReLu activation layers. As above, we use appropriately sized input layers prior to the LSTM, and two output layers after the LSTM to predict Gaussian mean and variance of the low-level actions output by the policy.
3. Translation model T. .r The translation model is parameterized hidden size of 48 units, and ReLu activation layers. We similarly use two output layers to predict Gaussian mean and variance of the translated skill encodings 28--7.
simple 4 layer MLP, with
4. Latent Skill Encoding Dimension: We use skill encoding dimension of 16 across all domains/ / robots.
5. RL High-level policy:
While training RL, we parameterize the high-level policy as a 4 layer MLP, with a hidden size of 48 units, and ReLu activation layers. As above, two output layers to predict Gaussian mean and variance of the predicted skill encodings, in whichever domain we train the policy in.
A.1.2 TRAINING DETAILS
While training the variational encoders and low-level policies, follow the training procedure specified by Shankar & Gupta (2020). We then freeze the variational encoders and low-level policies obtained, before training translation models. While training translation models, we simply optimize C = Cy + La using the Adam optimizer (Kingma & Ba, 2014), implemented in Pytorch. We use the default parameters of Adam, i.e. learning rate 10-4
A.1.3 RL TRAINING DETAILS
While training the downstream RL, we implement a hierarchical version of Proximal Policy Opti- mization (Schulman et al., 2017), by adapting Achiam (2018). This mirrors the hierarchical RL setup of Kulkarni al. (2016). We fix the low-level policies provided to the algorithm, and only train the high-level policies that predict the skill encodings fed into the low-level policies. The hierarchical RL baseline Kulkarni et al. (2016) is trained for 500 episodes, which we found to sufficient for algorithm's performance to saturate. The fine-tuning approach was allowed a budget of 50 episodes to adapt the translated task strategy. The results were evaluated against a fixed random seed of
1.1.4 RL TASKS
We train on the following set of tasks adapted from Mandlekar et al. (2018). particular, we create instances of these tasks on each of the 3 different robots. The only differences between the environments across different robots comes in the form of different initialization states for the objects concerned,
1. Reach: The robot must execute a sequence of skills to reach a block placed on the table. The robot gets a shaped reward based on the distance from the block, and a binary reward upon reaching within threshold distance of the block.
2. Push:
The robot must execute a sequence of skills to push a red block and a green block together. The reward received has shaped component based the distances of the end effector from the
13

---

Under review as a conference paper at ICLR 2022
red block, well as binary reward based on whether the blocks are within a threshold distance of one another.
3. Slide: The robot must execute a sequence of skills push a red block within a threshold distance of a green block placed farther away. The reward received is based on the distance of the end effector from the red block, as well as binary reward based on whether the blocks are within a threshold distance of one another.
A.1.5 HYPERPARAMETERS
We provide a list of the various hyperparameters use during training of the translation model itself, and the downstream RL training.
1. Number of GMM Kernels Ng and N,.: For each of the forward and backward GMMs, we use 500 kernels to parameterize the GMM.
2. GMM Kernel variance o²: For both forward and backward GMMs, we Gaussian Kernel variance of 0.5.
3. Relative weighting of forward and backward losses: We weight the forward and backward losses equally during training.
4. Learning Rate: For training our translation model, we use the Adam optimizer with a learning rate of 10-4.
5. Batch Size: For our training, use a batch size of 32.
6. Number of iterations: We train our translation models over 8000 epochs for each domain pair.
7. Random Seed: We set the random seed for our training to manually.
8. Epsilon Noise: We add in epsilon noise to our training during sampling from the learnt networks.
Here, we use an initial epsilon value of 0.3, and decay the epsilon value to 0.1 over 200 epochs.
While training downstream RL, we adopt the following hyperparameters:
1. Random Seed: When training downstream RL, we report results across 3 different seed values,
0, 1,
2. Epsilon Noise: We follow an epsilon-greedy exploration process during RL training, and use an initial epsilon value of 0.7, and decay the epsilon value to 0.3 over 200 epochs.
3. PPO Parameters: We follow the default PPO parameters used in Achiam (2018).
A.2 ASSUMPTIONS OF LEARNING
1. Assumptions of datasets: As stated in main paper, our choice of robots is dictated by the availability of demonstration datasets on a particular robot. Further, the demonstration datasets ideally also exhibit the following traits. The demonstration datasets need be directed., i.c. they need to contain demonstrations of the robots solving set of tasks, rather than containing random play data in an environment. While learning correspondences of skills from such play data is difficult. This is because our approach relies on directed sequences of skills to learn correspondences. Random play data often contains random sequences of skills, and so observe skills in contexts that they do not originally occur. This misleads our approach, which seeks to exploit context to learn correspondences.
possible to learn skills from such play data,
2. Assumptions about learnt skill spaces: As stated the analysis of our methods translations of skills, our model is able to learn good coarse correspondences, between clusters of skills in the original skill spaces. One trait of the original skill spaces that allows for this is the disentangled representation of the skills afforded by the skill-learning pipeline. Since similar semantic skills that occur similar contexts are placed in similar clusters the original skill space, this allows our approach to learn correct correpsondences between clusters of skills.
A.3 ABLATION STUDIES
To accurately assess our contribution, we would also like to quantify how much each the following components in approach contribute to successfully learning correspondences:
14

---

Under review as a conference paper at ICLR 2022
1. The forward objective, S.-
2. The backward objective, Ls.
3. The use of sequential information, i.e. matching skill-tuple distributions as opposed to matching skill distributions.
4. Learning translation model across frozen skill spaces, opposed directly optimizing the skill representation itself.
We do so by removing each of these components individually, while keeping the rest of the method as is, and train our approach on all 3 domain pairs. We observe the following.
1. Without forward objective Ly, i.e., only optimizing the backward objective La. we observe the translation model is able to cover each of the modes of the target skill-tuple by translated source skills. However, since there is no objective that encourages how realistic the translated source skills are, we observe that there are several modes of translated skills that are not observed in the target skills space. This results in spurious correspondences. For example, this translation model learns to map reaching skills z's that are decoded into random jerky motions on the target robot, that are out of distribution for the target robot decoder.
2. Without backward objective Ls: While only optimizing the forward objective L,, we observe that each of the translated skills appear very realistic with respect to the target domain, i.e. each of the translated skills look like skill in the target domain. However, these models end up suffering from the same issue as the GAN based approaches or the domain adversarial style training, where several modes of the target skill space go unrepresented by target skills. This limits the ability of the learnt correspondences to effectively transfer strategies or represent source motions, since many of the target robot skills are never chosen by the translation model. Together, these ablations indicate the importance both directions of our approach in learning successful correspondences.
3. Without sequential information: We can also optimize our objective defined in terms of skills themselves, rather than skill-tuples. While matching skill distributions, as opposed skill-tuple distributions, we observe that the densities of skill distributions can be matched well by our objective. However, without access to sequential information, the model learns correspondences that often wrong. For example, it learns map reaching skills on the Baxter right hand to returning skills on the Baxter left hand, and vice versa. Skills in different contexts are often mapped to each other, since there no contrary sequential information that the model has access to to suggest otherwise. This suggests that as described our main paper, learning with access to sequential information and matching skill tuple distributions is also key the success of our approach.
4. By directly optimizing the skill-reprsenntations, rather than training a translation model with fixed skill spaces: Instead of freezing the source and target skill spaces, we can also directly optimize the skill representations based on our objective. One may also think of this as maintain- ing translation model that is simply an identity function. When allowing either both of the source or target skill spaces to be finetuned, we must also optimize the original reconstruction style objective that is used in Shankar & Gupta (2020). Despite this, we observe a collapse of the skill space into unimodal distribution, that is both unable to reconstruct skills in either domain, and be mapped across domains. This suggests freezing the representations and training separate translation model is key to our approach.
15


Reviews:
[{"review_id": "iGbhxhnatxz", "reviewer": "Reviewer_rKBg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written clearly and is well structured. The contribution is presented in a clear way, the technical formulation is sound and claims are supported by experimental results.\nThe visual representations (diagrams, figures and videos) are clear and help understand the contribution. \nImplementation details are provided and comparisons with alternative methods allow to appreciate the benefits of the proposed method.\nIt could be argued that the reach skill is not complex enough to appreciate the accuracy of the proposed method in translating from one domain to another. This is partly captured by the results in table 2. Can you discuss these in more detail? In particular, can you explain the cases where the reward in the target domain is greater than that in the source domain?\nHow do you expect your method to transfer to real robots?\n", "summary_of_the_paper": "This paper frames learning skill correspondences as a problem of matching distributions of sequences of skills across robots, and presents an unsupervised objective that is able to learn semantically meaningful correspondences between skills across diverse robot domains.\n", "summary_of_the_review": "The paper presents a method that is well formulated, sound and well motivated. The experimental results are convincing but require some further explanation. Details of implementation and experimental results are provided, so that it could be possible to reproduce the results.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T1m6RJwPG2v", "reviewer": "Reviewer_rjam", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe idea of being able to reproduce skill sequences between different robots based on videos or visual demonstrations is a very hard problem with potentially very impactful applications!\n\nIn this current work, I am not quite sure what is considered as morphologically different? Does it refer to different kinematics or robots with different DOFs?\nDepending on this definition, the assumption: “different morphological robots use similar task strategies (in terms of sequences of skills) to solve similar tasks” might not always hold. Although the high-level of semantic skill sequences might be the same, the actual trajectories corresponding to those skills can be significantly different.\n\nIt would be useful to clearly emphasise the contributions of the approach, as many parts rely on TVI introduced in Shankar & Gupta (2020).\n\nI am not sure how Eysenbach et al. (2019) and Sharma et al. (2020) are related to the presented approach. However, I think there are several older related works, which would help provide context and give motivation to the proposed approach, such as:\n- Movement primitives:\n  - Kober, J. and Peters, J. Learning motor primitives for robotics. ICRA, 2009.\n  - Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., and Schaal, S. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 2013.\n- Movement scaling/retargeting:\n  - https://mediatum.ub.tum.de/doc/1192176/file.pdf\n  - https://hal.archives-ouvertes.fr/hal-02398106/document\n  - https://hal.archives-ouvertes.fr/file/index/docid/1054887/filename/Sakka2014.pdf\n  - https://www.worldscientific.com/doi/abs/10.1142/9789814623353_0008\n\nAs well as work on imitation learning, learning from demonstrations and hierarchical RL, similarly as it is discussed in the related work of Shankar & Gupta (2020).\n\nIt would also be beneficial to show the performance of the proposed approach in comparison to simple skill transfer or learning from demonstration approaches as a baseline.\n\nText around the images in Fig2 is too small and difficult to interpret, maybe some alternative way of displaying it would aid visibility\n", "summary_of_the_paper": "This paper addresses the problem of transferring skills between morphologically different robots. This approach to learning skill correspondences is framed as a problem of matching distributions of sequences of skills across robots. The paper proposes an unsupervised objective, inspired by work in unsupervised machine translation, that makes the skill translation model learn to match the distribution of skill sequences. The proposed approach is experimentally evaluated on 3 transfer settings in a simulated robot environment.", "summary_of_the_review": "- Clarify contributions and assumptions\n- More discussion on the limits of the approach w.r.t. robot morphologies.\n- Update related work discussion\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KtUAR4N82bA", "reviewer": "Reviewer_KQQm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper avoids the need to manually annotate datasets with low-level correspondences between robots, which requires expertise when dealing with robots with different morphologies. Instead they provide correspondences over skills, which is much easier to specify, even by non-robotics-expert users. \n\nTheir experiments show superiority over baselines from the literature. \n\nWeaknesses:\n\nAlthough the Baxter and Sawyer have different dynamics, their kinematics are not that different. Yes, the link dimensions are different but they have the same number of degrees. I would have loved to see experiments on robots with different number of degrees of freedom as that is more challenging. The argument that the choice of robots is dictated by the availability of the datasets is kind of weak given experiments are run in simulations. It can't be hard to generate a similar dataset for other robots. \n\nI would have loved to see an ablation study on the quality of skills across the robots. Surely, the data generation process needs to make sure there is some alignment of tasks across the robots. How much of this has an effect on the quality of the translated skills?\n\nI would have also loved to see an analysis of the training time of the translation model in relation to the time it takes to learn the skill from scratch with the target robot.\n\nOne important aspect of transfer that is usually ignored, is the cost of transfer compared to learning from scratch. And the cost could be the time it takes to collect data to learn the translation model compared to learning the skills from scratch using the target robot. Of course, using data already existing avoids this but is still a problem for robot for which data doesn't exist. See [1] as an example, albeit in a different domain.\n\nIt's not clear if the translation model is evaluated on unseen skills or trajectories. My guess is no. How well does it generalize to unseen skills?\n\nIn Table 2, how come rewards in the source domain are higher than in the target for the HRL baseline? Especially between Bax-R and Bax-L. How different are these tasks when performed with either the right or left hands?\n\nMoreover, why should we expect the translated trajectories obtain higher rewards in the target domain than in the source domain? Especially between left and right hands.\n\n1. Makondo, Ndivhuwo, and Benjamin Rosman. \"Towards improving incremental learning of manipulator kinematics with inter-robot knowledge transfer.\" 2019 Southern African Universities Power Engineering Conference/Robotics and Mechatronics/Pattern Recognition Association of South Africa (SAUPEC/RobMech/PRASA). IEEE, 2019.", "summary_of_the_paper": "The paper proposed to learn skills correspondences between robots of different morphologies in an unsupervised way - without requiring paired data from the robots. They learn a skill translation model that maps skills from a source robot to a target robot. The translation model is learned by minimizing a loss composed of two components: (1) the likelihood of the translated skills under the target distribution estimated from target robot skills data, and (2) the likelihood of the target skills under the translated distribution estimated from the translated skills data.\n\nThe proposed approaches hinges on the assumption that robots with different morphologies follow similar high-level strategies when performing similar tasks. So, a sequence of tasks across the robots is used to learn the skills correspondences. ", "summary_of_the_review": "The paper proposes a novel idea of learning skills correspondences between robots with different morphologies in an unsupervised way, borrowing ideas from machine translation. The contributions are solid, with a few minor evaluation weaknesses as highlighted in the main review above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-Sei8gg9Mn", "reviewer": "Reviewer_dwTB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n+ The idea of transferring skills between robots with different morphologies is interesting. \n\n+ Presentation of the technical approach is clear.\n\nWeaknesses:\n\n- What are the theoretical novelties? The novelty of the paper is not clearly discussed and justified in the paper. The idea of using forward and backward losses in the objective function is widely used to maintain consistency in domain adaptation and transfer learning. \n\n- The insight/premise that differently morphological robots follow similar strategies to address similar tasks seems questionable. For robot manipulation, this can be true. But for robot locomotion as an example, this assumption may not hold: robots can perform locomotion by walking, crawling, hopping, and wheeling, based on the morphology of the robot. How does the premise hold in general robotics tasks?\n\n- The terms (skill, strategy, and mode) are confusing, and it is hard to imagine these terms in the context of a robotics application. Can the authors provide a concrete scenario of robots using a sequence of skills to complete a task? What are the examples of skills, strategies, and modes in that concrete scenario?\n\n- What is the runtime (or computational cost) of the proposed approach and other compared methods? How many iterations are used for the algorithm to converge? The paper argues efficiency several times, but does not provide clear theoretical analysis or experimental support. \n\n- How to segment a sequence of observations into skill tuples? How the start and end points of each skill are determined when segmenting a long sequence?\n\n- “Analysing qualitative skill translations” in the experimental section 4.1 has no experimental support. No experimental results are provided to justify the statements in this subsection.\n\n- It is not convincing to argue the left arm and the right arm of a Baxter robot have a different morphology in the experiments. The Sawyer robot and the Baxter robot’s arms also have similar design and configurations. In the application of robot manipulation, suction cups, soft grippers and traditional mechanical grippers can be considered as morphologically different robots, or maybe robot arms with obvious differences, e.g., in terms of degrees of freedom. How the proposed approach can be generalized to transfer skills among these robots with obvious morphological differences?\n", "summary_of_the_paper": "This paper presents an unsupervised learning method to identify skill correspondences for translating skills between robots with different morphology. The approach is inspired by unsupervised machine translation, and uses an unsupervised objective to learn a skill translation model to match the distributions across domains.", "summary_of_the_review": "While the idea of transferring skills between robots with different morphology is interesting, the paper is preliminary and will need improvements on explaining its novelty, premise, and terminology; more convincing experiments using robots with obvious morphological differences are also expected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an algorithm for unsupervised skill transfer between robots with different kinematics. Integral to the approach is the idea that while the robots differ, they may use similar strategies to perform similar tasks. Without access to paired data, the paper formulates the problem of learning correspondences between robots as one of matching skill distributions across robots. Drawing insights from work in machine translation, the paper proposes an unsupervised objective that encourages the model to learn to align the distribution over skill sequences. Experimental results demonstrate the ability to use learned skill correspondences to support transfer across different robots in different domains.

As several reviewers point out, the problem of learning to transfer skills across robots with different kinematic designs from video demonstrations raises a number of interesting challenges that are relevant to the robotics and learning communities. Among them, a fundamental contribution of the paper is the ability to learn skill correspondences in an unsupervised manner based on unlabeled demonstrations. The approach by which this is achieved (i.e., using distribution matching) is sensible and clearly described. While the reviewers agree on the significance of the research problem, they raise a few key concerns regarding the initial submission. Among them are questions about the nature and extent of the domain variations that the method can handle (e.g., between robots with different degrees-of-freedom); the significance of the contributions; and how this work is situated in the context of existing approaches to robot skill learning. Several reviewers question the definition of morphological variation and comment that these variations may violate the assumption that task strategies are similar across designs. The authors provided detailed feedback to each of the reviews, which helps to clarify several of these concerns. Unfortunately, several reviewers did not respond to multiple requests to update their reviews. The one who did decided to maintain their score.

The paper tackles an important problem in robot learning and the work has the potential to have significant impact on the way in which robots acquire new skills. The original submission together with the author responses suggest that there is are solid contributions here. The authors are strongly encouraged to revisit the discussion of the approach to more clearly convey the novelty of the approach and to consider experimental evaluations that better support these claims.
Decision: Reject