TRANSLATING ROBOT SKILLS: LEARNING UNSUPER-
Year: 2021

Abstract:
We explore how we can endow robots with the ability to learn correspondences between its own skills, and those of morphologically different robots in different domains, in an entirely unsupervised manner. Our insight and premise is that morphologically different robots use similar strategies (high-level skills sequences) to solve similar tasks. Based on this insight, we frame learning skill correspondences as a problem of matching distributions of sequences of skills across robots. We then present an unsupervised objective that encourages a learnt skill translation model to match these distributions across domains inspired by recent advances in unsupervised machine translation. Our approach is able to learn semantically meaningful correspondences between skills across 3 robot domain pairs despite being completely unsupervised. Further, the learnt correspondences enable the transfer of task strategies across robots and domains. Dynamic visualization of our results can be found here: https://sites.google.com/view/translatingrobotskills/home

heading: 1 INTRODUCTION
text: Humans have a remarkable ability to efficiently learn to perform tasks by watching others demonstrate similar tasks. For example, children quickly learn the skills needed to play a new sport by watching their parents perform skills such as kicking a ball. Notably, they are able to learn from these visual demonstrations despite significant differences between themselves and the demonstrator, including visual perspectives, environments, kinematic and dynamic properties, and morphologies. This ability may be attributed to two factors; firstly, humans have well-developed basic motor skills that we can execute with little effort. Second, we can recognize the sequence of skills (or the high-level strategy) the demonstrator uses, and understand a corresponding set of skills that we can execute ourselves (Meltzoff & Moore, 1977).
In this paper, we explore how we can endow robots with this ability - i.e., to adopt task strategies from morphologically different robot demonstrators, and then execute corresponding skills to solve similar tasks. Key to solving this problem is for the robot to identify how its owns skills correspond to those of the demonstrator, which is tremendously powerful. First, it allows the robot to adopt the demonstrator’s strategies for solving various tasks. Second, by adopting and adapting these
strategies for itself, the robot can efficiently learn to solve a variety of tasks previously outside its repertoire. Finally, it allows us to understand the task strategies used by various robots in a unified manner, enabling robots to learn from data collected with a heterogeneous collection of tasks and robot morphologies. Skills provide a natural framework to facilitate such concise knowledge transfer across robots compared to low-level robot controls; skills inherently abstract away low-level details that may differ across the demonstrator and the learner, and instead focus on the commonalities between them, such as the task strategy. For example, skills such as reaching and placing on robot manipulators abstract away differences in morphologies or configuration, and are thus well grounded across robots, while there may not be obvious correspondence in their low-level actions.
How can one acquire correspondences between skills? Stated more formally, given morphologically different robots, a set of unlabelled demonstrations of each robot solving a variety of tasks, and a method for extracting skills from those demonstrations, how can one learn correspondences between the skills of the different robots? Learning such correspondences is straightforward when supervised pairs of skills or trajectories are available. However, collecting and annotating such data is timeconsuming and tedious, and requires a high degree of human expertise, particularly at the scale necessary to successfully learn correspondences across a diverse set of skills. In contrast, we explore whether we can learn such skill correspondences in an unsupervised manner, i.e., without access to supervised skill or trajectory pairs. Learning unsupervised skill correspondences potentially allows for learning a broad scope of correspondences from the diverse unlabelled and unsegmented data newly available in robotics (Sharma et al., 2018; Mandlekar et al., 2018). Unfortunately, learning unsupervised correspondences is a difficult problem to solve - without supervision, it is difficult to guide learning towards the right correspondences. While this may be alleviated to some extent by incorporating unsupervised constraints (Zhu et al., 2017; Ganin et al., 2016; Zhou et al., 2019), there are many spurious correspondences that satisfy these constraints.
How can one then go about learning such skill correspondences in an unsupervised manner? We propose leveraging the two following insights to do so. Our first insight is that different morphological robots use similar task strategies (in terms of sequences of skills) to solve similar tasks; in other words, the sequences of skills executed by different robots to solve similar tasks ought to belong to similar distributions. We observe that this is generally true; for example, a robot pouring out a cup of tea would likely first reach for the kettle, grasp it, move it appropriately over the cup, and then begin to pour the tea, irrespective of its exact morphology. Our second insight is that learning skill correspondences without access to supervised data closely mirrors unsupervised machine translation (UMT), where the objective is to learn a translation between representations in different languages (such as between word embeddings across languages), without access to parallel data (Conneau et al., 2017; Lample et al., 2018). Inspired by these two insights, we derive an unsupervised objective to guide our learning towards meaningful skill correspondences.
We approach the problem of learning unsupervised skill correspondences by learning a translation model to map from the skill space on one “source” robot to the skill space on another “target” robot. We construct an unsupervised objective that encourages the translation model to respect our first insight - i.e., to preserve the sequences of skills observed across both the (translated source) and target robots. To do this, we take inspiration from our second insight, and specifically from Zhou et al. (2019), and construct explicit probability density models over the sequences of skills observed on the source and target robots, and then train the translation model to match these distributions.
We evaluate the ability of our approach to learn unsupervised skill correspondences across three different robots, the Sawyer robot, and the left and right hands of the Baxter robot. Our approach is able to learn semantically meaningful correspondences across each of the 3 pairs of robots, despite being completely unsupervised, as depicted in fig. 1. The learnt correspondences facilitate transferring task strategies across across domains, as we demonstrate on a set of downstream tasks. Our results are visualized at https://sites.google.com/view/translatingrobotskills/home.

heading: 2 RELATED WORK
text: Skill Learning: Eysenbach et al. (2019); Sharma et al. (2020) both address unsuperivsed skill learning from interaction data, by constructing information theoretic approaches. Fox et al. (2017); Krishnan et al. (2017); Shankar et al. (2020); Sharma et al. (2018); Shankar & Gupta (2020); Kipf et al. (2019); Gregor et al. (2019); Kim et al. (2019) instead learn skills or abstractions from unlabelled
demonstration data by performing latent variable inference. These frameworks all learn skills in the context of a single domain, while we seek to learn correspondences of skills across domains.
Unsupervised Correspondence Learning: Our problem bears a close resemblance with unsupervised machine translation (Conneau et al., 2017; Zhou et al., 2019), and unpaired image translation (Zhu et al., 2017; Park et al., 2020). Specifically, they all share the notion of learning correspondences across representations learnt from unpaired, “monolingual” data:
• Unsupervised Correspondence in Machine Translation:i i i l ti :s r s rr s r se e e e ce ac e a ai i i l ti :r rr r Conneau et al. (2017) leveraged domainadversarial training (Ganin et al., 2016) to align word embeddings of two languages. Sennrich et al. (2016); Lample et al. (2018) use the idea of back-translation to constraint the learnt translation models across languages. Zhou et al. (2019) learn bilingual word embeddings by matching explicit density functions over the word embedding spaces across languages.
• Unsupervised Correspondence in Image Translation:i i l ti :s r s rr s I r se e e e ce a e a ai i l ti :r rr I r The vision community has similarly explored the unpaired image-to-image translation setting (Zhu et al., 2017; Park et al., 2020), using cycle-consistency losses (Zhu et al., 2017) or contrastive losses (Park et al., 2020).
• Unsupervised Correspondence in Video:i i i :s r s rr se e e e ce ei i i :r rr Bansal et al. (2018); Wang et al. (2019) extend CycleGAN (Zhu et al., 2017) to the video domain, by incorporating temporal consistency losses.
Domain Transfer in Robotics and Graphics: The robotics and graphics communities have taken interest in cross-domain transfer of policies in recent years:
• Policy Transfer with Paired Data:li it i t :r sf r rc a e a e a ali it i t :r f r r Gupta et al. (2017); Sermanet et al. learn morphology and viewpoint invariant feature spaces for policy transfer respectively, but require paired data to do so.
• Policy Transfer via Modularity:li i l it :r sf r rc a e a ali i l it :r f r r Hejna et al. (2020); Devin et al. (2017); Sharma et al. (2019) address morphological transfer by modularity in their policies. Hejna et al. (2020); Sharma et al. (2019) adopt modularity in a hierarchical sense, but leverage common grounding of subgoals across morphologies to perform transfer. We do not assume access to such common grounding.
• State based Policy Transfer:t t li :s r sf ra e a e c a et t li :r f r Liu et al. (2020); Schroecker & Isbell (2017); Ammar et al. (2015) address cross-morphology transfer by state based imitation learning.
• Motion Retargetting:ti t tti :re a eti t tti :r The graphics community has addressed transferring behaviors across morphologically different characters (Hecker et al., 2008; Aberman et al., 2020; Villegas et al., 2018; Abdul-Massih et al., 2017), using carefully handcrafted kinematic models. These works transfer behaviors by imitating joint positions, and performing inverse kinematics to retrieve the full character state, which is often infeasible for widely different morphological characters.
• Unsupervised Action Correspondence:i ti :s r s rr se e c e e cei ti :r rr Zhang et al. (2021); Kim et al. (2020); Smith et al. (2020) address learning low-level state and action correspondences from unpaired, unaligned interaction and demonstration data respectively. While similar in spirit, our work argues that learning highlevel skill correspondence instead is a more natural choice, as mentioned in the introduction.
Applicability to learning skill correspondences: While successful in their respective problem domains, existing approaches are not trivially applicable to our problem. The adversarial training used in most of these approaches is notoriously difficult to train, and is prone to the mode dropping problem (Li & Malik, 2019). They require strong constraints such as pixel-wise or joint-wise identity losses (Zhu et al., 2017; Zhang et al., 2021), or inherent similarity of the spaces to be aligned, such as word embeddings across languages (Conneau et al., 2017; Zhou et al., 2019). Learnt skill representations do not possess this property in general. Shortcomings notwithstanding, these approaches provide insight into how we may pursue learning unsupervised skill correspondences.

heading: 3 APPROACH

heading: 3.1 PRE-REQUISITES
text: We begin by first reviewing an important prerequisite - the skill learning pipeline of Shankar & Gupta (2020), which provides us a learnt representation of robotic skills given an unlabelled robot demonstration dataset. Their method first represents robotic skills as continuous latent variables z, and introduce a Temporal Variational Inference (TVI) to infer these skills or latent variables. TVI trains a variational encoder q(z|τ) that takes as input a robot trajectory τ = {s1, a1, ...sn−1, an−1, sn}, and outputs a sequence of skill encodings z = {z1, z2, ...zn}. Note that these skill encodings repeat over time for the duration of a given skill. TVI also trains a latent conditioned policy π(a|s, z) that takes as input robot state s, and the chosen skill encoding z, and predicts the low-level action a that the robot
should execute. We direct the reader to Shankar & Gupta (2020) for a more thorough description of their skill learning approach.

heading: 3.2 PROBLEM SETTING
text: Consider two robots with potentially differing morphologies and environments (or “domains”), represented as source and target domains, or MS and MT respectively. Associated with each domain is an unlabelled and unsegmented dataset of demonstrations of the robot performing various tasks represented as D
S and D T respectively. D S and D T are collected independently, on an intersecting
(but not identical) set of tasks. This is equivalent to the setting in machine translation without parallel data, with access only to monolingual corpora.
We also assume access to a skill-learning pipeline, that can take in such an unlabelled demonstration dataset D of a robot M performing various tasks, and learns a representation Z of skills on a given robot. We specifically use TVI (Shankar & Gupta, 2020), but we believe our approach is compatible with any unsupervised skill-learning approach that affords a continuous representation space. We train the skill-learning pipeline independently on each of the domain datasets D
S and D T . This
affords us skill encoders q S and q T , and latent conditioned policies π S and π T for each domain. We may then encode trajectories in both domains τ
S ∈ D S and τ T ∈ D T into their constituent skills
{zt S }|τS |t=1 and {ztT} |τ T |
t=1 , via qS and qT respectively. We represent the space of skills learnt on the entire dataset in each of the two domains as Z
S and Z T respectively.
Our goal is to learn semantically meaningful correspondences between skills in the source domain, zS ∈ ZS , and those in the target domain, zT ∈ ZT , that helps solve tasks in the target domain MT . We specify these correspondences by learning a “translation” function TS→T : ZS → ZT , that maps skills in the source domain zS , to translated skills in the target domain zS→T . This mirrors the word translation models present in Conneau et al. (2017); Zhou et al. (2019). Learning meaningful skill-correspondences thus amounts to learning a good translation model T . We can also translate the entire source space Z
S to the target domain, to retrieve the translated source space, Z S→T .

heading: 3.3 USING SEQUENTIAL INFORMATION OF SKILLS
text: Our first insight is that different morphological robots follow similar strategies to address similar tasks - in other words, sequences of skills executed by two different robots ought to belong to similar distributions. It is important to consider sequences of skills here rather than skills in isolation, since the ordering of skills gives us additional information to bias our learning towards the right correspondences. Constraining correspondences based on individual skills alone could lead to incorrect results. For example, learning that “reaching” skills in one domain correspond to “grasping” skills in the other domain is evidently incorrect, and is an error that sequential information could bias learning away from, since grasping skills are systematically executed after reaching skills.
However, processing entire sequences of skills is computationally infeasible for arbitrarily long trajectories. Instead, we draw inspiration from the simple yet powerful bigram models in the NLP community, and consider consecutive pairs, or tuples of skills. While less expressive than the full skill-sequences, these tuples retain enough information of the ordering of skills to guide the correspondence learning towards the right solution, and are far more computationally tractable. We thus construct a skill-tuple space, which represents the various transitions between skills observed in a given domain. Each consecutive pair of skills (zt, zt+1) in the original space Z of a given domain is represented as a single point xt in the skill-tuple space X of that domain. For initial and terminal skills zt=0 and zt=|τ |, we pad these tuples, and treat the preceeding and succeeding skill encodings as appropriately dimensioned 0’s respectively.
The skill-tuple spaces for source and target domains are represented as X S and X T respectively, and are implemented as a set of skill-tuples from each domain, i.e. X
S = {x S,i} N S i=1,XT = {xT,i} N T
i=1. The procedure to construct these spaces is presented in the sub-routine of algorithm 1. We also construct a translated skill-tuple space, X
S→T , that is simply the translation of all skills present in XS , i.e. X
S→T = {xS→T ,i} N S i=1. Translating a skill-tuple x t S = (zt S , zt+1 S ) from source to target is done by
translating the individual skills zt S and zt+1 S to the target domain,zt S→T and zt+1 S→T , and constructing the translated skill-tuple as xt
S→T = (zt S→T , zt+1 S→T ).

heading: 3.4 DISTRIBUTIONAL PERSPECTIVE ON LEARNING TRANSLATION FUNCTIONS
text: We would like our translation model to exhibit two properties - first, to translate source skills z
S such
that they belong to the distribution of target skills, and second, to capture all modes of skills that exist in the target domain. Dropping modes of skills limits the set of skills the target robot can use, reducing its capabilities. GANs (Goodfellow et al., 2014) and domain-adversarial approaches (Ganin et al., 2016) satify the first property since they optimize for how realistic the generated samples are (or how indistinguishable source and target domain features are (Ganin et al., 2016)), but often drop modes of the “real” data (Li & Malik, 2019).
Instead, we approach this problem from an explicit distribution perspective, and maintain explicit probability density estimates over the translated source and target skill-tuple spaces, X
S→T and XT , p(x
S→T) and p(xT) respectively. Following our insight that sequences of skills on different robots ought to belong to similar distributions, we would like these distributions p(x
S→T) and p(xT) to match. We can optimize our translation model to enforce this, by maximizing the likelihood of translated skill-tuples xS→T under the target skill-tuple distribution p(xT), and the likelihood of target skill-tuples xT under the translated skill-tuple distribution p(xS→T). This is similar in spirit to optimizing both the forward and reverse KL between two distributions. We formalize this objective and provide intuition for it below.
Consider constructing an explicit probability density function p T (x), over the target domain skilltuple space X T . We follow Zhou et al. (2019)’s choice of representing p T (x) using a Gaussian Mixture Model (GMM), owing to its expressive power and efficiency in low-dimensional spaces, but note that a wide variety of explicit density estimators may be used here. Here, pT(x) is a GMM with NT Gaussian kernels, each centered at a target skill-tuple x ′ T,i ∼ XT :
p T (x) = NT∑ i=1 p(x′ T,i) p(x|x ′ T,i) = NT∑ i=1 p(x|x′ T,i) (1)
Zhou et al. (2019) use kernel weights p(x′ T,i ) proportional to the frequency of the word the kernel represents. In our setting, each kernel is simply one of the skill-tuples in the continuous target skill space XT , motivating our choice of equal mixture weights over all kernels (i.e. skill-tuples), i.e. p(x′
T,i ) = 1/NT . The mixture components p(x|x′T,i) are specified by a fixed variance Gaussian
centered at x′ T,i , i.e. p(x|x′ T,i ) = N (x|x′ T,i , σ2), where σ specifies the standard deviation of the Gaussian kernel. We can train a translation model to translate skills in such a way that the translated skill tuples x
S→T belong to the distribution of target skill-tuples pT(x), or that they are realistic with respect to the target distribution. We do this by maximizing the “forward” likelihood Lf of the translated skill-tuples x
S→T , under the target density pT(x):
Lf = Ex S ∼p(x S ),x S→T ∼ TS→T (.|xS )
[ log p
T (x S→T) ]
(2)
Algorithm 1 Translating Robot Skills

heading: Input: D
text: S , D T , q S , q T , π S , π T
. Require demonstration datasets, trained skill encoders & decoders for source & target domains

heading: Output: T
text: S→T( . |zS) . Output translation model 1: Initialize TS→T( . |zS) . Initialize translation model 2: XS ← BUILDSKILLTUPLESPACE(Source) . Construct source skill-tuple space 3: XT ← BUILDSKILLTUPLESPACE(Target) . Construct target skill-tuple space 4: p
T (x)← ∑NT i=1 p( . |x′T,i) . Update target GMM density via eq. (1)
5: for i ∈ [1, 2, ..., Niterations] do 6: XS→T ← TS→T( . |XS) . Update translated-source skill-tuple space 7: p S→T(x)← ∑NS i=1 p( . |x′S→T,i) . Update translated-source GMM density via eq. (3) 8: x S ∼ X S , x T ∼ X T . Get batch of source and target skill-tuples 9: x S→T ∼ TS→T( . |xS) . Translate source skill-tuple to target domain 10: Lf ← log pT(xS→T) . Evaluate forward objective 11: Lb ← log pS→T(xT) . Evaluate backward objective 12: L ← Lf + Lb . Evaluate full objective 13: Update T
S→T via ∇L . Update translation model with gradient ascent
Sub-routine: Build Skill-Tuple Space Input: Domain M Output: Skill-Tuple space X
M
14: function BUILDSKILLTUPLESPACE(Domain M )

heading: 15: X
text: M ← {} . Initialize empty skill-tuple space
16: for i ∈ [1, 2, ..., NM ] do 17: τM,i ∼ DM . Retrieve batch of trajectories from domain dataset 18: {zt
M }|τ |t=1 ∼ qM( . |τM,i) . Encode trajectories as sequence of skills via domain encoder
19: {xt M }|τ |−1t=1 ← {(ztM , z t+1 M )}|τ |−1t=1 . Assemble skill tuples

heading: 20: X
text: M ← X M ∪ {xt M }|τ |−1t=1 . Add skill-tuples to skill-tuple space
21: return X M
To encourage the second mode-covering property in the translation model, we can also optimize the likelihood of the target skill-tuples under the translated source skill-tuple space. To do so, we first construct an explicit probability density function p
S→T(x) over the translated source skill-tuple space, X
S→T , that is equivalent to pT(x). pS→T(x) is also similarly represented as a GMM, with NS→T equally weighted Gaussian kernels, each centered at a translated skill-tuple x ′ S→T,i ∼ XS→T :
pS→T(x) = N S→T∑ i=1 p(x′ S→T,i) p(x|x ′ S→T,i) = N S→T∑ i=1 p(x|x′ S→T,i) (3)
As before, the conditional p(x|x′ S→T,i ) = N (x|x′ S→T,i , σ2). We may then optimize the “backward” likelihood Lb of a set of target skill-tuples xT under pS→T(x):
Lb = Ex T ∼p(x T ) [ log pS→T(xT) ] (4)
By combining these two objectives we may construct our final objective L for training the translation model, by simply combining these two objectives, L = Lf + Lb:
L = Ex S ∼p(x S ),x S→T ∼ TS→T (.|xS )
[ log p
T (x S→T) ] + Ex
T ∼p(x T )
[ log p S→T(xT) ]
(5)
Parsing the objective: In contrast with adversarial training methods, which require alternating training phases and stability tricks such as gradient penalties, our objective amounts to simple maximum likelihood (albeit in two directions). It is hence simpler, more stable, and quicker to converge.Intuitively, Lf captures how “realistic” each translated skill-tuple looks with respect to the target skill-tuple distribution, while Lb captures how well the translated skills cover the modes of the target skill-tuple space. We present our full approach for learning unsupervised skill correspondences in algorithm algorithm 1, and a pictorial representation of it in fig. 2.

heading: 4 EXPERIMENTS
text: Robots: We evaluate our approach’s ability to learn skills correspondences across the following robots - the Sawyer robot, the Baxter left hand, and the Baxter right hand. Our choices of robots are dictated by the availability of demonstration datasets for these respective robots. We consider 3 of the possible 6 unique domain pairs - Baxter Right-Hand to Baxter Left-Hand (Bax-R to Bax-L), Baxter Left-Hand to Sawyer (Bax-L to Saw), Baxter Right-Hand to Sawyer (Bax-R to Saw).
Datasets: For the Sawyer robot, we make use of the Roboturk dataset (Mandlekar et al., 2018), which consists of roughly 2000 demonstrations across 8 different tasks. For the Baxter robot, we utilize the MIME dataset (Sharma et al., 2018). We partition the MIME dataset into two disjoint sets with solely left-handed and right-handed trajectories respectively. Each single-handed dataset also roughly contains 2000 demonstrations across 16 tasks. We treat each single-handed dataset as the respective dataset for the Baxter left-hand robot and the Baxter right-hand robot.
Skill Representation: We learn skill-representations for each of the 3 robots independently from their respective datasets, using TVI (Shankar & Gupta, 2020). We use a 16-dimensional skillrepresentation space for each robot, that is learnt from robot joint-states and joint velocities. We then freeze these learnt skill representations for all of our experiments. We follow the preprocessing steps and training parameters specified by (Shankar & Gupta, 2020). We also manually annotate 50 skills in each domain with one of 6 semantic labels of what type of skills they were. We emphasize that our approach is not trained with these labels, but rather is only evaluated against them.

heading: 4.1 LEARNING MEANINGFUL SKILL CORRESPONDENCES
text: The first question we would like to answer is - “Can our unsupervised method learn meaningful skill-correspondences between skill spaces?” We first present qualitative results towards this.
Translating Individual Skills and Entire Trajectories: We present two sets of visualizations; individual skills and their translations, as well as entire trajectories (or sequences of skills) and their translations. We first visualize a set of source domain trajectories τS , obtained by rolling out their skill encodings {zt
S }|τ |t=1 via the source domain policy πS . We then translate these sequences
of skill encodings to the target domain, i.e. {zt S→T }|τ |t=1, and visualize rollouts of the target policy π T
conditioned on these translated skills. We present visualizations of individual skills and their translations, as well as visualizations of entire trajectories and their translations in fig. 1 and at https://sites.google.com/view/translatingrobotskills/home.
Analysing Qualitative Skill Translations: Despite being learned in a purely unsupervised manner, we observe our translation model is able to learn good semantically meaningful coarse skill correspondences between semantic clusters of skills that emerge in the original spaces, across all 3 domain pairs. For example, from fig. 1, we see source domain reaching skills correspond to target domain reaching skills across all 3 domain pairs, as do placing and pushing / sliding skills to the left or to the right, and returning skills to their rest configuration. However, our model is unable to guarantee learning perfect finer correspondences (such as between twisting or grasping skills), or between variations of skills that belong to the same semantic cluster in the original skill spaces (such as reaching with different arm shapes, as in fig. 1). We emphasize that our method is unsupervised, and instead exploits similar contexts of skills across domains to learn correspondences; these results are understandable and expected as a result. Finer skills and intra-cluster variations of such skills are often executed in similar contexts, such as after reaching or before placing skills, making them difficult to disambiguate. We note that despite this, our model often does translate finer skills (such as twisting and grasping) correctly, suggesting its potential for improvement via techniques such as iterative refinement (Conneau et al., 2017), or via additional inductive biases.
Using the learnt correspondences, we are able to transfer the source domain trajectory’s overall structure, and sequence of skills executed, remarkably well to the target domain. Our model does so despite variations in the precise shape of the robot arms, or start and end positions of these respective skills. This provides further evidence that the learnt correspondences are indeed semantically meaningful, and additionally suggests these learnt correspondences would be useful in transferring task strategies across domains. We believe this is a powerful result, especially so considering approach does so in a completely unsupervised manner.
Quantitative Evaluation of Skill-Tuple Distribution Matching: To quantitatively measure how well our approach can match the distributions of skill-tuples across robots, we evaluate the following unsupervised metrics. We first evaluate the forward and backward GMM densities Lf and Lb. We also compute the Chamfer distance (Fan et al., 2017) between the skill-tuple spaces, which represents the nearest neighbor distances across two point sets. Together, these three metrics specify how close the skill-tuple distributions across domains are. We also evaluate the following supervised metrics, using the manually annotated semantic labels associated with 50 skills in each domain. We reiterate these labels are only used for evaluation purposes, and are unseen at train time. First, we evaluate how accurately the learnt correspondences capture semantics of skills across domains, by measuring how well the semantic labels associated with a set of skills in the source domain match with those in the target domain upon translation, reported as the label accuracy in table 1. We also evaluate a supervised Z error - i.e. the average distance in latent space between the translated source domain skills, and the nearest target domain skills with the same semantic label.
We compare our approach against the following alignment baselines, and present the results in table 1. Our first baseline is Domain-adversarial training (Ganin et al., 2016), where we match skill-tuple distributions by training the translation model to fool a discriminator network trained to identify domains given skill-tuples. We also compare against Cycle-GAN (Zhu et al., 2017), where we train two translation models between skills from source and target domains to be cycle-consistent with one another, while also optimizing a domain-adversarial style objective (Ganin et al., 2016). Finally, we also compare against a State-based Imitation approach (Liu et al., 2020), where we transfer trajectories across domains by copying end-effector states across robots, using inverse kinematics to retrieve the closest feasible joint state.
Analysis of Qualitative Results of Skill-Tuple Distribution Matching: From table 1, we observe that our approach is notable able to learn correspondences that achieves a much higher semantic label transfer accuracy, consistent with our approach learning good coarse correspondences. The baseline approaches in contrast, are only able to achieve random-level label transfer accuracy, indicating their inability to learn correct correspondences. For example, the domain adversarial baseline ends up mapping several different types of skills in the target domain to single modes of skills in the target domain, across all 3 domain pairs. The Cycle-GAN baseline somewhat mitigates this approach, but takes a shortcut in the learning and simply places a single source skill nearby every mode of target skill. In contrast, our approach explicitly optimizes for distribution matching, and achieves significantly better unsupervised distribution matching metrics than the implicit baseline approaches. The state based imitation approach requires careful engineering to match end-effector states, and even so often fails to find feasible joint configurations while imitating trajectories.

heading: 4.2 TRANSFERRING TASK STRATEGIES ACROSS DOMAINS
text: If our learnt correspondences capture our first insight, i.e. that different robots use similar skill sequences to address similar tasks, it follows that we can transfer task strategies across these robots by translating the skill sequence that specifies the task strategy. We evaluate how well the learnt correspondences help transfer task strategies across robots as follows.
Task Strategy Transfer Setup: We consider a set of tasks adapted from Mandlekar et al. (2018), described in the appendix. We then train high-level policies (Kulkarni et al., 2016) in the source domain to predict sequences of skills (i.e., the task strategy) to execute on each of these tasks. We then evaluate how well the translation of this task strategy performs on the same tasks in the target domain, both without fine-tuning (i.e., in a zero-shot manner), and after fine-tuning for 50 episodes. We evaluate this transfer against a hierarchical RL baseline, i.e. a high-level policy trained in the target domain over 500 episodes, and report our results in table 2. We also provide visualizations of the rollouts from the original and translated strategies in https://sites.google.com/view/translatingrobotskills/home.
Analysing results on task strategy transfer: We observe that the task strategies translated by our method are able to achieve appreciable task performance across all domain pairs and tasks, even without fine-tuning these strategies. Without fine-tuning, we see that the translated task strategies follow a semantically reasonable sequence of skills given the target task, achieving an appreciable task reward, but often reach slightly differing goal states than desired. Given our translation model only observes skill encodings, and no additional state information, this is expected. Upon fine-tuning these translated task strategies in the target domain, we observe a consistent increase in task performance, since fine-tuning allows for picking slightly different (but semantically similar) skills that reach more appropriate goal locations, etc. In contrast, hierarchical RL on the target domain is often slow to converge, and must randomly explore appropriate skill sequences. Even with fine-tuning, our approach needs roughly 10 times less training episodes to superior performance than that of the hierarchical baseline. These results suggest that our approach does indeed learn correspondences that facilitate transferring task strategies across robots.

heading: 5 CONCLUSION
text: We introduce a purely unsupervised approach to learn skill correspondences across different morphological robots from different domains. Our approach learns semantically meaningful correspondences across 3 robot domain pairs, and helps transfer task-strategies across domains, without the need for any fine-tuning, despite being completely unsupervised. We believe our approach could enable learning of correspondences across more general temporal abstractions, such as between skills and language instructions, or between skills and human video demonstrations. We hope this work helps enable robots learn from data from heterogenous robots in different domains, and helps further the state of robot learning from demonstration.

heading: A APPENDIX
text: A.1 IMPLEMENTATION DETAILS

heading: A.1.1 NETWORK ARCHITECTURES
text: We parameterize the various functions involved, namely the source and target domain encoders and low-level policies q and π respectively, and the learnt translation model T
S→T as neural networks, with following specific architectures.
1. Variational encoders q: In each domain, the variational encoder q is parameterized as a 4 layer LSTM network, with a hidden size of 48 and ReLu activation layers. We further use an input layer from the appropriate input dimensions prior to the LSTM, and two output layers to predict a Gaussian mean and variance of the skill encodings predicted by the variational encoders.
2. Low-level policies π: In each domain, the low level policies π are parameterized as 3 layer LSTM networks, with hidden sizes of 48 and ReLu activation layers. As above, we use appropriately sized input layers prior to the LSTM, and two output layers after the LSTM to predict a Gaussian mean and variance of the low-level actions output by the policy.
3. Translation model T S→T : The translation model is parameterized as a simple 4 layer MLP, with
a hidden size of 48 units, and ReLu activation layers. We similarly use two output layers to predict a Gaussian mean and variance of the translated skill encodings z
S→T . 4. Latent Skill Encoding Dimension: We use a skill encoding dimension of 16 across all domains /
robots. 5. RL High-level policy: While training RL, we parameterize the high-level policy as a 4 layer
MLP, with a hidden size of 48 units, and ReLu activation layers. As above, we use two output layers to predict a Gaussian mean and variance of the predicted skill encodings, in whichever domain we train the policy in.

heading: A.1.2 TRAINING DETAILS
text: While training the variational encoders and low-level policies, we follow the training procedure specified by Shankar & Gupta (2020). We then freeze the variational encoders and low-level policies obtained, before training our translation models. While training our translation models, we simply optimize L = Lf + Lb using the Adam optimizer (Kingma & Ba, 2014), implemented in Pytorch. We use the default parameters of Adam, i.e. a learning rate of 10−4.

heading: A.1.3 RL TRAINING DETAILS
text: While training the downstream RL, we implement a hierarchical version of Proximal Policy Optimization (Schulman et al., 2017), by adapting Achiam (2018). This mirrors the hierarchical RL setup of Kulkarni et al. (2016). We fix the low-level policies provided to the algorithm, and only train the high-level policies that predict the skill encodings fed into the low-level policies. The hierarchical RL baseline Kulkarni et al. (2016) is trained for 500 episodes, which we found to be sufficient for the algorithm’s performance to saturate. The fine-tuning approach was allowed a budget of 50 episodes to adapt the translated task strategy. The results were evaluated against a fixed random seed of 0.

heading: A.1.4 RL TASKS
text: We train on the following set of tasks adapted from Mandlekar et al. (2018). In particular, we create instances of these tasks on each of the 3 different robots. The only differences between the environments across different robots comes in the form of different initialization states for the objects concerned,
1. Reach: The robot must execute a sequence of skills to reach a block placed on the table. The robot gets a shaped reward based on the distance from the block, and a binary reward upon reaching within a threshold distance of the block.
2. Push: The robot must execute a sequence of skills to push a red block and a green block together. The reward received has a shaped component based on the distances of the end effector from the
red block, as well as a binary reward based on whether the blocks are within a threshold distance of one another.
3. Slide: The robot must execute a sequence of skills to push a red block within a threshold distance of a green block placed farther away. The reward received is based on the distance of the end effector from the red block, as well as a binary reward based on whether the blocks are within a threshold distance of one another.

heading: A.1.5 HYPERPARAMETERS
text: We provide a list of the various hyperparameters use during training of the translation model itself, and the downstream RL training.
1. Number of GMM Kernels N S and N T : For each of the forward and backward GMMs, we use 500 kernels to parameterize the GMM.
2. GMM Kernel variance σ2: For both forward and backward GMMs, we use a Gaussian Kernel variance of 0.5.
3. Relative weighting of forward and backward losses: We weight the forward and backward losses equally during training.
4. Learning Rate: For training our translation model, we use the Adam optimizer with a learning rate of 10−4.
5. Batch Size: For our training, we use a batch size of 32. 6. Number of iterations: We train our translation models over 8000 epochs for each domain pair. 7. Random Seed: We set the random seed for our training to 0 manually. 8. Epsilon Noise: We add in epsilon noise to our training during sampling from the learnt networks.
Here, we use an initial epsilon value of 0.3, and decay the epsilon value to 0.1 over 200 epochs.
While training downstream RL, we adopt the following hyperparameters:
1. Random Seed: When training downstream RL, we report results across 3 different seed values, 0, 1, 2.
2. Epsilon Noise: We follow an epsilon-greedy exploration process during RL training, and use an initial epsilon value of 0.7, and decay the epsilon value to 0.3 over 200 epochs.
3. PPO Parameters: We follow the default PPO parameters used in Achiam (2018).

heading: A.2 ASSUMPTIONS OF LEARNING
text: 1. Assumptions of datasets: As stated in our main paper, our choice of robots is dictated by the availability of demonstration datasets on a particular robot. Further, the demonstration datasets ideally also exhibit the following traits. The demonstration datasets need to be directed., i.e. they need to contain demonstrations of the robots solving a set of tasks, rather than containing random play data in an environment. While it is possible to learn skills from such play data, learning correspondences of skills from such play data is difficult. This is because our approach relies on directed sequences of skills to learn correspondences. Random play data often contains random sequences of skills, and so observe skills in contexts that they do not originally occur. This misleads our approach, which seeks to exploit context to learn correspondences.
2. Assumptions about learnt skill spaces: As stated in the analysis of our methods translations of skills, our model is able to learn good coarse correspondences, between clusters of skills in the original skill spaces. One trait of the original skill spaces that allows for this is the disentangled representation of the skills afforded by the skill-learning pipeline. Since similar semantic skills that occur in similar contexts are placed in similar clusters in the original skill space, this allows our approach to learn correct correpsondences between clusters of skills.

heading: A.3 ABLATION STUDIES
text: To accurately assess our contribution, we would also like to quantify how much each of the following components in our approach contribute to successfully learning correspondences:
1. The forward objective, Lf . 2. The backward objective, Lb. 3. The use of sequential information, i.e. matching skill-tuple distributions as opposed to matching
skill distributions. 4. Learning a translation model across frozen skill spaces, as opposed to directly optimizing the
skill representation itself.
We do so by removing each of these components individually, while keeping the rest of the method as is, and train our approach on all 3 domain pairs. We observe the following.
1. Without forward objective Lf , i.e., only optimizing the backward objective Lb, we observe the translation model is able to cover each of the modes of the target skill-tuple by translated source skills. However, since there is no objective that encourages how realistic the translated source skills are, we observe that there are several modes of translated skills that are not observed in the target skills space. This results in spurious correspondences. For example, this translation model learns to map reaching skills to z’s that are decoded into random jerky motions on the target robot, that are out of distribution for the target robot decoder.
2. Without backward objective Lb: While only optimizing the forward objective Lf , we observe that each of the translated skills appear very realistic with respect to the target domain, i.e. each of the translated skills look like a skill in the target domain. However, these models end up suffering from the same issue as the GAN based approaches or the domain adversarial style training, where several modes of the target skill space go unrepresented by target skills. This limits the ability of the learnt correspondences to effectively transfer strategies or represent source motions, since many of the target robot skills are never chosen by the translation model. Together, these ablations indicate the importance of both directions of our approach in learning successful correspondences.
3. Without sequential information: We can also optimize our objective defined in terms of skills themselves, rather than skill-tuples. While matching skill distributions, as opposed to skill-tuple distributions, we observe that the densities of skill distributions can be matched well by our objective. However, without access to sequential information, the model learns correspondences that are often wrong. For example, it learns to map reaching skills on the Baxter right hand to returning skills on the Baxter left hand, and vice versa. Skills in different contexts are often mapped to each other, since there is no contrary sequential information that the model has access to to suggest otherwise. This suggests that as described in our main paper, learning with access to sequential information and matching skill tuple distributions is also key to the success of our approach.
4. By directly optimizing the skill-representations, rather than training a translation model with fixed skill spaces: Instead of freezing the source and target skill spaces, we can also directly optimize the skill representations based on our objective. One may also think of this as maintaining a translation model that is simply an identity function. When allowing either or both of the source or target skill spaces to be finetuned, we must also optimize the original reconstruction style objective that is used in Shankar & Gupta (2020). Despite this, we observe a collapse of the skill space into a unimodal distribution, that is both unable to reconstruct skills in either domain, and be mapped across domains. This suggests freezing the representations and training a separate translation model is key to our approach.

Review summaries:
The paper proposed to learn skills correspondences between robots of different morphologies in an unsupervised way - without requiring paired data from the robots. They learn a skill translation model that maps skills from a source robot to a target robot. The translation model is learned by minimizing a loss composed of two components: (1) the likelihood of the translated skills under the target distribution estimated from target robot skills data, and (2) the likelihood of the target skills under the translated distribution estimated from the translated skills data.  The proposed approaches hinges on the assumption that robots with different morphologies follow similar high-level strategies when performing similar tasks. So, a sequence of tasks across the robots is used to learn the skills correspondences. 

This paper addresses the problem of transferring skills between morphologically different robots. This approach to learning skill correspondences is framed as a problem of matching distributions of sequences of skills across robots. The paper proposes an unsupervised objective, inspired by work in unsupervised machine translation, that makes the skill translation model learn to match the distribution of skill sequences. The proposed approach is experimentally evaluated on 3 transfer settings in a simulated robot environment.