Translating Robot Skills: Learning Unsupervised Skill Correspondences Across Robots

1 INTRODUCTION . Humans have a remarkable ability to efficiently learn to perform tasks by watching others demonstrate similar tasks . For example , children quickly learn the skills needed to play a new sport by watching their parents perform skills such as kicking a ball . Notably , they are able to learn from these visual demonstrations despite significant differences between themselves and the demonstrator , including visual perspectives , environments , kinematic and dynamic properties , and morphologies . This ability may be attributed to two factors ; firstly , humans have well-developed basic motor skills that we can execute with little effort . Second , we can recognize the sequence of skills ( or the high-level strategy ) the demonstrator uses , and understand a corresponding set of skills that we can execute ourselves ( Meltzoff & Moore , 1977 ) . In this paper , we explore how we can endow robots with this ability - i.e. , to adopt task strategies from morphologically different robot demonstrators , and then execute corresponding skills to solve similar tasks . Key to solving this problem is for the robot to identify how its owns skills correspond to those of the demonstrator , which is tremendously powerful . First , it allows the robot to adopt the demonstrator ’ s strategies for solving various tasks . Second , by adopting and adapting these strategies for itself , the robot can efficiently learn to solve a variety of tasks previously outside its repertoire . Finally , it allows us to understand the task strategies used by various robots in a unified manner , enabling robots to learn from data collected with a heterogeneous collection of tasks and robot morphologies . Skills provide a natural framework to facilitate such concise knowledge transfer across robots compared to low-level robot controls ; skills inherently abstract away low-level details that may differ across the demonstrator and the learner , and instead focus on the commonalities between them , such as the task strategy . For example , skills such as reaching and placing on robot manipulators abstract away differences in morphologies or configuration , and are thus well grounded across robots , while there may not be obvious correspondence in their low-level actions . How can one acquire correspondences between skills ? Stated more formally , given morphologically different robots , a set of unlabelled demonstrations of each robot solving a variety of tasks , and a method for extracting skills from those demonstrations , how can one learn correspondences between the skills of the different robots ? Learning such correspondences is straightforward when supervised pairs of skills or trajectories are available . However , collecting and annotating such data is timeconsuming and tedious , and requires a high degree of human expertise , particularly at the scale necessary to successfully learn correspondences across a diverse set of skills . In contrast , we explore whether we can learn such skill correspondences in an unsupervised manner , i.e. , without access to supervised skill or trajectory pairs . Learning unsupervised skill correspondences potentially allows for learning a broad scope of correspondences from the diverse unlabelled and unsegmented data newly available in robotics ( Sharma et al. , 2018 ; Mandlekar et al. , 2018 ) . Unfortunately , learning unsupervised correspondences is a difficult problem to solve - without supervision , it is difficult to guide learning towards the right correspondences . While this may be alleviated to some extent by incorporating unsupervised constraints ( Zhu et al. , 2017 ; Ganin et al. , 2016 ; Zhou et al. , 2019 ) , there are many spurious correspondences that satisfy these constraints . How can one then go about learning such skill correspondences in an unsupervised manner ? We propose leveraging the two following insights to do so . Our first insight is that different morphological robots use similar task strategies ( in terms of sequences of skills ) to solve similar tasks ; in other words , the sequences of skills executed by different robots to solve similar tasks ought to belong to similar distributions . We observe that this is generally true ; for example , a robot pouring out a cup of tea would likely first reach for the kettle , grasp it , move it appropriately over the cup , and then begin to pour the tea , irrespective of its exact morphology . Our second insight is that learning skill correspondences without access to supervised data closely mirrors unsupervised machine translation ( UMT ) , where the objective is to learn a translation between representations in different languages ( such as between word embeddings across languages ) , without access to parallel data ( Conneau et al. , 2017 ; Lample et al. , 2018 ) . Inspired by these two insights , we derive an unsupervised objective to guide our learning towards meaningful skill correspondences . We approach the problem of learning unsupervised skill correspondences by learning a translation model to map from the skill space on one “ source ” robot to the skill space on another “ target ” robot . We construct an unsupervised objective that encourages the translation model to respect our first insight - i.e. , to preserve the sequences of skills observed across both the ( translated source ) and target robots . To do this , we take inspiration from our second insight , and specifically from Zhou et al . ( 2019 ) , and construct explicit probability density models over the sequences of skills observed on the source and target robots , and then train the translation model to match these distributions . We evaluate the ability of our approach to learn unsupervised skill correspondences across three different robots , the Sawyer robot , and the left and right hands of the Baxter robot . Our approach is able to learn semantically meaningful correspondences across each of the 3 pairs of robots , despite being completely unsupervised , as depicted in fig . 1 . The learnt correspondences facilitate transferring task strategies across across domains , as we demonstrate on a set of downstream tasks . Our results are visualized at https : //sites.google.com/view/translatingrobotskills/home . 2 RELATED WORK . Skill Learning : Eysenbach et al . ( 2019 ) ; Sharma et al . ( 2020 ) both address unsuperivsed skill learning from interaction data , by constructing information theoretic approaches . Fox et al . ( 2017 ) ; Krishnan et al . ( 2017 ) ; Shankar et al . ( 2020 ) ; Sharma et al . ( 2018 ) ; Shankar & Gupta ( 2020 ) ; Kipf et al . ( 2019 ) ; Gregor et al . ( 2019 ) ; Kim et al . ( 2019 ) instead learn skills or abstractions from unlabelled demonstration data by performing latent variable inference . These frameworks all learn skills in the context of a single domain , while we seek to learn correspondences of skills across domains . Unsupervised Correspondence Learning : Our problem bears a close resemblance with unsupervised machine translation ( Conneau et al. , 2017 ; Zhou et al. , 2019 ) , and unpaired image translation ( Zhu et al. , 2017 ; Park et al. , 2020 ) . Specifically , they all share the notion of learning correspondences across representations learnt from unpaired , “ monolingual ” data : • Unsupervised Correspondence in Machine Translation : i i i l ti : s r s rr s r se e e e ce ac e a ai i i l ti : r rr r Conneau et al . ( 2017 ) leveraged domainadversarial training ( Ganin et al. , 2016 ) to align word embeddings of two languages . Sennrich et al . ( 2016 ) ; Lample et al . ( 2018 ) use the idea of back-translation to constraint the learnt translation models across languages . Zhou et al . ( 2019 ) learn bilingual word embeddings by matching explicit density functions over the word embedding spaces across languages . • Unsupervised Correspondence in Image Translation : i i l ti : s r s rr s I r se e e e ce a e a ai i l ti : r rr I r The vision community has similarly explored the unpaired image-to-image translation setting ( Zhu et al. , 2017 ; Park et al. , 2020 ) , using cycle-consistency losses ( Zhu et al. , 2017 ) or contrastive losses ( Park et al. , 2020 ) . • Unsupervised Correspondence in Video : i i i : s r s rr se e e e ce ei i i : r rr Bansal et al . ( 2018 ) ; Wang et al . ( 2019 ) extend CycleGAN ( Zhu et al. , 2017 ) to the video domain , by incorporating temporal consistency losses . Domain Transfer in Robotics and Graphics : The robotics and graphics communities have taken interest in cross-domain transfer of policies in recent years : • Policy Transfer with Paired Data : li it i t : r sf r rc a e a e a ali it i t : r f r r Gupta et al . ( 2017 ) ; Sermanet et al . learn morphology and viewpoint invariant feature spaces for policy transfer respectively , but require paired data to do so . • Policy Transfer via Modularity : li i l it : r sf r rc a e a ali i l it : r f r r Hejna et al . ( 2020 ) ; Devin et al . ( 2017 ) ; Sharma et al . ( 2019 ) address morphological transfer by modularity in their policies . Hejna et al . ( 2020 ) ; Sharma et al . ( 2019 ) adopt modularity in a hierarchical sense , but leverage common grounding of subgoals across morphologies to perform transfer . We do not assume access to such common grounding . • State based Policy Transfer : t t li : s r sf ra e a e c a et t li : r f r Liu et al . ( 2020 ) ; Schroecker & Isbell ( 2017 ) ; Ammar et al . ( 2015 ) address cross-morphology transfer by state based imitation learning . • Motion Retargetting : ti t tti : re a eti t tti : r The graphics community has addressed transferring behaviors across morphologically different characters ( Hecker et al. , 2008 ; Aberman et al. , 2020 ; Villegas et al. , 2018 ; Abdul-Massih et al. , 2017 ) , using carefully handcrafted kinematic models . These works transfer behaviors by imitating joint positions , and performing inverse kinematics to retrieve the full character state , which is often infeasible for widely different morphological characters . • Unsupervised Action Correspondence : i ti : s r s rr se e c e e cei ti : r rr Zhang et al . ( 2021 ) ; Kim et al . ( 2020 ) ; Smith et al . ( 2020 ) address learning low-level state and action correspondences from unpaired , unaligned interaction and demonstration data respectively . While similar in spirit , our work argues that learning highlevel skill correspondence instead is a more natural choice , as mentioned in the introduction . Applicability to learning skill correspondences : While successful in their respective problem domains , existing approaches are not trivially applicable to our problem . The adversarial training used in most of these approaches is notoriously difficult to train , and is prone to the mode dropping problem ( Li & Malik , 2019 ) . They require strong constraints such as pixel-wise or joint-wise identity losses ( Zhu et al. , 2017 ; Zhang et al. , 2021 ) , or inherent similarity of the spaces to be aligned , such as word embeddings across languages ( Conneau et al. , 2017 ; Zhou et al. , 2019 ) . Learnt skill representations do not possess this property in general . Shortcomings notwithstanding , these approaches provide insight into how we may pursue learning unsupervised skill correspondences .

Summary: This paper addresses the problem of transferring skills between morphologically different robots. This approach to learning skill correspondences is framed as a problem of matching distributions of sequences of skills across robots. The paper proposes an unsupervised objective, inspired by work in unsupervised machine translation, that makes the skill translation model learn to match the distribution of skill sequences. The proposed approach is experimentally evaluated on 3 transfer settings in a simulated robot environment.