Discovering Motor Programs by Recomposing Demonstrations

1 INTRODUCTION . We have seen impressive progress over the recent years in learning based approaches to perform a plethora of manipulation tasks ( Levine et al. , 2016 ; Andrychowicz et al. , 2018 ; Levine et al. , 2018 ; Pinto & Gupta , 2016 ; Agrawal et al. , 2016 ) . However , these systems are typically task-centric savants – able to only execute a single task that they were trained for . This is because these systems , whether leveraging demonstrations or environmental rewards , attempt to learn each task tabula rasa , where low to high level motor behaviours , are all acquired from scratch in context of the specified task . In contrast , we humans are adept at a variety of basic manipulation skills e.g . picking , pushing , grasping etc. , and can effortlessly perform these diverse tasks via a unified manipulation system . How can we step-away from the paradigm of learning task-centric savants , and move towards building similar unified manipulation systems ? We can begin by not treating these tasks independently , but via instead exploiting the commonalities across them . One such commonality relates to the primitive actions executed to accomplish the tasks – while the high-level semantics of tasks may differ significantly , the low and mid-level motor programs across them are often shared e.g . to either pick or push an object , one must move the hand towards it . This concept of motor programs can be traced back to the work of Lashley , who noted that human motor movements consist of ‘ orderly sequences ’ that are not simply sequences of stimulus-response patterns . The term ‘ motor programs ’ is however better attributed to Keele ( 1968 ) as being representative of ‘ muscle commands that execute a movement sequence uninfluenced by peripheral feedback ’ , though later works shifted the focus from muscle commands to the movement itself , while allowing for some feedback ( Adams , 1971 ) . More directly relevant to our motivation is Schmidt ’ s notion of ‘ generalized ’ motor programs ( Schmidt , 1975 ) that can allow abstracting a class of movement patterns instead of a singular one . In this work , we present an approach to discover the shared space of ( generalized ) motor programs underlying a variety of tasks , and show that elements from this space can be composed to accomplish diverse tasks . Not only does this allow understanding the commonalities and shared structure across diverse skills , the discovered space of motor programs can provide a high-level abstraction using which new skills can be acquired quickly by simply learning the set of desired motor programs to compose . We are not the first to advocate the use of such mid-level primitives for efficient learning or generalization , and there have been several reincarnations of this idea over the decades , from ‘ operators ’ in the classical STRIPS algorithm ( Fikes & Nilsson , 1971 ) , to ‘ options ’ ( Sutton et al. , 1999 ) or ‘ primitives ’ ( Schaal et al. , 2005 ) in modern usage . These previous approaches however assume a set of manually defined/programmed primitives and therefore bypass the difficulty of discovering them . While some attempts have been made to simultaneously learn the desired skill and the underlying primitives , learning both from scratch is difficult , and are therefore restricted to narrow tasks . Towards overcoming this difficulty , we observe that instead of learning the primitives from scratch in the context of a specific task , we can instead discover them using demonstrations of a diverse set of tasks . Concretely , by leveraging demonstrations for different skills e.g . pouring , grasping , opening etc. , we discover the motor programs ( or movement primitives ) that occur across these . We present an approach to discover movement primitives from a set of unstructured robot demonstration i.e . demonstrations without additional parsing or segmentation labels available . This is a challenging task as each demonstration is composed of a varying number of unknown primitives , and therefore the process of learning entails both , learning the space of primitives as well as understanding the available demonstrations in context of these . Our approach is based on the insight that an abstraction of a demonstrations into a sequence of motor programs or primitives , each of which correspond to an implied movement sequence , and must yield back the demonstration when the inferred primitives are ‘ recomposed ’ . We build on this and formulate an unsupervised approach to jointly learn the space of movement primitives , as well as a parsing of the available demonstrations into a high-level sequence of these primitives . We demonstrate that our method allows us to learn a primitive space that captures the shared motions required across diverse skills , and that these motor programs can be adapted and composed to further perform specific tasks . Furthermore , we show that these motor programs are semantically meaningful , and can be recombined to solved robotic tasks using reinforcement learning . Specifically , solving reaching and pushing tasks with reinforcement learning over the space of primitives achieves 2 orders of magnitude faster training than reinforcement learning in the low-level control space . 2 RELATED WORK . Our work is broadly related to several different lines of work which either learn task policies from demonstrations , or leverage known primitives for various applications , or learn primitives in context of known segments . While we discuss these relations in more detail below , we note that previous primitive based approaches either require : a ) a known and fixed primitive space , or b ) annotated segments each corresponding to a primitive . In contrast , we learn the space of primitives without requiring this segmentation annotation , and would like to emphasize that ours is the first work to do so for a diverse set of demonstrations spanning multiple tasks . Learning from Demonstration : The field of learning from demonstrations ( LfD ) ( Nicolescu & Mataric , 2003 ) has sought to learn to perform tasks from a set of demonstrated behaviors . A number of techniques exist to do so , including cloning the demonstrated behavior ( Esmaili et al. , 1995 ) , fitting a parametric model to the demonstrations ( Kober & Peters , 2009 ; Peters et al. , 2013 ) , or first segmenting the demonstrations and fitting a model to each of the resultant segments ( Niekum et al. , 2012 ; Krishnan et al. , 2018 ; Murali et al. , 2016 ; Meier et al. , 2011 ) . We point the reader to Argall et al . ( 2009 ) for a comprehensive overview of the field . Rather than directly learn to perform tasks , we use demonstrations to learn a diverse set of composable and reusable primitives , or motor-programs that may be used to perform a variety of downstream tasks . Learning and Sequencing Motion Primitives : Several works ( Kober & Peters , 2009 ; Peters et al. , 2013 ) learn motion primitives from demonstrations using predetermined representations of skills such as Dynamic Movement Primitives ( DMPs ) ( Schaal et al. , 2005 ) . A few other works have approached the problem from an optimization perspective ( Dragan et al. , 2015 ) . Given these primitives , a question that arises is how to then sequence learned skills to perform downstream tasks . Several works have attempted to answer this question - ( Neumann et al. , 2014 ) builds a layered approach to adapt , select , and sequence DMPs . Konidaris et al . ( 2012 ) ; Konidaris & Barto ( 2009 ) segments demonstrations into sequences of skills , and also merge these skills into skill-trees . However , the predetermined representations of primitives adopted in these works can prove restrictive . In particular , it prevents learning arbitrarily expressive motions and adapting these motions to a generic downstream task . We seek to move away from these fixed representations of primitives , and instead learn representations of primitives along with the primitives themselves . Latent Variable Models : The family of latent variable models ( LVMs ) provides us with fitting machinery to do so . Indeed , the success of LVMs in learning representations in deep learning has inspired several recent works ( Co-Reyes et al. , 2018 ; Kipf et al. , 2019 ; Haarnoja et al. , 2018 ; Lynch et al. , 2019 ) to learn latent representations of trajectories . SeCTAR ( Co-Reyes et al. , 2018 ) builds a latent variable conditioned policy and model that are constrained to be consistent with one another , and uses the learned policies and model for hierarchical reinforcement learning . The CompILE framework ( Kipf et al. , 2019 ) seeks to learn variable length trajectory segments from demonstrations instead , and uses latent variables to represent these trajectory segments , but is evaluated in relatively low-dimensional domains . We adopt a similar perspective to these works and learn continuous latent variable representations ( or abstractions ) of trajectory segments . Hierarchical RL and the Options Framework : The related field of hierarchical reinforcement learning ( HRL ) learns a layering of policies that each abstract away details of control of the policies of a lower level . The Options framework ( Sutton et al. , 1999 ) also learns similar temporal abstractions over sequences of atomic actions . While promising , its application has traditionally been restricted to simple domains due to the difficulty of jointly learning internal option policies along with a policy over those options . Recent works have managed to do so with only a reward function as feedback . The Option-Critic framework ( Bacon et al. , 2017 ) employs a policy-gradient formulation of options to do so , while the Option-Gradient ( Smith et al. , 2018 ) learns options in an off-policy manner . In contrast with most prior work in the options framework , ( Daniel et al. , 2016 ; Fox et al. , 2017 ; Krishnan et al. , 2017 ) learn options from a set of demonstrations , rather than in the RL setting . In similar spirit to these works , we too seek to learn abstractions from a given set of demonstrations , however unlike DDO ( Fox et al. , 2017 ) , DDCO ( Krishnan et al. , 2017 ) , and CompILE ( Kipf et al. , 2019 ) , we can learn primitives beyond a discrete set of options in a relatively high dimensional domain . Hierarchical representations of demonstrations : The idea of hierarchical task representations has permeated into LfD as well . In contrast to reasoning about demonstrations in a flat manner , one may also infer the hierarchical structure of tasks performed in demonstrations . A few recent works have striven to do so , by representing these tasks as programs ( Xu et al. , 2018 ; Sun et al. , 2018 ) , or as task graphs ( Huang et al. , 2019 ) . Both Xu et al . ( 2018 ) and Huang et al . ( 2019 ) address generalizing to new instances of manipulation tasks in the low-shot regime by abstracting away low-level controls . The idea of policy sketches , i.e . a sketch of the sub-tasks to be accomplished in a particular task , has become popular ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) . Andreas et al . ( 2017 ) learn modular policies in the RL setting provided with such policy sketches . Shiarlis et al . ( 2018 ) provides a modular LfD framework based on this idea of policy sketches . While all of these works address learning policies at various levels from demonstrations , unlike our approach , they each assume access to heavy supervision over demonstrations to do so .

Summary: The paper proposes a method for learning a set of primitives for robotic movements from a dataset of demonstrations, showing a diverse set of tasks, in an unsupervised fashion. The central underlying idea is that robotic tasks can be solved by combining fundamental building blocks, the so-called "motor programs", in the right way. The described algorithm takes a demonstration and uses a transformer network to embed the trajectory into a sequence of latent variables. Then each individual latent is transformed to a 10 step trajectory for the joint space of the robot via an LSTM network. Finally the individual trajectories are concatenated and the reconstruction is compared to the original demonstration through dynamic time warping. In this structure the latent variables represent a query to a specific learned primitive, which can be accessed using the LSTM.