Discovering Motor Programs by Recomposing Demonstrations

1 INTRODUCTION . We have seen impressive progress over the recent years in learning based approaches to perform a plethora of manipulation tasks ( Levine et al. , 2016 ; Andrychowicz et al. , 2018 ; Levine et al. , 2018 ; Pinto & Gupta , 2016 ; Agrawal et al. , 2016 ) . However , these systems are typically task-centric savants – able to only execute a single task that they were trained for . This is because these systems , whether leveraging demonstrations or environmental rewards , attempt to learn each task tabula rasa , where low to high level motor behaviours , are all acquired from scratch in context of the specified task . In contrast , we humans are adept at a variety of basic manipulation skills e.g . picking , pushing , grasping etc. , and can effortlessly perform these diverse tasks via a unified manipulation system . How can we step-away from the paradigm of learning task-centric savants , and move towards building similar unified manipulation systems ? We can begin by not treating these tasks independently , but via instead exploiting the commonalities across them . One such commonality relates to the primitive actions executed to accomplish the tasks – while the high-level semantics of tasks may differ significantly , the low and mid-level motor programs across them are often shared e.g . to either pick or push an object , one must move the hand towards it . This concept of motor programs can be traced back to the work of Lashley , who noted that human motor movements consist of ‘ orderly sequences ’ that are not simply sequences of stimulus-response patterns . The term ‘ motor programs ’ is however better attributed to Keele ( 1968 ) as being representative of ‘ muscle commands that execute a movement sequence uninfluenced by peripheral feedback ’ , though later works shifted the focus from muscle commands to the movement itself , while allowing for some feedback ( Adams , 1971 ) . More directly relevant to our motivation is Schmidt ’ s notion of ‘ generalized ’ motor programs ( Schmidt , 1975 ) that can allow abstracting a class of movement patterns instead of a singular one . In this work , we present an approach to discover the shared space of ( generalized ) motor programs underlying a variety of tasks , and show that elements from this space can be composed to accomplish diverse tasks . Not only does this allow understanding the commonalities and shared structure across diverse skills , the discovered space of motor programs can provide a high-level abstraction using which new skills can be acquired quickly by simply learning the set of desired motor programs to compose . We are not the first to advocate the use of such mid-level primitives for efficient learning or generalization , and there have been several reincarnations of this idea over the decades , from ‘ operators ’ in the classical STRIPS algorithm ( Fikes & Nilsson , 1971 ) , to ‘ options ’ ( Sutton et al. , 1999 ) or ‘ primitives ’ ( Schaal et al. , 2005 ) in modern usage . These previous approaches however assume a set of manually defined/programmed primitives and therefore bypass the difficulty of discovering them . While some attempts have been made to simultaneously learn the desired skill and the underlying primitives , learning both from scratch is difficult , and are therefore restricted to narrow tasks . Towards overcoming this difficulty , we observe that instead of learning the primitives from scratch in the context of a specific task , we can instead discover them using demonstrations of a diverse set of tasks . Concretely , by leveraging demonstrations for different skills e.g . pouring , grasping , opening etc. , we discover the motor programs ( or movement primitives ) that occur across these . We present an approach to discover movement primitives from a set of unstructured robot demonstration i.e . demonstrations without additional parsing or segmentation labels available . This is a challenging task as each demonstration is composed of a varying number of unknown primitives , and therefore the process of learning entails both , learning the space of primitives as well as understanding the available demonstrations in context of these . Our approach is based on the insight that an abstraction of a demonstrations into a sequence of motor programs or primitives , each of which correspond to an implied movement sequence , and must yield back the demonstration when the inferred primitives are ‘ recomposed ’ . We build on this and formulate an unsupervised approach to jointly learn the space of movement primitives , as well as a parsing of the available demonstrations into a high-level sequence of these primitives . We demonstrate that our method allows us to learn a primitive space that captures the shared motions required across diverse skills , and that these motor programs can be adapted and composed to further perform specific tasks . Furthermore , we show that these motor programs are semantically meaningful , and can be recombined to solved robotic tasks using reinforcement learning . Specifically , solving reaching and pushing tasks with reinforcement learning over the space of primitives achieves 2 orders of magnitude faster training than reinforcement learning in the low-level control space . 2 RELATED WORK . Our work is broadly related to several different lines of work which either learn task policies from demonstrations , or leverage known primitives for various applications , or learn primitives in context of known segments . While we discuss these relations in more detail below , we note that previous primitive based approaches either require : a ) a known and fixed primitive space , or b ) annotated segments each corresponding to a primitive . In contrast , we learn the space of primitives without requiring this segmentation annotation , and would like to emphasize that ours is the first work to do so for a diverse set of demonstrations spanning multiple tasks . Learning from Demonstration : The field of learning from demonstrations ( LfD ) ( Nicolescu & Mataric , 2003 ) has sought to learn to perform tasks from a set of demonstrated behaviors . A number of techniques exist to do so , including cloning the demonstrated behavior ( Esmaili et al. , 1995 ) , fitting a parametric model to the demonstrations ( Kober & Peters , 2009 ; Peters et al. , 2013 ) , or first segmenting the demonstrations and fitting a model to each of the resultant segments ( Niekum et al. , 2012 ; Krishnan et al. , 2018 ; Murali et al. , 2016 ; Meier et al. , 2011 ) . We point the reader to Argall et al . ( 2009 ) for a comprehensive overview of the field . Rather than directly learn to perform tasks , we use demonstrations to learn a diverse set of composable and reusable primitives , or motor-programs that may be used to perform a variety of downstream tasks . Learning and Sequencing Motion Primitives : Several works ( Kober & Peters , 2009 ; Peters et al. , 2013 ) learn motion primitives from demonstrations using predetermined representations of skills such as Dynamic Movement Primitives ( DMPs ) ( Schaal et al. , 2005 ) . A few other works have approached the problem from an optimization perspective ( Dragan et al. , 2015 ) . Given these primitives , a question that arises is how to then sequence learned skills to perform downstream tasks . Several works have attempted to answer this question - ( Neumann et al. , 2014 ) builds a layered approach to adapt , select , and sequence DMPs . Konidaris et al . ( 2012 ) ; Konidaris & Barto ( 2009 ) segments demonstrations into sequences of skills , and also merge these skills into skill-trees . However , the predetermined representations of primitives adopted in these works can prove restrictive . In particular , it prevents learning arbitrarily expressive motions and adapting these motions to a generic downstream task . We seek to move away from these fixed representations of primitives , and instead learn representations of primitives along with the primitives themselves . Latent Variable Models : The family of latent variable models ( LVMs ) provides us with fitting machinery to do so . Indeed , the success of LVMs in learning representations in deep learning has inspired several recent works ( Co-Reyes et al. , 2018 ; Kipf et al. , 2019 ; Haarnoja et al. , 2018 ; Lynch et al. , 2019 ) to learn latent representations of trajectories . SeCTAR ( Co-Reyes et al. , 2018 ) builds a latent variable conditioned policy and model that are constrained to be consistent with one another , and uses the learned policies and model for hierarchical reinforcement learning . The CompILE framework ( Kipf et al. , 2019 ) seeks to learn variable length trajectory segments from demonstrations instead , and uses latent variables to represent these trajectory segments , but is evaluated in relatively low-dimensional domains . We adopt a similar perspective to these works and learn continuous latent variable representations ( or abstractions ) of trajectory segments . Hierarchical RL and the Options Framework : The related field of hierarchical reinforcement learning ( HRL ) learns a layering of policies that each abstract away details of control of the policies of a lower level . The Options framework ( Sutton et al. , 1999 ) also learns similar temporal abstractions over sequences of atomic actions . While promising , its application has traditionally been restricted to simple domains due to the difficulty of jointly learning internal option policies along with a policy over those options . Recent works have managed to do so with only a reward function as feedback . The Option-Critic framework ( Bacon et al. , 2017 ) employs a policy-gradient formulation of options to do so , while the Option-Gradient ( Smith et al. , 2018 ) learns options in an off-policy manner . In contrast with most prior work in the options framework , ( Daniel et al. , 2016 ; Fox et al. , 2017 ; Krishnan et al. , 2017 ) learn options from a set of demonstrations , rather than in the RL setting . In similar spirit to these works , we too seek to learn abstractions from a given set of demonstrations , however unlike DDO ( Fox et al. , 2017 ) , DDCO ( Krishnan et al. , 2017 ) , and CompILE ( Kipf et al. , 2019 ) , we can learn primitives beyond a discrete set of options in a relatively high dimensional domain . Hierarchical representations of demonstrations : The idea of hierarchical task representations has permeated into LfD as well . In contrast to reasoning about demonstrations in a flat manner , one may also infer the hierarchical structure of tasks performed in demonstrations . A few recent works have striven to do so , by representing these tasks as programs ( Xu et al. , 2018 ; Sun et al. , 2018 ) , or as task graphs ( Huang et al. , 2019 ) . Both Xu et al . ( 2018 ) and Huang et al . ( 2019 ) address generalizing to new instances of manipulation tasks in the low-shot regime by abstracting away low-level controls . The idea of policy sketches , i.e . a sketch of the sub-tasks to be accomplished in a particular task , has become popular ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) . Andreas et al . ( 2017 ) learn modular policies in the RL setting provided with such policy sketches . Shiarlis et al . ( 2018 ) provides a modular LfD framework based on this idea of policy sketches . While all of these works address learning policies at various levels from demonstrations , unlike our approach , they each assume access to heavy supervision over demonstrations to do so .

Summary: This work presents a novel approach to extracting reusable motor primitives from task demonstrations.  The approach taken in this work involves learning a deep encoder network which translates an arbitrary length trajectory in a robot's configuration space (is this right?) into a sequence of vectors describing different motor primitives.  A second decoder network translates these vectors into a sequence of trajectory segments.  These networks are trained to minimize the distance between the original trajectory, and the trajectory generated by encoding and reconstructing the original as a sequence of primitives and reconstructing.  An additional regularization term discourages the network from learning trivial, one step primitives.  The decoder network is also initialized by training on a set of simple trajectories generated by a robotic planning algorithm.