Neural Probabilistic Motor Primitives for Humanoid Control

1 INTRODUCTION . A broad challenge in machine learning for control and robotics is to produce policies capable of general , flexible , and adaptive behavior of complex , physical bodies . To build policies that can effectively control simulated humanoid bodies , researchers must simultaneously overcome foundational challenges related to high-dimensional control , body balance , and locomotion . Recent progress in deep reinforcement learning has raised hopes that such behaviors can be learned end-to-end with minimal manual intervention . Yet , even though significant progress has been made thanks to better algorithms , training regimes , and computational infrastructure , the resulting behaviors still tend to exhibit significant idiosyncrasies ( e.g . Heess et al. , 2017 ; Bansal et al. , 2018 ) . One advantage of working with humanoids in this context is that motion capture data is widely available and can serve to help design controllers that produce apparently humanlike movement . Indeed , recent developments are now allowing for the production of highly specialized expert policies which robustly , albeit narrowly , reproduce single motion capture clips ( e.g . Liu et al . ( 2010 ) ; Peng et al . ( 2018 ) ) . A remaining challenge on the way to truly flexible and general purpose control is to be able to sequence and generalize individual movements or “ skills ” in a task-directed manner . Achieving this goal requires not just the ability to acquire individual skills in the first place , but also an architecture and associated training procedure that supports representation , recruitment , and composition of a large number of skills . This paper presents a step in this direction . Specifically , the setting we focus on will be one in which we have a large number of robust experts that perform single skills well and we wish to transfer these skills into a shared policy that can do what each expert does as well as the expert , while also generalizing to unseen behaviors within the distribution of skills . To this end we design a system ∗Equal contribution . that performs one-shot imitation as well as permits straightforward reuse ( or transfer ) of skills . We require our approach to scale to a very large number of individual skills while also keeping manual intervention and oversight to a minimum . Our primary contribution is the development of a neural network architecture that can represent and generate many motor behaviors , which we refer to as neural probabilistic motor primitives . This architecture is designed to perform one-shot imitation , while learning a dense embedding space of a large number of individual motor skills . Once trained , this module does not just reproduce individual behaviors in the training data , but can sequence and compose these behaviors in a controlled fashion as well as synthesize novel movements consistent with the training data distribution . Empirically , we also find that training controllers to reuse this learned motor primitive module for new tasks generates surprisingly human-like movement and the behavior generated seems to interpolate the space of behaviors well . In order to facilitate transfer and compression of expert skills at the scale of thousands of behaviors , we wish to avoid closed-loop RL training . We call the general , offline , functional transfer of policy content policy transfer or policy cloning and consider two approaches . The natural baseline approach involves the application of behavioral cloning to data gathered by executing experts many times , with noise , and logging intended expert actions , resembling the approach of Laskey et al . ( 2017 ) . This works well , as it ensures the student behaves like the expert not only along nominal expert rollouts but also at points arrived at by perturbing the expert . However , this approach may require many rollouts , which can be costly to obtain in many settings . As a more efficient alternative we therefore consider a second solution that operates by comprehensively transferring the functional properties of an expert to a student policy by matching the local noise-feedback properties along one or a small number of representative expert reference trajectories . We call this specific proposal linear feedback policy cloning ( LFPC ) , and we demonstrate that it is competitive with behavioral cloning from many more rollouts in our setting . 1.1 BACKGROUND & RELATED WORK . Recent efforts in RL for humanoid control build on a large body of research in robotics and animation . While contemporary results for learning from scratch ( Schulman et al. , 2015 ; Heess et al. , 2017 ) can be impressive the behaviors are not consistently human-like . Learning from motion capture ( mocap ) can provide strong constraints , especially for running ( Peng et al. , 2017 ; Merel et al. , 2017 ) . Several recent approaches have demonstrated that it is possible to acquire specific behavioral skills , possibly jointly with external RL objectives ( Merel et al. , 2017 ; Peng et al. , 2018 ; Liu & Hodgins , 2018 ) . At present , the policies produced tend to be restricted to single skills/behaviors and can require very large quantities of environment interactions , motivating us to seek methods which reuse existing single-skill expert policies . Knowledge transfer refers to the broad class of approaches which transfer the input-output functional mapping , to some extent or another , from a teacher ( or expert ) to a student ( Hinton et al. , 2015 ; Srinivas & Fleuret , 2018 ; Furlanello et al. , 2018 ) . Distillation connotes the transfer of function from one or more expert systems into a single student system often with the goal of compression or of combining multiple experts qualities ( Hinton et al. , 2015 ; Parisotto et al. , 2015 ; Rusu et al. , 2015 ; Teh et al. , 2017 ) . Imitation learning is the control-specific term for the production of a student policy from either an expert policy or the behavioral demonstrations of an expert . One basic algorithm is behavioral cloning , which refers to supervised training of the policy from state-action pairs . In the most simple case it only requires examples from the expert . A broader setting is that in which more liberal queries to the expert are permitted ; e.g . for the online-imitation setting as in DAGGER ( Ross et al. , 2011 ) . This setting is often satisfied e.g . if we wish to combine behavior from multiple experts . One-shot imitation is a concept which means that a trained system , at test time , can watch an example behavior and imitate it , as , for instance , in Duan et al . ( 2017 ) . More similar to our work is the setting examined by Wang et al . ( 2017 ) , in which full-body humanoid movements were studied . Compared with this latter work , we will employ an architecture here that encourages imitation of motor details , rather than overall movement type , and we scale our approach to more expert demonstrations . The most similar work also demonstrates large-scale one-shot humanoid tracking and was contemporaneously published ( Chentanez et al. , 2018 ) ; the approach they described involves direct tracking as well as failure recovery , but relative to our work the authors do not consider skill reuse . The notion of motor primitives is widespread in neuroscience , where there is evidence that lower dimensional control signals can selectively coordinate and blend behaviors produced by spinal circuits ( Bizzi et al. , 2008 ) , and that the cortex organizes the space of primitive motor behaviors ( Graziano , 2006 ) . In our setting , motor primitives refer to the reusable embedding space learned from many related behaviors and the associated context-modulable policy capable of generating sensory-feedback-stabilized motor behavior when executed in an environment . The particular architecture we consider is inspired by the formalization presented in Todorov & Ghahramani ( 2003 ) , which places a probabilistic latent bottleneck on the sensory-motor mapping . In the robotics literature , there is a rich line of research into various parameterizations of motion trajectories used for robot control . A class of these are referred to as “ movement primitives ” ( e.g . Schaal et al. , 2003 ) , including the “ probabilistic movement primitives ” of Paraschos et al . ( 2013 ) ( see also e.g . Neumann et al. , 2014 ) . These approaches can be seen as specific implementation choices for a certain notion of motor primitive , which emphasize the parameterization and learning of movement trajectories from repeated demonstrations ( Paraschos et al. , 2013 ; Meier & Schaal , 2016 ) , rather than learning the actuation/stabilization element , which is often handled by a prespecified PID controller . It has previously been recognized that linear-feedback policies can work well around optimal trajectories or limit cycles even for high DoF bodies . These can be obtained by sample-based optimization ( e.g . Ding et al . ( 2015 ) ) or by differential dynamic programming ( Morimoto & Atkeson , 2003 ; Tassa et al. , 2012 ; 2014 ) . For linear-quadratic-Gaussian control ( Athans , 1971 ) or differential dynamic programming ( Mayne , 1966 ; Jacobson & Mayne , 1970 ) , we obtain feedback policies where the feedback terms are computed from the value function , amounting effectively to feedbackstabilized plans . Work by Mordatch et al . ( 2015 ) has shown that linear-feedback policies resulting from trajectory optimization can be used to train neural networks . We employ a similar idea to transfer optimal behavior from an existing policy , observing that an optimal policy implicitly reflects the structure of the ( local ) value landscape and appropriately functions as a feedback controller . 2 TRANSFER AND COMPRESSION OF EXPERT BEHAVIORS . In this section , we will first briefly describe the expert policies used in this work ( Sec . 2.1 ) . We then describe the Neural Probabilistic Motor Primitive architecture and objective ( Sec . 2.2 ) . We then describe two approaches for training the module offline ( Sec . 2.3 ) . 2.1 OBTAINING EXPERTS FROM MOTION CAPTURE DATA . In order to study how to transfer and consolidate experts , we must be able to generate adequate quantities of expert data . For this work , we use expert policies trained to reproduce motion capture clips . The approach we use for producing experts is detailed more fully in Merel et al . ( 2018 ) and largely follows Peng et al . ( 2018 ) . It yields time-indexed neural network policies that are robust to moderate amounts of action noise ( see appendix A for additional details on the training procedure ) . Some examples of the resulting single-skill time-indexed policies that are obtained from this procedure are depicted in Fig . 1 . All our experts were trained in MuJoCo environments ( Todorov et al. , 2012 ) . Data We use the CMU Mocap database1 , which contains more than 2000 clips of varying lengths from more than 100 subjects . The motions in this dataset are quite varied , including many clips of walking , turning , running , jumping , dancing , various hand movements , and many more idiosyncratic behaviors . From this , we selected various clips of generic whole-body movements – any clips longer than 6 seconds were cut into smaller pieces yielding approximately 3000 , roughly 2-6 second snippets . Just over half of these are generic locomotion such as walking , running , jumping and turning . The rest of the clips mostly contained diverse hand movements while standing . We trained one expert policy per selected snippet , yielding 2707 expert policies in our training set .

Summary: This paper considers the problem of transferring motor skills from multiple experts to a student policy. To this end, the paper proposes two approaches: (1) an approach for policy cloning that learns to mimic the (local) linear feedback behavior of an expert (where the expert takes the form of a neural network), and (2) an approach that learns to compress a large number of experts via a latent space model. The approaches are applied to the problem of one-shot imitation from motion capture data (using the CMU motion capture database). The paper also considers an extension of the proposed approach to the problem of high-level planning; this is done by treating the learned latent space as a new action space and training a high-level policy that operates in this space.