Hierarchical Visuomotor Control of Humanoids

1 INTRODUCTION . In reinforcement learning ( RL ) , a major challenge is to simultaneously cope with high-dimensional input and high-dimensional action spaces . As techniques have matured , it is now possible to train high-dimensional vision-based policies from scratch to generate a range of interesting behaviors ranging from game-playing to navigation ( Jaderberg et al. , 2018 ; OpenAI , 2018 ; Wayne et al. , 2018 ) . Likewise , for controlling bodies with a large number of degrees of freedom ( DoFs ) , in simulation , reinforcement learning methods are beginning to surpass optimal control techniques . Here , we try to synthesize this progress and tackle high-dimensional input and output at the same time . We evaluate the feasibility of full-body visuomotor control by comparing several strategies for humanoid control from vision . Both to simplify the engineering of a visuomotor system and to reduce the complexity of taskdirected exploration , we construct modular agents in which a high-level system possessing egocentric vision and memory is coupled to a low-level , reactive motor control system . We build on recent advances in imitation learning to make flexible low-level motor controllers for high-DoF humanoids . The motor skills embodied by the low-level controllers are coordinated and sequenced by the high-level system , which is trained to maximize sparse task reward . Our approach is inspired by themes from neuroscience as well as ideas developed and made concrete algorithmically in the animation and robotics literatures . In motor neuroscience , studies of spinal reflexes in animals ranging from frogs to cats have led to the view that locomotion and reaching are highly prestructured , enabling subcortical structures such as the basal ganglia to coordinate a motor repertoire ; and cortical systems with access to visual input can send low complexity signals to motor systems in order to evoke elaborate movements ( Flash & Hochner , 2005 ; Bizzi et al. , 2008 ; Grillner et al. , 2005 ) . The study of “ movement primitives ” for robotics descends from the work of Ijspeert et al . ( 2002 ) . Subsequent research has focused on innovations for learning or constructing primitives for control ∗Equal contribution . 1https : //youtu.be/7GISvfbykLE of movments ( Ijspeert et al. , 2003 ; Kober & Peters , 2009 ) , deploying and sequencing them to solve tasks ( Sentis & Khatib , 2005 ; Kober & Peters , 2014 ; Konidaris et al. , 2012 ) , and increasing the complexity of the control inputs to the primitives ( Neumann et al. , 2014 ) . Particularly relevant to our cause is the work of Kober et al . ( 2008 ) in which primitives were coupled by reinforcement learning to external perceptual inputs . Research in the animation literature has also sought to produce physically simulated characters capable of distinct movements that can be flexibly sequenced . This ambition can be traced to the virtual stuntman ( Faloutsos et al. , 2001b ; a ) and has been advanced markedly in the work of Liu ( Liu et al. , 2012 ) . Further recent work has relied on reinforcement learning to schedule control policies known as “ control fragments ” , each one able to carry out only a specialized short movement segment ( Liu & Hodgins , 2017 ; 2018 ) . In work to date , such control fragments have yet to be coupled to visual input as we will pursue here . From the perspective of the RL literature ( Sutton et al. , 1999 ) , motor primitives and control fragments may be considered specialized instantiations of “ option ” sub-policies . Our work aims to contribute to this multi-disciplinary literature by demonstrating concretely how control-fragment-like low-level movements can be coupled to and controlled by a vision and memory-based high-level controller to solve tasks . Furthermore , we demonstrate the scalability of the approach to greater number of control fragments than previous works . Taken together , we demonstrate progress towards the goal of integrated agents with vision , memory , and motor control . 2 APPROACH . We present a system capable of solving tasks from vision by switching among low-level motor controllers for the humanoid body . This scheme involves a general separation of control where a low-level controller handles motor coordination and a high-level controller signals/selects lowlevel behavior based on task context ( see also Heess et al . 2016 ; Peng et al . 2017 ) . In the present work , the low-level motor controllers operate using proprioceptive observations , and the high-level controller operate using proprioception along with first-person/egocentric vision . We first describe the procedure for creating low-level controllers from motion capture data , then describe and contrast multiple approaches for interfacing the high- and low-level controllers . 2.1 TRACKING MOTION CAPTURE CLIPS . For simulated character control , there has been a line of research extracting humanoid behavior from motion capture ( “ mocap ” ) data . The SAMCON algorithm is a forward sampling approach that converts a possibly noisy , kinematic pose sequence into a physical trajectory . It relies on a beamsearch-like planning algorithm ( Liu et al. , 2010 ; 2015 ) that infers an action sequence corresponding to the pose sequence . In subsequent work , these behaviors have been adapted into policies ( Liu et al. , 2012 ; Ding et al. , 2015 ) . More recently , RL has also been used to produce time-indexed policies which serve as robust tracking controllers ( Peng et al. , 2018 ) . While the resulting time-indexed policies are somewhat less general as a result , time-indexing or phase-variables are common in the animation literature and also employed in kinematic control of characters ( Holden et al. , 2017 ) . We likewise use mocap trajectories as reference data , from which we derive policies that are single purpose – that is , each policy robustly tracks a short motion capture reference motion ( 2-6 sec ) , but that is all each policy is capable of . Humanoid body We use a 56 degree-of-freedom ( DoF ) humanoid body that was developed in previous work ( Merel et al. , 2017 ) , a version of which is available with motion-capture playback in the DeepMind control suite ( Tassa et al. , 2018 ) . Here , we actuate the joints with position-control : each joint is given an actuation range in [ −1 , 1 ] , and this is mapped to the angular range of that joint . Single-clip tracking policies For each clip , we train a policy πθ ( a|s , t ) with parameters θ such that it maximizes a discounted sum of rewards , rt , where the reward at each step comes from a custom scoring function ( see eqns . 1 , 2 defined immediately below ) . This tracking approach most closely follows Peng et al . ( 2018 ) . Note that here the state optionally includes a normalized time t that goes from 0 at the beginning of the clip to 1 at the end of the clip . For cyclical behaviors like locomotion , a gait cycle can be isolated manually and kinematically blended circularly by weighted Figure 1 : Illustration of tracking-based RL training . Training iteratively refines a policy to robustly track the reference trajectory as well as physically feasible . linear interpolation of the poses to produce a repeating walk . The time input is reset each gaitcycle ( i.e . it follows a sawtooth function ) . As proposed in Merel et al . ( 2017 ) ; Peng et al . ( 2018 ) , episodes are initialized along the motion capture trajectory ; and episodes can be terminated when it is determined that the behavior has failed significantly or irrecoverably . Our specific termination condition triggers if parts of the body other than hands or feet make contact with the ground . See Fig . 1 for a schematic . We first define an energy function most similar to SAMCON ’ s ( Liu et al. , 2010 ) : Etotal = wqposEqpos + wqvelEqvel + woriEori+ weeEee + wvelEvel + wgyroEgyro ( 1 ) where Eqpos is a energy defined on all joint angles , Eqvel on joint velocities , Eori on the body root ( global-space ) quaternion , Eee on egocentric vectors between the root and the end-effectors ( see Merel et al . ( 2017 ) ) , Evel on the ( global-space ) translational velocities , and Egyro on the body root rotational velocities . More specifically : Eqpos = 1 Nqpos ∑ |~qpos − ~q ? pos| Eqvel = 1 Nqvel ∑ |~qvel − ~q ? vel| Eori = || log ( ~qori · ~q ? −1ori ) ||2 Eee = 1 Nee ∑ ||~qee − ~q ? ee||2 Evel = 0.1 · 1 Nvel ∑ |~xvel − ~x ? vel| Egyro = 0.1 · ||~qgyro − ~q ? gyro||2 where ~q represents the pose and ~q ? represents the reference pose . In this work , we used coefficients wqpos = 5 , wqvel = 1 , wori = 20 , wgyro = 1 , wvel = 1 , wee = 2 . We tuned these by sweeping over parameters in a custom implementation of SAMCON ( not detailed here ) , and we have found these coefficients tend to work fairly well across a wide range of movements for this body . From the energy , we write the reward function : rt = exp ( −βEtotal/wtotal ) ( 2 ) where wtotal is the sum of the per energy-term weights and β is a sharpness parameter ( β = 10 throughout ) . Since all terms in the energy are non-negative , the reward is normalized rt ∈ ( 0 , 1 ] with perfect tracking giving a reward of 1 and large deviations tending toward 0 . Acquiring reference data features for some quantities required setting the body to the pose specified by the joint angles : e.g. , setting ~xpos , ~qpos , and ~qori to compute the end-effector vectors ~qee . Joint angle velocities , root rotational velocities , and translational velocities ( ~qvel , ~qgyro , ~xvel ) were derived from the motion capture data by finite difference calculations on the corresponding positions . Note that the reward function here was not restricted to egocentric features – indeed , the velocity and quaternion were non-egocentric . Importantly , however , the policy received exclusively egocentric observations , so that , for example , rotating the initial pose of the humanoid would not affect the policy ’ s ability to execute the behavior . The full set of proprioceptive features we provided the policy consists of joint angles ( ~qpos ) and velocities ( ~qvel ) , root-to-end-effector vectors ( ~qee ) , rootframe velocimeter ( ~qveloc ) , rotational velocity ( ~qgyro ) , root-frame accelerometers ( ~qaccel ) , and 3D orientation relative to the z-axis ( ~rz : functionally a gravity sensor ) . Low-level controller reinforcement learning details Because the body is position-controlled , ( at has the same dimension and semantics as a subset of the body pose ) , we can pre-train the policy to produce target poses by supervised learning maxθ ∑ t log π ( q ∗ pos , t+1|s∗t , t ) . This produces very poor control but facilitates the subsequent stage of RL-based imitation learning . We generally found that training with some pretraining considerably shortened the time the training took to converge and improved the resulting policies . For RL , we performed off-policy training using a distributed actor-critic implementation , closest to that used in ( Hausman et al. , 2018 ) . This implementation used a replay buffer and target networks as done in previous work ( Lillicrap et al. , 2015 ; Heess et al. , 2015 ) . The Q-function was learned off-policy using TD-learning using importance-weighted Retrace ( Munos et al. , 2016 ) , and the actor was learned off-policy using SVG ( 0 ) ( Heess et al. , 2015 ) . This is to say that we learned the policy by taking gradients with respect to the Q function ( target networks were updated every 500 learning steps ) . Gradient updates to the policy were performed using short time windows , { sτ , aτ } τ=1 ... T , sampled from replay : max πθ ∑ τ=1 ... T Ea∼π ( a|sτ ) [ Qtarget ( sτ , a ) ] − ηDKL [ πθ ( a|sτ ) ||πtarget ( a|sτ ) ] ( 3 ) where η was fixed in our experiments . While the general details of the RL algorithm are not pertinent to the success of this approach ( e.g . Peng et al . ( 2018 ) used on-policy RL ) , we found two details to be critical , and both were consistent with the results reported in Peng et al . ( 2018 ) . Policy updates needed to be performed conservatively with the update including a term which restricts DKL [ πnew||πold ] ( Heess et al. , 2015 ; Schulman et al. , 2017 ) . Secondly , we found that attempting to learn the variance of the policy actions tended to result in premature convergence , so best results were obtained using a stochastic policy with fixed noise ( we used noise with σ = .1 ) .

Summary: This paper proposes a hierarchical reinforcement learning (HRL) method for visual motor control of humanoid agents. The method is decomposed into a high-level controller that takes in visual input and proprioceptive information, and a low-level controller (they compare may ways of doing this) that takes care of the agent’s motor control. In experiments, the proposed method is tested on a variety of RL tasks where the many low-level controllers presented in the paper are compared against each other.