Complex Locomotion Skill Learning via Differentiable Physics

1 INTRODUCTION . Differentiable physical simulators deliver accurate analytical gradients of physical simulations , opening up a promising stage for efficient neural network ( NN ) controller training via gradient descent . Existing research demonstrates that on simple tasks , learning systems via differentiable physics can effectively leverage simulation gradient information and converge orders of magnitude faster ( see , e.g. , de Avila Belbute-Peres et al . ( 2018 ) ; Hu et al . ( 2019 ) ) than reinforcement learning . However , the capability of existing differentiable physics based learning systems is relatively limited . Typically , optimized controllers can only achieve relatively simple single-goal tasks ( e.g. , moving in one specific direction , as in ChainQueen Hu et al . ( 2019 ) ) . Those learned controllers often have difficulty generalizing the task to a perturbed version . In this work , we propose a learning framework for complex locomotion skill learning via differentiable physics . The complexity of our tasks comes from two aspects : first , the degrees of freedom of our soft agents are significantly higher compared with the rigid-body ones ; second , the agents are expected to learn multiple skills at the same time . We systematically propose a suite of enhancements ( Fig . 4 ) over existing training approaches ( such as Hu et al . ( 2020 ) ; Huang et al . ( 2021 ) ) to significantly improve the efficiency , robustness , and generalizability of learning systems based on differentiable physics . In addition , we investigated the contributions of each enhancements to training in detail by a series of ablation studies . We also evaluated our framework through comparisons against Proximal Policy Optimization ( PPO ) Schulman et al . ( 2017 ) , a state-of-the-art reinforcement learning algorithm . Results show that our system is simple yet effective and has a much-improved convergence rate compared to PPO . We demonstrate the versatility of our framework via various physical simulation environments ( mass-spring systems and the moving least squares material point method , MLS-MPM Hu et al . ( 2018 ) ) , robot designs ( structured and irregular ) , locomotion tasks ( moving , jumping , turning around , all at a controllable velocity ) . To the best of our knowledge , this is the first time when a neural network can be trained via differentiable physics to achieve tasks of such complexity ( Fig . 1 ) . Our agent is simultaneously trained with multiple goals , the learned skills can be continuously interpolated . For example , we can control both the velocity and height of an agent while it runs . The trained agent can also be controlled in real-time , enabling direct applications in soft robot control and video games . In summary , our key contributions are listed below : • To the best of our knowledge , we proposed the first differentiable physics based learning framework for multiple locomotion skills learning on soft agents using a single network . • We developed an end-to-end differentiable physical simulation environment for deformable robot locomotion skills learning , which supports mass-spring system and material point method as dynamic backends . • We systematically investigated the key factors contributing to the differentiable physics based learning framework . We believe our investigation can inspire further researches to explore the possibilities of differentiable physics for more complex learning tasks . 2 DIFFERENTIABLE SIMULATION ENVIRONMENTS . 2.1 SIMULATION SETTINGS . Our learning framework supports multiple types of differentiable physically-based deformable body simulators . The simulator can be seen as a black box that outputs the states for the next time step given the actuation signal and states from the current time step . The NN controller takes the information given by the simulator as part of its input features and decides the actuation signal for the simulator in the next time step . With our automatic differentiation system , we can evaluate the gradient information to guide the training of our NN controllers . We use the mass-spring systems and the moving least squares material point method ( MLS-MPM ) as our simulation environments . The design choices for each environment can be found in the appendix . 2.2 AGENT STRUCTURE AND ACTUATION MODE . Our framework supports a variety of differently shaped agents as shown in Fig . 2 and 3 . Our actuation signal exerts forces along with the ” muscle ” directions of the simulated agents . This signal falls within the interval [ −1 , +1 ] , where its sign determines whether an agent wants to contract ( negative sign ) or relax ( positive sign ) a muscle and its absolute value determines the magnitude of the force . In mass-spring systems , the muscle directions are represented by the spring directions , we change the rest-length of springs to generate forces . The actuation signal is used to scale the rest-length up to some limit ( usually at 20 % ) . In addition , only activated springs , i.e . actuators ( springs marked red or blue shown in Fig . 1 ) , are able to generate forces according to control signals . In material point methods , the muscle directions are always along the vertical axis in material space . In our implementation , we simply modify the Cauchy stress in the vertical direction of material space to apply this force . Again , the force is scaled to a user-defined bound which can be adjusted for different simulations . 3 LEARNING FRAMEWORK . In this section , we describe our training framework based on differentiable physics ( Fig . 4 ) . In summary , we develop a differentiable simulator with an embedded NN controller . In each time step , the program performs an NN controller inference and a differentiable simulator time integration . After a few hundred time steps , the program back-propagates gradients end-to-end via reverse-mode automatic differentiation , and updates the weights of the neural network using the Adam optimizer Kingma & Ba ( 2014 ) . 3.1 TASK REPRESENTATION . Given an agent A , its position and corresponding velocity are denoted as x = ( x , y , z ) and v = ( u , v , w ) . We presents three tasks ( running , jumping and crawling ) for 2D agents , two ( running and rotating ) for 3D . In a whole simulation with total steps T , an agent is expected to achieve all goals in a goal sequence G = { G1 , G2 ... Gn|nP = T } , where P is the time period for an agent to achieve a goal , n is the number of goals . For each goal G , there are multiple tasks in it . Each task is encoded using a target value g , which instructs the controller to drive the agent . For example , G = { gv , gh } represent performing running and jumping simultaneously , the target velocity and height are gv and gh . The agent is expected to switch between multiple goals during one simulation . Running . The running task is defined as a velocity tracking problem . Given a time step t , the center of mass c is used to represent the position of the agent , its velocity is defined as the averaged velocity ṽ over a time period Pr , ( Pr < P ) . Then the agent is expected to achieve a velocity ṽ close to the target velocity gv . ṽ ( t ) = 1 Pr ( c ( t ) − c ( t− Pr ) ) ( 1 ) Jumping . The jumping task is defined as reaching a given height gh in a time period P . Due to the gravity , it is not possible for a running robot to stay away from the ground . Therefore , the jumping height h̃h ( t ) of an agent is defined by the maximum vertical position of its lowest point in a certain period P . To be more formally , given S a set of nodes ( or points ) that constitute the agent , h is a function that can extract the vertical position given a node ( or point ) , the jumping height over the n-th period is defined as h̃h ( n ) = max t∈ [ 0 , P ] min s∈S h ( s , t+ nP ) ( 2 ) where n is the index of the period in a whole simulation . Crawling . The crawling task is defined as lowering the highest point as much as possible . The target of crawling gc is defined as an indicator function . The agent is controlled to crawl if gc = 1 otherwise if gc = 0 . Since crawling is a status that needs to be maintained , the crawling height is defined as the vertical position of its highest point at each time step . h̃c ( t ) = max s∈S h ( s , t ) ( 3 ) 3.2 LOSS FUNCTIONS . Consider a locomotion task where an agent is instructed to move at a specified velocity , we think a proper loss function should satisfy the following requirements : 1 . Periodicity . The agent is expected to move periodically following a predefined cyclic activation signal . Therefore , the loss function should encourage periodic motions . 2 . Delayed evaluation . Due to inertia , it takes time for the agent to start running or adjust running velocity . Therefore , an ideal loss function should take this delay into consideration . 3 . Fluctuation tolerance . During a single running cycle , requiring the center of mass to move at a constant velocity at each time step may induce highly fluctuated losses . To avoid that , our loss function should be smooth during one motion cycle . Task loss . For each task , we defined a tailored loss shown in equation 4 according to the requirements above . The loss for running Lv is defined as the accumulation of the difference between the target and the agent ’ s velocity from start to current time step . For jumping , we define a sparse loss Lh , which only evaluate once for each period . The crawling loss Lc is defined as the accumulation of the crawling height if the loss is applicable . L = λv ∑ n∈T P∑ t=Pr ( ṽ ( t ) − gv ( n ) ) 2︸ ︷︷ ︸ Lv +λh ∑ n∈T ( h̃h ( n ) − gh ( n ) ) 2︸ ︷︷ ︸ Lh +λc ∑ n∈T P∑ t=0 gc ( n ) h̃c ( t+ nP ) ︸ ︷︷ ︸ Lc ( 4 ) The λv , λh , λc are weights for losses of dfferent tasks . Our loss function accumulates the contributions of all steps in a “ sliding window ” . This prevents the loss function from being too small as well as the vanishing gradient problem . We find that having an accumulated loss evaluated at every step works much better when we want to control the speed of the agent explicitly . Regularization . Since the agents may keep shaking when no target velocity or height are given , we introduce an actuation loss as a regularization term . The intuition is to penalize comparatively large actuations when small goals value is given : La = ∑ n∈T ∑ t∈ [ 0 , P ] ( A ( t ) − µ||gv ( n ) || ) 2 ( 5 ) where A ( t ) is the actuation output of NN , µ is a normalizer . The actuation loss balances the importance of input channels of the NN , i.e. , different input channels produce similar contributions to the first layer of the trained NN .

Summary: This paper proposes to use differentiable simulators for learning locomotion skills for deformable characters. The key contribution is the experiments and analysis of a large number of design choices, including activation function, reward design, optimizer, batch size, etc. The paper finds that by combining differentiable simulator with the right design choices, the character can learn general and complex skills with one policy. The paper also shows that this combination can significantly outperform PPO.