Complex Locomotion Skill Learning via Differentiable Physics

1 INTRODUCTION . Differentiable physical simulators deliver accurate analytical gradients of physical simulations , opening up a promising stage for efficient neural network ( NN ) controller training via gradient descent . Existing research demonstrates that on simple tasks , learning systems via differentiable physics can effectively leverage simulation gradient information and converge orders of magnitude faster ( see , e.g. , de Avila Belbute-Peres et al . ( 2018 ) ; Hu et al . ( 2019 ) ) than reinforcement learning . However , the capability of existing differentiable physics based learning systems is relatively limited . Typically , optimized controllers can only achieve relatively simple single-goal tasks ( e.g. , moving in one specific direction , as in ChainQueen Hu et al . ( 2019 ) ) . Those learned controllers often have difficulty generalizing the task to a perturbed version . In this work , we propose a learning framework for complex locomotion skill learning via differentiable physics . The complexity of our tasks comes from two aspects : first , the degrees of freedom of our soft agents are significantly higher compared with the rigid-body ones ; second , the agents are expected to learn multiple skills at the same time . We systematically propose a suite of enhancements ( Fig . 4 ) over existing training approaches ( such as Hu et al . ( 2020 ) ; Huang et al . ( 2021 ) ) to significantly improve the efficiency , robustness , and generalizability of learning systems based on differentiable physics . In addition , we investigated the contributions of each enhancements to training in detail by a series of ablation studies . We also evaluated our framework through comparisons against Proximal Policy Optimization ( PPO ) Schulman et al . ( 2017 ) , a state-of-the-art reinforcement learning algorithm . Results show that our system is simple yet effective and has a much-improved convergence rate compared to PPO . We demonstrate the versatility of our framework via various physical simulation environments ( mass-spring systems and the moving least squares material point method , MLS-MPM Hu et al . ( 2018 ) ) , robot designs ( structured and irregular ) , locomotion tasks ( moving , jumping , turning around , all at a controllable velocity ) . To the best of our knowledge , this is the first time when a neural network can be trained via differentiable physics to achieve tasks of such complexity ( Fig . 1 ) . Our agent is simultaneously trained with multiple goals , the learned skills can be continuously interpolated . For example , we can control both the velocity and height of an agent while it runs . The trained agent can also be controlled in real-time , enabling direct applications in soft robot control and video games . In summary , our key contributions are listed below : • To the best of our knowledge , we proposed the first differentiable physics based learning framework for multiple locomotion skills learning on soft agents using a single network . • We developed an end-to-end differentiable physical simulation environment for deformable robot locomotion skills learning , which supports mass-spring system and material point method as dynamic backends . • We systematically investigated the key factors contributing to the differentiable physics based learning framework . We believe our investigation can inspire further researches to explore the possibilities of differentiable physics for more complex learning tasks . 2 DIFFERENTIABLE SIMULATION ENVIRONMENTS . 2.1 SIMULATION SETTINGS . Our learning framework supports multiple types of differentiable physically-based deformable body simulators . The simulator can be seen as a black box that outputs the states for the next time step given the actuation signal and states from the current time step . The NN controller takes the information given by the simulator as part of its input features and decides the actuation signal for the simulator in the next time step . With our automatic differentiation system , we can evaluate the gradient information to guide the training of our NN controllers . We use the mass-spring systems and the moving least squares material point method ( MLS-MPM ) as our simulation environments . The design choices for each environment can be found in the appendix . 2.2 AGENT STRUCTURE AND ACTUATION MODE . Our framework supports a variety of differently shaped agents as shown in Fig . 2 and 3 . Our actuation signal exerts forces along with the ” muscle ” directions of the simulated agents . This signal falls within the interval [ −1 , +1 ] , where its sign determines whether an agent wants to contract ( negative sign ) or relax ( positive sign ) a muscle and its absolute value determines the magnitude of the force . In mass-spring systems , the muscle directions are represented by the spring directions , we change the rest-length of springs to generate forces . The actuation signal is used to scale the rest-length up to some limit ( usually at 20 % ) . In addition , only activated springs , i.e . actuators ( springs marked red or blue shown in Fig . 1 ) , are able to generate forces according to control signals . In material point methods , the muscle directions are always along the vertical axis in material space . In our implementation , we simply modify the Cauchy stress in the vertical direction of material space to apply this force . Again , the force is scaled to a user-defined bound which can be adjusted for different simulations . 3 LEARNING FRAMEWORK . In this section , we describe our training framework based on differentiable physics ( Fig . 4 ) . In summary , we develop a differentiable simulator with an embedded NN controller . In each time step , the program performs an NN controller inference and a differentiable simulator time integration . After a few hundred time steps , the program back-propagates gradients end-to-end via reverse-mode automatic differentiation , and updates the weights of the neural network using the Adam optimizer Kingma & Ba ( 2014 ) . 3.1 TASK REPRESENTATION . Given an agent A , its position and corresponding velocity are denoted as x = ( x , y , z ) and v = ( u , v , w ) . We presents three tasks ( running , jumping and crawling ) for 2D agents , two ( running and rotating ) for 3D . In a whole simulation with total steps T , an agent is expected to achieve all goals in a goal sequence G = { G1 , G2 ... Gn|nP = T } , where P is the time period for an agent to achieve a goal , n is the number of goals . For each goal G , there are multiple tasks in it . Each task is encoded using a target value g , which instructs the controller to drive the agent . For example , G = { gv , gh } represent performing running and jumping simultaneously , the target velocity and height are gv and gh . The agent is expected to switch between multiple goals during one simulation . Running . The running task is defined as a velocity tracking problem . Given a time step t , the center of mass c is used to represent the position of the agent , its velocity is defined as the averaged velocity ṽ over a time period Pr , ( Pr < P ) . Then the agent is expected to achieve a velocity ṽ close to the target velocity gv . ṽ ( t ) = 1 Pr ( c ( t ) − c ( t− Pr ) ) ( 1 ) Jumping . The jumping task is defined as reaching a given height gh in a time period P . Due to the gravity , it is not possible for a running robot to stay away from the ground . Therefore , the jumping height h̃h ( t ) of an agent is defined by the maximum vertical position of its lowest point in a certain period P . To be more formally , given S a set of nodes ( or points ) that constitute the agent , h is a function that can extract the vertical position given a node ( or point ) , the jumping height over the n-th period is defined as h̃h ( n ) = max t∈ [ 0 , P ] min s∈S h ( s , t+ nP ) ( 2 ) where n is the index of the period in a whole simulation . Crawling . The crawling task is defined as lowering the highest point as much as possible . The target of crawling gc is defined as an indicator function . The agent is controlled to crawl if gc = 1 otherwise if gc = 0 . Since crawling is a status that needs to be maintained , the crawling height is defined as the vertical position of its highest point at each time step . h̃c ( t ) = max s∈S h ( s , t ) ( 3 ) 3.2 LOSS FUNCTIONS . Consider a locomotion task where an agent is instructed to move at a specified velocity , we think a proper loss function should satisfy the following requirements : 1 . Periodicity . The agent is expected to move periodically following a predefined cyclic activation signal . Therefore , the loss function should encourage periodic motions . 2 . Delayed evaluation . Due to inertia , it takes time for the agent to start running or adjust running velocity . Therefore , an ideal loss function should take this delay into consideration . 3 . Fluctuation tolerance . During a single running cycle , requiring the center of mass to move at a constant velocity at each time step may induce highly fluctuated losses . To avoid that , our loss function should be smooth during one motion cycle . Task loss . For each task , we defined a tailored loss shown in equation 4 according to the requirements above . The loss for running Lv is defined as the accumulation of the difference between the target and the agent ’ s velocity from start to current time step . For jumping , we define a sparse loss Lh , which only evaluate once for each period . The crawling loss Lc is defined as the accumulation of the crawling height if the loss is applicable . L = λv ∑ n∈T P∑ t=Pr ( ṽ ( t ) − gv ( n ) ) 2︸ ︷︷ ︸ Lv +λh ∑ n∈T ( h̃h ( n ) − gh ( n ) ) 2︸ ︷︷ ︸ Lh +λc ∑ n∈T P∑ t=0 gc ( n ) h̃c ( t+ nP ) ︸ ︷︷ ︸ Lc ( 4 ) The λv , λh , λc are weights for losses of dfferent tasks . Our loss function accumulates the contributions of all steps in a “ sliding window ” . This prevents the loss function from being too small as well as the vanishing gradient problem . We find that having an accumulated loss evaluated at every step works much better when we want to control the speed of the agent explicitly . Regularization . Since the agents may keep shaking when no target velocity or height are given , we introduce an actuation loss as a regularization term . The intuition is to penalize comparatively large actuations when small goals value is given : La = ∑ n∈T ∑ t∈ [ 0 , P ] ( A ( t ) − µ||gv ( n ) || ) 2 ( 5 ) where A ( t ) is the actuation output of NN , µ is a normalizer . The actuation loss balances the importance of input channels of the NN , i.e. , different input channels produce similar contributions to the first layer of the trained NN .

Summary: This work proposes to learn locomotion skills on soft robots made of springs. It proposes to leverage differentiable physics along with a NN controller based on SIREN, which allows to directly learn policies by minimizing loss functions defined on trajectories. The methods allows to learn locomotion and jump behaviors on a various set of 2D and 3D mass-spring systems.