Learning Quadruped Locomotion Using Differentiable Simulation
Authors: Yunlong Song
Sangbae Kim
Davide Scaramuzza
Venue: CoRL 2024 (2024)
Keywords: Differentiable Simulation
Legged Locomotion
Reinforcement Learning
TL;DR: Differentiable simulation allows for learning quadruped locomotion over challenging terrains effectively.

Abstract: This work explores the potential of using differentiable simulation for learning robot control. Differentiable simulation promises fast convergence and stable training by computing low-variance first-order gradients using the robot model. Still, so far, its usage for legged robots is limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics.  This work proposes a new differentiable simulation framework to overcome these challenges. The key idea involves decoupling the complex whole-body simulation, which may exhibit discontinuities due to contact into two separate continuous domains. Subsequently, we align the robot state resulting from the simplified model with a more precise, non-differentiable simulator to maintain sufficient simulation accuracy. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains in minutes. We demonstrate that differentiable simulation outperforms a reinforcement 
learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. Our policy achieves robust locomotion performance in the real world zero-shot.

Learning Quadruped Locomotion Using Differentiable Simulation
Yunlong Song University of Zurich, Switzerland song@ifi. www.uz.... ch
Sangbae Kim MIT. USA sangbae@mit. edu
Davide Scaramuzza University Zurich, Switzerland sdavide@ifi. wuw.eehffiiuu...e.i. .uzh.ch ch
Abstract: This work explores the potential of using differentiable simulation for learning quadruped locomotion. Differentiable simulation promises fast conver- gence and stable training by computing low-variance first-order gradients using robot dynamics. However, its usage for legged robots is still limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics. This work proposes a new differentiable simu- lation framework to overcome these challenges. Our approach combines a high- fidelity, con-differentiable simulator for forward dynamics with simplified sur- rogate model for gradient hack-propagation. This approach maintains simulation accuracy by aligning the robot states from the surrogate model with those of the precise, non-differentiable simulator. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains minutes. We demonstrate that dif- ferentiable simulation outperforms reinforcement learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effective- ness in handling large-scale environments. Our method represents of the first successful applications of differentiable simulation real-world quadruped loco- motion, offering compelling alternative to traditional RL methods. Video: https:ttp..... : /youtuuuuuuuuooooo be/ weNg_wTg w715MM
Keywords: Differentiable Simulation, Legged Locomotion
1 Introduction
Traditional model-free reinforcement learning (RL) of- ten requires extensive parallelization to achieve stable walking policy for legged robots [1]. How can legged robots master walking quickly with few trials and er- rors? Recent progress in legged robot control has been largely driven by combining model-free RL with mas- sively parallelized simulation [1, 2, 3, 4, 5, 6, 7]. Cen- tral these advancements the policy gradient algo- rithm [8, 9], where the gradient of control policy be calculated via (0) E,wra [R(r)Va log p(T; 0)]; which is a zeroth-order estimate of the true gradient of J based on sampled trajectories following the policy ma- Here, R(T) is the reward trajectory т. Such zeroth- order gradient estimate provides powerful and general framework for robot control since it can handle non-differentiable optimization objectives and dis- continuous dynamics. However, the gradient has high variances. Consequently, several additional strategies required for stable training, such as using clipped surrogate objective [10] and in- creasing the sample size.
Differentiable Simulation
O+
t
forward bockward
f.
f(e,00
Uhe action (f control policy
Lg
differentiable
differentiabll dynamics
Figure 1: Graphical model for policy learning using differentiable simulation.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

In robotics, leveraging well eslabbisall knowledges about robot dynamics can enable the construe- tion of first-order gradient estimates (0. = Ve R(T), which typically exhibit significantly lower variance than their zeroth-order counterparts and hold great potential for more stable training and faster convergence. Recently, policy training using first-order gradient has been notably advanced through differentiable simulation [11. 12, 13, 14, 15]: these works have shown promising results in reducing both the number of simulation samples and total training time compared to zeroth-order methods. Differentiable simulation combines the advantages of model-based control and data-driven learning. Similar to numerical optimal control methods, such as shooting techniques, differentiable simulation utilizes robot dynamics to simulate trajectories and iteratively optimize decision variables using analytical gradients derived from the robot model. Like reinforcement learning, differentiable simulation benefits from large-scale simulations and deep learning to train neural network control policies that map observations directly to control actions. Despite its advantages. differentiable simulation faces several challenges, including discontinuous dynamics, complex optimization land- scapes, and issues like exploding vanishing gradients in long-horizon tasks, which can limit the effectiveness of first-order gradient methods.
This work investigates the potential of training policies through surrogate models within differ- entiable simulation, specifically focusing on quadruped locomotion. We show that differentiable simulation offers considerable advantages over model-free RL for policy training in legged locomo- quickly
tion. Notably, demonstrate that single robot, without the need for parallelization learn to walk within minutes in simulation using our approach. Leveraging the advantage of GPUs for parallelized simulation, our robot learns diverse walking skills over challenging terrain in min- utes. Specifically, we train a quadruped robot to walk with different gait patterns, including trot, pace, bound, and gallop, and with varying gait frequencies. While model-free RL. can also achieve similar performance given sufficient parallelization, our approach requíres much less data, achiev- ing significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. More importantly, we show that the policy trained via differentiable simulation can be transferred the real world directly without fine-tuning, This work presents of the carliest demonstrations of using differentiable simulation to train control policies for real quadruped robots, highlighting the potential of differentiable simulation for real-world applications.
Contribution: The key to our approach a novel policy training framework that combines the smooth gradients obtained from a simple surrogate dynamics model for efficient backpropagation with the high fidelity of more complex, non-differentiable simulator for accurate forward simu- lation. Instead of simulating the quadruped robot using whole-body dynamics, which by definition has discontinuities due to contact, we propose separating the simulation into its floating base space and its joint space. For the robot body, we employ an approximation using single rigid-body dynam- ics, which offers a continuous and effective representation of the robot body. To address simulation errors arising from our simplified rigid-body dynamics model, we incorporate a more precise, non- differentiable simulator. This non-differentiable simulator can simulate complex contact dynamics and is used to align the state in our simplified model, thereby ensuring that our training pipeline remains grounded in realistic dynamics. Figure 2 provides an overview of our system.
Methodology
2.1 Problem Formulation
We formulate legged robot control as an optimization problem. The robot is modeled as a discrete- time dynamical system, characterized by continuous state and control input spaces, denoted as X and l4, respectively. At each time step k, the system state is Ia E X, and the corresponding control input is 4g E U. An observation 03 E O is generated at each time step based on the current state Ikk through a sensor model h X - O, such that Og = 1.(x))) The system's dynamics are governed by the function f X U + X, which describes the úmmeddicrretzeed evolution of the system as Ik+1 f ).). At each time step k the robot receives cost signal I(2k, uz), which is function of the current state 23 and the control input ug. The control policy is represented

---

Whol-Ooo Dynankk
Alignment
dbservatore
Miceelidty Dysuimica
FA-s
Diffroniaatle
Polic
jortang's
PD Contral
Bingle Rigil Buity Dynamies
bave stat
Vip
V&C
fifs
Ditorestible
comvandd
Slngle
Figure 2: System overview of learning quadruped locomotion using differentiable simulation. Our approach decouples the robot dynamics into two separate spaces: joint and floating base spaces. single rigid-body dynamics for the robot's
We leverage the diffcrentiability and smoothness main body, which takes the ground reaction force from its legs the control inputs. We use the state from a non-differentiable simulator (IsaacGym) to align the state in the differentiable simulation.
deterministic, differentiable function such as a neural network uy = Th TT(())) The neural network takes the observation og as input and outputs control input ug. The optimization objective is to find the optimal policy parameters 6* by minimizing the total loss via gradient descent
N-1
N-1
minCo = Σ Ma ux)
Σ (1rk To(o)))
(1)
K=0
k=0
θ - VoLo,
(2)
where a is the learning rate and (0*, ux) is the differentiable loss at simulation time step k.
2.2 Forward Simulation
We represent the robot's main body using single rigid-body dynamics. Single rigid-body dynamics have been shown be useful for dynamic locomotion using model predictive control [16, 17]. The single rigid-body model offers a continuous representation of the robot base dynamics, avoiding the complex optimization landscape introduced by contacts. We develop our differentiable simulation using the single rigid-body dynamics, which is expressed as follows,
VWB m Σ
pwB = VwB
1 qwB A(wB) qWB
wB I (n WB x (IwB))
In this approximation, the state of our system x (p, q. V. w], where PWB E R3 is the position and VwB R is the linear velocity of the center of mass in world frame W, We use unit quaternion QwB to represent the orientation of the body in the world frame and use wn to denote the body rates in the body frame B. Here, is the robot's inertia tensor, n the body torque, and A(wB) is a skew-symmetric matrix. The control inputs the ground reaction foree f, from the legs that have contacts.
The ground reaction forces are required to simulate the rigid-body dynamics. One option for the control policy design to output the ground reaction force directly, similar to the MPC design [16, 17]. Another option control the robot the joint space, e.g., output desired joint position, which allows more control authority for the policy and adaptive behavior. Our policy maps observations to the desired joint position qd while assuming zero joint velocity, i.e., qud 0. In this case, the neural network output (joint position) is required to be converted into the control input of the single rigid-body model, which is the ground reaction force. This conversion is achieved using PD controller for forward propagation
kp (d - q) + ka (qree 4).
(3)
which calculates the required motor torque Here, kp and ka fixed gains, re and q the reference joint position and the current joint position separately, and qee and å the reference
3

---

joint velocity and the current joint velocity respectively. Subsequently, the motor torques are then converted to ground reaction forces f using the foot Jacobian J: f (J²)- The continuous nature of PD control enables the backpropagation of policy gradients. Consequently, our simulation framework treats the PD controller as a differentiable layer.
2.3 Backpropagation Through Time
In differentiable simulation for policy learning, the backward pass is crucial for computing the an- alytic gradient of the objective function with respect to the policy parameters. Following [18), the policy gradient can be expressed as follows
N-1 VoLy
Ol
Oxj θrz-1
dx; 00
Ölg
Ouk
(4)
N
Dæ
Oue 00
jim
differentiahle dynamics
where the matrix of partial derivatives 0 is the Jacobian of the dynamical system f. There- fore, we can compute the policy gradient directly by backpropagating through the differentiable physics model and a loss function 1 that differentiable with respect to the system state and con- trol inputs. A graphical model for gradient backpropagation policy learning using differentiable simulation is given in Figure Due to the usage multiplication I[. there are two potential issues in using Eq. (4) for policy gradient: gradient vanishing or exploding, 2) long computation time. We tackle these two problems via short-horizon policy training.
2.4 State Alignment with A Non Don-Diffeernttable Simulator
Due to the simplification of our single rigid-body dynamics and the decoupling of the simulation space, the robot state can diverge from its actual one. Over time, even minor discrepancies can ac- cumulate, leading to unrealistic states and ultimately causing the failure of policy training. We pro- posed to align the body state in our differentiable simulation using information from other simulators that use accurate whole-body dynamics. Specifically, we align the robot state in our differentiable simulation with the state information from IsaacCimm [19].
We use the following equation to align the robot state R = Xt+1
x + a (xx+1 diff, detuch )
X
Here, ditt, dstach and xdiff represent the same robot state of our differentiable simulator time step x++1
t + 1; hence, they share the same value. The word detach indicates that x detach is detached from the computational graph for automatic differentiation. Therefore, we can reset the robot state using the value from the non-differeentiale simulation during forward simulation: Sdiif x mom- +
O. During backpropagation, the gradient of the state at given time is computed follows 0x:liff /0xefit 0 ix!liff /xxf Here, 0 < is used to decay the gradient.
The non-differentiable simulator can simulate complex contact dynamics and is used to align our simplified model, thereby ensuring that our differentiable training pipeline remains grounded in realistic dynamics. At the same time, our approach benefits from the simplified differentiable simu- lator, which offers smooth gradients for backpropagation. Figure shows the computational graph of the forward propagation and backpropagation using state alignment.
2.5 Short-Horizon Policy Training
Although smoothed physical models (c.g., single rigid-body dynamics) improve the local optimiza- tion landscape, the complexity of the optimization problem escalates significantly long-horizon problems involving extensive concatenation of simulation steps. The situation further deteriorates when the actions within each step are interconnected through a nonlinear and nonconvex neural network control policy. The complexity of the resulting optimization landscape can make gradient- based methods to become trapped in local optima quickly. Instead of directly solving long-horize policy training problem, following [13, 1]. where long-horizon simulation tasks are truncated into

---

short- hort-oo simulation, we utilize short- horizon policy learning to achieve stable gradient back- propagation over a short horizon. Mathematically, short-horizon policy learning involves truncating a long-horizon trajectory of length N into several shorter segments to compute the policy gradient shown in Eq. (4). For instance, for a trajectory of length 240, we divide into ten segments of N 24. However, it's important to note that while a shorter horizon could simplity computation and facilitate optimization, may also limit the model's ability anticipate future events, introduc- ing bias into the gradient calculation. This bias could, in turn, affect the types of tasks the model can effectively learn.
2.6 Differentiable Loss Function
fundamental difference between policy training using differentiable simulation and reinforcement learning is that the loss function has to be differentiable when using differentiable simulation. RL allows the direct optimization of non-differentiable rewards, such as binary rewards 0/1. On the contrary, differentiable simulation requires smooth differentiable cost function to provide learning signals for the desired control inputs. We formulate a differentiable loss function [(x,, u,) tailored for velocity tracking, where the main objective is to follow specified velocity command, denoted as vr Additionally, we maintain the robot's body height, represented by p=- To enhance the robustness of this system, we incorporate several regularization terms: one to mitigate large angular velocity, thus controlling the body rate w; another to limit the output action u, preventing large control actions; and term aimed at stabilizing the robot's orientation using projected gravity vector Sproj- The loss function is defined
(xx, Us)
a1llv a1llv- 1 + azllp= P + ag ///
+aallu|/² as Bproy ag Proot pjat 112
(5)
where Pfoot is the foot position.
3 Experimental Results
3.1
A Toy Example: Control of A Double Integrator
Training Progress
Trajectories
Control Input
DilSinnenw=))
250
PPO
PPO/exv=1020 1024)
Valockay (2)
Looe
LQR (optimal)
Commund
200
DidSiim
100
250
750
1000
Triining Fieracioon
Pocition
Timee Step
Figure 3: Control of a double integrator using optimal control, reinforcement learning, and differentialee simulation. (left): Learning curves. (middle): Trajectories of different control poli- cies. We initialize the system at the same states for all methods. (right): DiffSim achieves control commands close to optimal control.
Inspired by [20], where the author studied the connection between reinforcement learning and op- timal control using toy double integrator example, we begin by examining the same toy example: control of a double integrator, which is fundamental problem in system and control theory. The double integrator a second-order control system that models simple point mass dynamics one- dimensional space. The state variables are position æ and velocity æ;, with control inputs u = H. The objective is to drive the system state from any initial point to the origin, [æ, = [0, 0]. Therefore, this problem can be solved by the Linear quadratic regulator (LQR), which can provide an optimal solution. We train 2-layer multiplayer perception using backpropagation through time via differ- entiable simulation and the proximal policy optimization (PPO) method [10]. We show that the
5

---

policy trained using differentiable simulation (DiffSim) achieves nearly optimal performance within a handful of training iterations and samples. In contrast, even when scaling the number of simulation environments to 1024 and training the policy with considerably more iterations, PPO fails to achieve the same level of control performance as LQR or DiffSim. This toy example suggests that scaling might be insufficient for PPO achieve an optimal solution.
3.2 Learning Walk with One Robot
This section explores whether a robot can learn to walk without parallclization and with very limited data points at each training iteration. We design a simple veloeity-tracking task where the robot is re- quired to follow constant velocity in the x-axis, e.g., = 0.2 m/s. /. At each training iteration, we simulate 24-time steps with one single robot. Hence, each training iteration contains 24 data points. We trained the policy for total of 1000 iterations. The result given by the learning curve in Figure De- spite very limited data, our policy successfully learns to walk after minutes of training. As comparison, PPO failed to achieve useful locomotion skills un- der the same condition. By providing first-order gra- dients with low variance, differentiable simulations offer a fundamental advantage: they allow for more efficient and reliable optimization of the control policy. This because the gradient provides a clear direction for updating parameters, and the low variance ensures that this direction is consistent and reliable, even when data is limited.
Learning Curve
2001
4001 6001
5.00
1000
Tritiig
Figure Learning to walk with one sim- ulated robot. We run 10 experiments with different random seeds. The plot is smoothed using moving average.
3.3 Learning Diverse Walking Skills on Challenging Terrains
We design a more difficult task: learning diverse locomotion skills over challenging terrains. Specif- ically, we design four different gait patterns, including trot, pace, bound, and gallop. Additionally, vary the gait frequency from Hz to 4 Hz. The robot receives high-level velocity command and is required to track randomly commanded velocity. Figure 5 shows the learning curves using different numbers of robots for policy training. We compare the learning performance of our method with model-free RL algorithm (PPO) [10]. Given minimal samples, e.g., only four robots, RL has a slow convergence speed and does not learn meaningful walking skills, e.g., the policy constantly falls after a few simulation steps. In contrast, differentiable simulation achieves much higher rewards and can acquire useful walking skills.
As the number of robots increases, the performance of both algorithms improves. Notably, the performance improvement for RL is much more significant than our approach. This indicates that the zeroth-order gradient estimates used by reinforcement learning are generally inaccurate and require many more samples to achieve stable training. On the contrary, the first-order gradient estimates used by our differentiable simulation can have very stable and accurate gradients, even given very limited simulation samples. Figure demonstrates diverse walking skills over challenging terrains using a blind policy trained via differentiable simulation.
We demonstrate the performance of our policy in the real world using Mini Cheetah [21]. Mini Cheetah is a small and inexpensive, yet powerful and mechanically robust quadruped robot, intended to enable the rapid development of control systems for legged robots. The robot uses custom back- driveable modular actuators, which enable high- bandwidhdwdth force control, high force density, and robustness to impacts [21]. The control policy runs 100 Hz during deployment. Figure shows several snapshots of the robot's 's walking behavior over different terrains using different gait patterns. We trained blind policy in simulation using 64 robots and then transferred the policy directly to

---

Differentiable Simulation
Reinforcement Leamning (PPO)
Reward
10
-15
Robess
-15
64 Robots
1024 Raboos
200
600
1000
200
400
600
800
1000
Trot
Bound
Pace
Gallop
Figure 5: Learning to walk on challenging terrains, reinforcement learning versus differen- tiable simulation. Differentiable simulation exhibits significant advantages over PPO in terms of sample efficiency and learning stability. After training in simulation, the policy can be transferred to the real world without tne-tuning.
the real world without fine-tuning. The robot can walk forward and backward with different gait patterns and frequencies. Moreover, the policy enables robot manage disturbances, such as unexpected forces and deformable terrain.
Additionally, we compare the total training wall-clock time and the fi-
Final
Training [min]
nal performance of the resulting poli-
Rohots
PPO
Ours
PPO
Ours
cies. Differentiable simulation of-
25.82 2.16
-3.83 3.01
7.27 0.12
7.28 + 0.12
ten involves the creation of com- plex computational graph to facilitate gradient computation. As the simu- lation horizon extends, the length of the computational graph can increase proportionally, leading to a substan- tial increase in the total training time. This increase can diminish bene-
16
-8.07
-2.03
+ 0.06
-2.49 0.37
1024
0.13
-2.69
0.06
11.44 + 0.13 11.55 - 0.09
Table I: A comparison of final reward and total training time. We both methods for the same number of training iterations and collect the same amount of samples.
fits in situations where gathering large number of samples both cheap and fast. Therefore, we ask the question: Can differentiable simulation be effectively applied to large-scale settings, for both PPO and train-
example, thousands of environments? To ensure fair comparison, ing framework for the same number of training iterations, namely 1000 iterations. We compare the total training time, which includes the time for simulating dynamics and updating policies. As shown in Table 1, our results indicate that differentiable simulation can be as time efficient as PPO in large-scale environments despite the requirement of backpropagation through time (BPTT) process known for computational demands. However, notice that the training time reach certain reward is much lower with our method and requires less robot and data. For example, our method requires only roughly 64 robots and 200 training iterations solve this task, while PPO still performs less after 1000 training iterations with 1024 robots.

---

Related Work
Previous efforts in leveraging differentiable simulation to robot control can be broadly categorized into two classes: improving the simulator itself and enhancing the associated optimization algo- rithms. The first kind has led to the development of several general- differentiable simula- tors, including Brax [11], Nimble [22], NeuralSim [23], and Dojo [24]. These simulators enable the integration of robot models directly into policy optimization, allowing neural network control policies to be trained using more mathematically precise analytical gradients. However, noted in [18, 25], in scenarios where the underlying system exhibits chaotic behavior-where small per- turbations in initial conditions lead to significantly divergent states, such as in robotic simulations involving contact dynamics-thee
gradients can become unmanageable, leading to divergence in optimization processes. Our work highlights the benefits of employing well-behaved proxy dynam- ies as an alternative to the true robot dynamics involving contact. Such a design choice has roots in model predictive control, which often leverages a simplified model for the optimization control complex robotic systems [16, 17, 26).
Besides developing a better simulator, various methods have been proposed to improve policy train- ing algorithms. For example, [25] proposes an a-order gradient estimator that aims to combine the efficiency of first-order estimates with the robustness zeroth-order methods. SHAC [13] ad- dresses local minima issues by learning a smooth value function and mitigates vanishing/exploding gradients through truncated backpropagation. Gradient norm clipping strategies have also proven effective in managing the exploding gradients issue [27]. Furthermore, since differentiating through chaotic dynamics is challenging, it is often useful to learn approximation of these functions and use the approximation for gradient computation. This approach commonly has connection with model-based reinforcement learning [28, 29, 30, 31, 32], which involves learning a predictive model of the environment, followed by training a controller. Similar to [13], our method leverages a truncated backpropagation to facilitate optimization. Our method could improved further by combining other ideas, such learning a critic function or clipping the gradients.
5 Limitations
A key advantage of model-free RL lies in its ability to directly optimize task-level rewards [33], especially non-differentiable rewards. In contrast, differential simulation depends on the backprop- agation of information from its objective function, making challenging to explore novel solutions guided by task objectives. Consequently, our system requires specifying foot positions using the Raibert heuristic [34], as cannot explore foot placement motion through velocity tracking loss alone. In addition, RL's flexibility allows for more robust performance enhancements, such im- plementing simple termination penalties discourage falling behaviors or employing constant sur- vival rewards to encourage continued operation. Furthermore, differentiable simulations are often tailored to specific optimization tasks and might not generalize well across different problems. It requires specific domain knowledge in both building the simulation pipeline and formulating the problem. To generalize our approach to other tasks, additional engineering efforts are required.
Conclusion
This work proposes a new perspective on learning quadruped locomotion policies using differen- tiable simulation by integrating a simplified rigid-body model with non-differentiable model for whole-body dynamics. Our key insight is that neural network control policy can be optimized directly by backpropagating the differentiable loss function through simplified model while simul- tancously aligning the robot's state using a more accurate whole-body dynamics model. Future work should address several limitations of the proposed framework, such as removing the assumption of gait scheduling and the Raibert heuristic for foot contact sequence and foot position planning. Ad- ditionally, more sim-to-real analysis helpful in evaluating the benefits of using a simplified model compared to the true but chaotic gradients to the policy trained via domain randomization. Finally, should demonstrate the true advantages of stable training introduced by the first-order gradient by tackling more challenging tasks, such as large-scale training and vision-based control.
8

---

7 Acknowledgment
This work was supported by the European Union's Horizon Europe Research and Innovation Pro- gramme under grant agreement No. 101 120732 (AUTOASSESS) and the European Research Coun- cil (ERC) under grant agreement No. 864042 (AGILEFLIGHT) The authors especially thank Yuang Zhang for his help with the differentiable simulation development and for the useful discussion, Se- ungwoo Hong for the useful discussion about rigid-body dynamics and nonlinear model predictive control, and Nico Messikommer for his valuable feedback. Also, the authors thank Se Hwan Jeon, Ho Jae Lee, Matthew Chignoli, Steve Heim, Elijah Stanger-Jones, and Charles Khazoom for their helpful support.
References
[1] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning, In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[2] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26))cau5582,, 2019.
[3] G. Margolis, G. Yang, Paigwar, T. Chen, and Agrawal. Rapid locomotion via reinforce- ment learning. In Robotics: Science and Systems, 2022.
[4] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149, PMLR, 2023.
[5] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL), 2023.
[6] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
[7] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11443-11450. IEEE, 2024.
[8] R. S. Sutton, McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[9] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8:229-256, 1992.
[10] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063777 2017.
[11] C. D. Freeman, F. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax-a differen- tiable physics engine for large scale rigid body simulation. arXiv preprint 10fi::200...288... 2021.
[12] Z. Huang, Y. Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan. Plasticinelab: A soft- body manipulation benchmark with differentiable physics. arXiv preprint arX1v:2104.03311, 2021.
[13] Xu, V. Makoviychuk, Narang, F. Ramos, W. Matusik, A. Garg, and M. Macklin. Accel- erated policy learning with parallel differentiable simulation. In International Conference Learning Representations, 2021.

---

[14] J. Ren, C. Yu, S. Chen, X. Ma, L. Pan, and Z. Liu. Diffmimic: Efficient motion mimicking with differentiable physics. 2023.
[15] J. Xu, S. Kim, I. Chen, A. R. Garcia, P. Agrawal, W. Matusik, and S. Sueda. Efficient tactile simulation with differentiability for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022.
[16] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim. Dynamic locomotion in the mit cheetah 3 through convex model-predictive control. In 2018 IEEE/RSJ international confer- ence intelligent robots and systems (IROS), pages 1-9. IEFE, 2018
[17] S. Hong, J. Kim, and H.-W. Park. Real-time constrained nonlinear model predictive control (3) for dynamic legged locomotion. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3982-3989. IEEE, 2020.
[18] L.. Metz, C. D. Freeman, S. Schoenholz, and T. Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05800 2021.
[19] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin,
A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[20] B. Recht. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2:253-279, 2019.
[21] B. Katz, J. D. Carlo, and S. Kim. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. In 2019 International Conference on Robotics and Automation (ICRA), pages
2019. doi:10. 1119///A..2199.78952...
[22] K. Werling, D. Omens, J. Lee, I. Exarchos, and C. K. Liu. Fast and feature-complete differ- entiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.1602) 2021.
[23] E. Heiden, D. Millard, E. Coumans, Y. Sheng, and G. S. Sukhatme. NeuralSim: Augment- ing differentiable simulators with neural networks. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2021. URL https: //gpt/ub/ com/ simulator.
google google-reseee wwweerrreeeeeeeeeeeeeeeeeeeeeeee oe iabee
[24] T. A. Howell, S. L. Cleac" h, J. Brüidgamm, J. Z. Kolter, M. Schwager, and Z. Manchester. Dojo: A differentiable physics engine for robotics. arXiv preprint 2xXiv:2233000800 2022.
[25] H. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? In International Conference Machine Learning, pages 20668-20696.
PMLR, 2022.
[26] Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, Permenter, T. Koolen, P. Marion, and Tedrake. Optimization-base locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 40(3):429 455, 2016.
[27] R. Pascanu, Mikolov, and Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
[28] D. Ha and Schmidhuber. World models. arXiv preprint /803. 10122, 2018.
[29] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference machine learning (ICML-
11), pages 465-472, 2011.
[30] D. Hafner, J. Pasukonis. J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04144, 2023.
10

---

[31] J. Schrittwieser. I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839) 609, 2020.
[32] P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg, Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226 2240. PMLR, 2023.
[33] Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza. Reaching the limit in au- tonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82): cadg1462, 2023.
[34] M. Raibert. Legged robots that balance. MIT press, 1986.
[35] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots. Fast and efficient locomotion via learned gait transitions. In Conference on Robot Learning, pages 773-783 PMLR, 2022.
[36] Y. Yang, G. Shi, X. Meng, W. Yu, T. Zhang, Tan, and B. Boots. Cajun: Continuous adaptive jumping using learned centroidal controller. arXiv preprint 27X5v:2306.09557, 2023.

---

Appendix
Double Integrator
double integrator system is characterized by its position and velocity. The control input a directly controls the acceleration of the system. For a fair comparison, we formulate the problem discrete-time
optimal control problem. The discrete-time state space representation of a double integrator is
Fk+1 Ark + Buk
1 dz and B
df/2 dt
where
Here, de is the simulation time step. The objective is to minimize quadratic loss function
1
J ==
Σ
(o Quy + u Rus)
k=0
[
Q Q= Q =
[1].
0
This is discrete-time finite-horizon Linear-Quadratie Quadratic Regulator problem, where the optimal con- trol law can be found using dynamic programming
u -R PE+I PR+IAUEs 0, N 1,
where Pn Qf and Ps can be found from the Riccati recursion.
2 Experimental Setup
We develop our own differentiable simulator using PyTorch and CUDA. Our differentiable simulator allows for both forward propagation the robot dynamics and backprop- agation of the policy gradient. Additionally, we run IsaacGym [19] alongside our differentiable simulation and use it to align the robot state resulting from our simplified robot dynamics. Isaac- Gym simulates the whole-body dynamics and complicated contacts between the robot and its en- vironment. Both simulations are parallelized on GPU. We used discretized simulation time step of 0.002 and a control frequency of 100 Hz. We use PPO baseline from [1]. Additionally, we follow prior works [35, 36] for the implementation of the gait schedule and Raibert Heuristic.
Simulation Setup:
Observation and Action: The policy observation includes random commands (emdrand) for the reference velocity, sinusoidal and cosinusoidal representations of gait phases, the base velocity (VwB), the base orientation (qwB), the angular velocity (WB), motor position deviations from default (q quelautt), and a projected gravity vector (gprojected). The policy action öq is the desired joint position offset from the default joint position.
Observation
Dimension
Action
Dimension
emd,,st
simi phase)
cos(gait phase)
Vw'
89
qw.B
W
q Qleresti
Speiected
Table 2: Policy observation and action for quadruped locomotion.
12

---

3 Foot Trajectory Planning
We use the foot position loss | Proal Il to provide a learning signal for the swing legs. This loss term critical for the swing leg since it contains information about the motor position. Following prior works [35, 36], the swing leg trajectory can be computed by fitting quadratie polynomial over the offt-off Pfoce mid-air Pfooe and landing position Pinot of each foot, where the lift-off position is the foot location at the beginning of the swing phase, the landing position Pioot the Raibert Heuristics [34], which is expressed as the following function
land is calculated using
P Foot
land
Pfoot
hip
V
stance
where arace is the expected time the foot will spend on the ground, Pieat is the location on the ground beneath the robot's hip, v CoM is the body velocity projected on the xy-plane. The desired contact state each leg. e.g., swing or stance, is determined via a gait generator. The gait is modulated by a phase variable [o, 2T]. The phase defined through dynamic function Φt+1 Φr 2TT f At, where is the stepping frequency. We can design different locomotion patterns by adapting the stepping frequency f and the phase difference between each leg.
4 On the Importance of Non-differentiable Terminal Penalty
We highlight one important benefit of RL compared to differentiable simulation: RL can signifi- cantly enhance its robustness by directly optimizing through non-differentiable rewards or penalties. Specifically, we use -on-differentiable value 200 to penalize the robot when the robot experi- ences termination during training, e.g., falling on the ground or lifting its legs above its body.
-1(xx, u,) -p p if termination r(x, u,) = -1((x,, u,) otherwise.
Fig. shows a study of using non-differentiable terminal penalty for both RL and differentiable simulation. The results show that adding a final penalty can greatly affect how well RL works. Without penalty, RL might get trapped in a local minimum. However, with a large penalty the end, RL can achieve better task rewards well as more robust control performance. This is because RL optimizes discounted return, which estimates "how good" it is to be in given state. RL uses state-value function to encode this information
V (5) S. s)
E
Σ 7d Re+1|So
On the other hand, terminal penalty has no impact on differentiable simulation since the gradient of a constant value is equal to zero, and we do not leverage a state value function. As a result, differentiable simulation requires well-defined continuous functions, e.g., a potential function or control barrier functions for robust control.
Ablation Study
We conducted an ablation study to investigate the significance of the proposed state alignment mech- anism. In our implementation, we use the state from IsaacGym align the robot state within our simplified rigid-body differentiable simulation. To assess the impact of state alignment, formed experiments where we removed the state alignment and compared the resulting learning curves to those obtained with state alignment. The results are presented in Figure 7. The results indicate that without state alignment, the robot fails to learn any useful walking skills.
per-
6 Hyperparameters
13

---

0.0
M
-2.5
Episodo Lesgeta
-3.0
Reward
7.8
diff (withoot terminal penuity)
10.0
dEf (with terminal penalty)
(without terminal peuallyy)
-12.5
pgo (with terminni penclty')
200
400
6000
500
1000
2003
400
500
800
1000
Iteration
Figure 6: A comparison of non-differentiable terminal penalty for policy training. Using non- differentiable terminal penalty, PPO can achieve robust control performance, e.g., longer episode length. We use 1024 robots for simulation.
-10
Reward
-20
-30
(With) Alignment
(Without) Alignment
200
400
600
800
1000
Iteration
Figure 7: Ablation study for state alignment using IsaacGym.
Parameter
Value
learning rate
0.001
discount factor
0.95
GAE-À
0.95
learning epoch
10
policy network
MLP [256, 256]
value network
MLP [256, 256]
clip range
0.2
entropy coefficient
0.002
number of epoch
10
Table PPO hyperparameters.
Parameter
Value
learning rate
0.001
policy network
MLP [256, 256]
gradient decay factor
0.9
Table 4: Hyperparameters for policy training using differentiable simulation.
14


Reviews:
[{"review_id": "st8OVHQXY1", "reviewer": "Reviewer_Mbex", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors formulate a single rigid body dynamics approximation of a legged robot which is differentiable. The reward function is differentiable with respect to the model parameters which lets us calculate the lower variance first order policy gradient. The state is aligned with a non-differentiable simulator to avoid tracking errors.", "summary_of_recommendation": "Well written paper well thought out experiments to show the clear benefits of differentiable simulation for legged locomotion.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. It's interesting that the training time seems to be the same for both the approaches in Table 1. The authors mention \"our results indicate that differentiable simulation can be as time-efficient as PPO in large-scale environments despite the requirement of backpropagation through time (BPTT)—a process known for its computational demands.\" It would be great if the authors could comment this is the case?\n\n2. Was there a specific reason for choosing IsaacSim as the simulator of choice?  \n\n3. \"we utilize short-horizon policy learning to achieve stable gradient back-propagation over a short horizon.\" a short mathematical description of it would be nice. Wouldn't the shorter horizon impact the nature of tasks learnable in this setup?\n\n4. would the ground reaction forces also need to be aligned? \n\n5. Having a model-based RL baseline would have been nice to show the benefits of proposed approach as well as limitations of MBRL.\n\n6. A small ablation study to show the effect of state alignment would be interesting to show how the states drift from the real simulator.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3J8fDyyte", "reviewer": "Reviewer_ersp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed a novel differentiable simulation framework (DiffSim) for quadruped locomotion. To address the inherent non-differentiability of the dynamics, DiffSim decomposed it into two differentiable components. DiffSim also used a high-fidelity, non-differentiable simulator (IssacGym) for forward simulation to correct that of the simplified, differentiable simulation. DiffSim demonstrates significantly better sample efficiency compared to a model-free RL baseline, and transfers to real world zero-shot.", "summary_of_recommendation": "The paper is well-written, the method is interesting, and the experiments are good overall. But, it has a notable weakness of having a single baseline, and not comparing to existing differentiable simulators.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Referring to the review section:\n  \n- Improve the quality of writing of the *Related Work* section \n- Choice of baselines: Compare with existing differentiable simulators. How does the differentiable simulation tailored to quadruped compare with existing general purpose differentiable simulator? How does the quality of the gradients compare?\n\nClarifications: \n\t\n- Can you elaborate on the sources of misalignment between the simplified, differentiable simulator and the high-fidelity, non-differentiable one? My understanding is 1) linearization error from foot Jacobian, and 2) compounding error over time. \n- What does blind policy (line 234) mean? Does it mean the robot does not have visual perception? \n- One rollout consists of 24 time steps. Does back-propagation through time (BPTT) differentiable through all 24 time steps? Or is the computation graph detached at every k step? It would be interesting to ablate what the optimal k is. \n\t\nQuestions: \n\t\n- In Table 1, the performance of DiffSim actually degrades beyond 16 robots. Why is that?\n- I am curious about how much engineering effort is required for such a tailored solution. More importantly, how much additional effort would it take to apply the method to another quadruped, such as Spot or Go1?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u32FgWkCMX", "reviewer": "Reviewer_VLv4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method that uses a differentiable simulator to aid the gradient computation in RL. The differentiable simulator adopts a simpler dynamics model and is iteratively aligned with the full dynamics. The result shows efficient learning of high-performance policies.", "summary_of_recommendation": "Overall a great paper with interesting new ideas. It should be one of the top papers this year.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I have a few detailed questions/comments:\n\n1. Eq.(3) seems to suggest that the desired action space is both the reference joint position/velocity, while the actual action space seems to be position only (from the appendix)? Can you clarify this in the revision?\n\n2. Are gaits / stepping frequencies the *commands* to the policy, or are those figured out by the policy itself? Would the same framework automatically determine the contact sequences/frequencies?\n\n3. How is contact determined in the SRBD model? That is, how does the model know which legs provide GRFs and which legs don't? Is it based on the reference gait, or the simulator's contact detection? How would this contact detection affect the learning process?\n\n4. It would be great to discuss the \"short-horizon policy training\" in detail. How short is the horizon used? How does this horizon affect the training time and policy performance?\n\n5. It seems that you are using vanilla policy gradient (Eq.(4)) for diffsim, but the PPO policy gradient for non-diff-sim baseline. I understand that diffsim provides lower variance gradients, which might make the variance-reduction techniques in PPO redundant. However, would the long chains of back-propagation lead to any vanishing/exploding gradients? In these cases, would gradient-clipping techniques in PPO help in stabilizing training?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a differentiable simulation framework for learning legged locomotion, which tries to overcome the issues of discrete dynamics. The reviewers found that the paper is well written and conveys impressive technical insights, which may have a huge impact on the follow-up works. However, there are several clarification requests from the reviewers- please refer to the individual comments.

The paper has been unanimously supported by reviewers, and the AC also recommends acceptance. Congratulations.
Decision: Accept