Learning Quadruped Locomotion Using Differentiable Simulation

Authors: yunlong  song
sangbae  kim
davide  scaramuzza
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:03:46

Abstract: This work explores the potential of using differentiable simulation for learning robot control. Differentiable simulation promises fast convergence and stable training by computing low-variance first-order gradients using the robot model. Still, so far, its usage for legged robots is limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics.  This work proposes a new differentiable simulation framework to overcome these challenges. The key idea involves decoupling the complex whole-body simulation, which may exhibit discontinuities due to contact into two separate continuous domains. Subsequently, we align the robot state resulting from the simplified model with a more precise, non-differentiable simulator to maintain sufficient simulation accuracy. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains in minutes. We demonstrate that differentiable simulation outperforms a reinforcement 
learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. Our policy achieves robust locomotion performance in the real world zero-shot.

Main paper: 
 introduction: 
 Figure 1 : Graphical model for policy learning using differentiable simulation.Traditional model-free reinforcement learning (RL) often requires extensive parallelization to achieve a stable walking policy for legged robots [1] . How can legged robots master walking quickly with a few trials and errors? Recent progress in legged robot control has been largely driven by combining model-free RL with massively parallelized simulation [1, 2, 3, 4, 5, 6, 7] . Central to these advancements is the policy gradient algorithm [8, 9] , where the gradient of a control policy can be calculated via ∇ θ J(θ) = E τ ∼π θ [R(τ )∇ θ log p(τ ; θ)]; which is a zeroth-order estimate of the true gradient of J based on sampled trajectories following the policy π θ . Here, R(τ ) is the reward of a trajectory τ . Such zerothorder gradient estimate provides a powerful and general framework for robot control since it can handle non-differentiable optimization objectives and discontinuous dynamics. However, the gradient has high variances. Consequently, several additional strategies are required for stable training, such as using a clipped surrogate objective [10] and increasing the sample size.In robotics, leveraging well-established knowledges about robot dynamics can enable the construction of first-order gradient estimates ∇ θ J(θ) = ∇ θ R(τ ), which typically exhibit significantly lower variance than their zeroth-order counterparts and hold great potential for more stable training and faster convergence. Recently, policy training using first-order gradient has been notably advanced through differentiable simulation [11, 12, 13, 14, 15] ; these works have shown promising results in reducing both the number of simulation samples and total training time compared to zeroth-order methods. Differentiable simulation combines the advantages of model-based control and data-driven learning. Similar to numerical optimal control methods, such as shooting techniques, differentiable simulation utilizes robot dynamics to simulate trajectories and iteratively optimize decision variables using analytical gradients derived from the robot model. Like reinforcement learning, differentiable simulation benefits from large-scale simulations and deep learning to train neural network control policies that map observations directly to control actions. Despite its advantages, differentiable simulation faces several challenges, including discontinuous dynamics, complex optimization landscapes, and issues like exploding or vanishing gradients in long-horizon tasks, which can limit the effectiveness of first-order gradient methods. This work investigates the potential of training policies through surrogate models within differentiable simulation, specifically focusing on quadruped locomotion. We show that differentiable simulation offers considerable advantages over model-free RL for policy training in legged locomotion. Notably, we demonstrate that a single robot, without the need for parallelization can quickly learn to walk within minutes in simulation using our approach. Leveraging the advantage of GPUs for parallelized simulation, our robot learns diverse walking skills over challenging terrain in minutes. Specifically, we train a quadruped robot to walk with different gait patterns, including trot, pace, bound, and gallop, and with varying gait frequencies. While model-free RL can also achieve similar performance given sufficient parallelization, our approach requires much less data, achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. More importantly, we show that the policy trained via differentiable simulation can be transferred to the real world directly without fine-tuning. This work presents one of the earliest demonstrations of using differentiable simulation to train control policies for real quadruped robots, highlighting the potential of differentiable simulation for real-world applications.The key to our approach is a novel policy training framework that combines the smooth gradients obtained from a simple surrogate dynamics model for efficient backpropagation with the high fidelity of a more complex, non-differentiable simulator for accurate forward simulation. Instead of simulating the quadruped robot using whole-body dynamics, which by definition has discontinuities due to contact, we propose separating the simulation into its floating base space and its joint space. For the robot body, we employ an approximation using single rigid-body dynamics, which offers a continuous and effective representation of the robot body. To address simulation errors arising from our simplified rigid-body dynamics model, we incorporate a more precise, nondifferentiable simulator. This non-differentiable simulator can simulate complex contact dynamics and is used to align the state in our simplified model, thereby ensuring that our training pipeline remains grounded in realistic dynamics. Figure 2 provides an overview of our system. 
problem formulation: 
 We formulate legged robot control as an optimization problem. The robot is modeled as a discretetime dynamical system, characterized by continuous state and control input spaces, denoted as X and U, respectively. At each time step k, the system state is x k ∈ X , and the corresponding control input is u k ∈ U. An observation o k ∈ O is generated at each time step based on the current state x k through a sensor model h : X → O, such that o k = h(x k ). The system's dynamics are governed by the function f : X × U → X , which describes the time-discretized evolution of the system as x k+1 = f (x k , u k ). At each time step k, the robot receives a cost signal l k = l(x k , u k ), which is a function of the current state x k and the control input u k . The control policy is represented as a Figure 2 : System overview of learning quadruped locomotion using differentiable simulation. Our approach decouples the robot dynamics into two separate spaces: joint and floating base spaces. We leverage the differentiability and smoothness of a single rigid-body dynamics for the robot's main body, which takes the ground reaction force from its legs as the control inputs. We use the state from a non-differentiable simulator (IsaacGym) to align the state in the differentiable simulation. deterministic, differentiable function such as a neural network u k = π θ (o k ). The neural network takes the observation o k as input and outputs the control input u k . The optimization objective is to find the optimal policy parameters θ * by minimizing the total loss via gradient descentEQUATIONEQUATIONwhere α is the learning rate and l(x k , u k ) is the differentiable loss at simulation time step k.We represent the robot's main body using single rigid-body dynamics. Single rigid-body dynamics have been shown to be useful for dynamic locomotion using model predictive control [16, 17] . The single rigid-body model offers a continuous representation of the robot base dynamics, avoiding the complex optimization landscape introduced by contacts. We develop our differentiable simulation using the single rigid-body dynamics, which is expressed as follows,ṗW B = v W B vW B = 1 m i f i + g qW B = 1 2 Λ(ω B ) • q W B ωB = I -1 (η -ω B × (Iω B )) .In this approximation, the state of our system is x = [p, q, v, ω], where p WB ∈ R 3 is the position and v WB ∈ R 3 is the linear velocity of the center of mass in the world frame W , We use a unit quaternion q WB to represent the orientation of the body in the world frame and use ω B to denote the body rates in the body frame B. Here, I is the robot's inertia tensor, η is the body torque, and Λ(ω B ) is a skew-symmetric matrix. The control inputs are the ground reaction force f i from the legs that have contacts.The ground reaction forces are required to simulate the rigid-body dynamics. One option for the control policy design is to output the ground reaction force directly, similar to the MPC design [16, 17] . Another option is to control the robot in the joint space, e.g., output desired joint position, which allows more control authority for the policy and adaptive behavior. Our policy maps observations to the desired joint position q ref , while assuming zero joint velocity, i.e., qref = 0. In this case, the neural network output (joint position) is required to be converted into the control input of the single rigid-body model, which is the ground reaction force. This conversion is achieved using a PD controller for forward propagationEQUATIONwhich calculates the required motor torque τ . Here, k p and k d are fixed gains, q ref and q are the reference joint position and the current joint position separately, and qref and q are the reference joint velocity and the current joint velocity respectively. Subsequently, the motor torques are then converted to ground reaction forces f using the foot Jacobian J: f = (J T ) -1 τ . The continuous nature of PD control enables the backpropagation of policy gradients. Consequently, our simulation framework treats the PD controller as a differentiable layer.In differentiable simulation for policy learning, the backward pass is crucial for computing the analytic gradient of the objective function with respect to the policy parameters. Following [18] , the policy gradient can be expressed as followsEQUATIONwhere the matrix of partial derivatives ∂x j /∂x j-1 is the Jacobian of the dynamical system f . Therefore, we can compute the policy gradient directly by backpropagating through the differentiable physics model and a loss function l k that is differentiable with respect to the system state and control inputs. A graphical model for gradient backpropagation in policy learning using differentiable simulation is given in Figure 1 . Due to the usage of multiplication , there are two potential issues in using Eq. ( 4) for policy gradient: 1) gradient vanishing or exploding, 2) long computation time.We tackle these two problems via short-horizon policy training.Due to the simplification of our single rigid-body dynamics and the decoupling of the simulation space, the robot state can diverge from its actual one. Over time, even minor discrepancies can accumulate, leading to unrealistic states and ultimately causing the failure of policy training. We proposed to align the body state in our differentiable simulation using information from other simulators that use accurate whole-body dynamics. Specifically, we align the robot state in our differentiable simulation with the state information from IsaacGym [19] .We use the following equation to align the robot stateEQUATION). Here, x diff, detach t+1 and x diff t+1 represent the same robot state of our differentiable simulator at time step t + 1; hence, they share the same value. The word detach indicates that xdiff, detach t+1 is detached from the computational graph for automatic differentiation. Therefore, we can reset the robot state using the value from the non-differentiable simulation during forward simulation: xdiff t+1 = x non-diff t+1 + α * 0. During backpropagation, the gradient of the state at a given time t is computed as follows∂x diff t+1 /∂x diff t = 0 + α * ∂x diff t+1 /∂x diff t .Here, 0 < α ≤ 1 is used to decay the gradient. The non-differentiable simulator can simulate complex contact dynamics and is used to align our simplified model, thereby ensuring that our differentiable training pipeline remains grounded in realistic dynamics. At the same time, our approach benefits from the simplified differentiable simulator, which offers smooth gradients for backpropagation. Figure 2 shows the computational graph of the forward propagation and backpropagation using state alignment.Although smoothed physical models (e.g., single rigid-body dynamics) improve the local optimization landscape, the complexity of the optimization problem escalates significantly in long-horizon problems involving extensive concatenation of simulation steps. The situation further deteriorates when the actions within each step are interconnected through a nonlinear and nonconvex neural network control policy. The complexity of the resulting optimization landscape can make gradientbased methods to become trapped in local optima quickly. Instead of directly solving a long-horizon policy training problem, following [13, 1] , where long-horizon simulation tasks are truncated into short-horizon simulation, we utilize short-horizon policy learning to achieve stable gradient backpropagation over a short horizon. Mathematically, short-horizon policy learning involves truncating a long-horizon trajectory of length N into several shorter segments to compute the policy gradient shown in Eq. ( 4). For instance, for a trajectory of length 240, we divide it into ten segments of N = 24. However, it's important to note that while a shorter horizon could simplify computation and facilitate optimization, it may also limit the model's ability to anticipate future events, introducing bias into the gradient calculation. This bias could, in turn, affect the types of tasks the model can effectively learn.A fundamental difference between policy training using differentiable simulation and reinforcement learning is that the loss function has to be differentiable when using differentiable simulation. RL allows the direct optimization of non-differentiable rewards, such as binary rewards 0/1. On the contrary, differentiable simulation requires a smooth differentiable cost function to provide learning signals for the desired control inputs. We formulate a differentiable loss function l(x t , u t ) tailored for velocity tracking, where the main objective is to follow a specified velocity command, denoted as v ref . Additionally, we maintain the robot's body height, represented by p z . To enhance the robustness of this system, we incorporate several regularization terms: one to mitigate a large angular velocity, thus controlling the body rate ω; another to limit the output action u, preventing large control actions; and a term aimed at stabilizing the robot's orientation using projected gravity vector g proj . The loss function is defined asEQUATIONwhere p foot is the foot position.3 Experimental Results Inspired by [20] , where the author studied the connection between reinforcement learning and optimal control using a toy double integrator example, we begin by examining the same toy example: control of a double integrator, which is a fundamental problem in system and control theory. The double integrator is a second-order control system that models simple point mass dynamics in onedimensional space. The state variables are position x and velocity ẋ, with control inputs u = ẍ. The objective is to drive the system state from any initial point to the origin, [x, ẋ] = [0, 0]. Therefore, this problem can be solved by the Linear quadratic regulator (LQR), which can provide an optimal solution. We train a 2-layer multiplayer perception using backpropagation through time via differentiable simulation and the proximal policy optimization (PPO) method [10] . We show that the policy trained using differentiable simulation (DiffSim) achieves nearly optimal performance within a handful of training iterations and samples. In contrast, even when scaling the number of simulation environments to 1024 and training the policy with considerably more iterations, PPO fails to achieve the same level of control performance as LQR or DiffSim. This toy example suggests that scaling might be insufficient for PPO to achieve an optimal solution. This section explores whether a robot can learn to walk without parallelization and with very limited data points at each training iteration. We design a simple velocity-tracking task where the robot is required to follow a constant velocity in the x-axis, e.g., v x = 0.2 m/s. At each training iteration, we simulate 24-time steps with one single robot. Hence, each training iteration contains 24 data points. We trained the policy for a total of 1000 iterations. The result is given by the learning curve in Figure 4 . Despite very limited data, our policy successfully learns to walk after minutes of training. As a comparison, PPO failed to achieve useful locomotion skills under the same condition. By providing first-order gradients with low variance, differentiable simulations offer a fundamental advantage: they allow for more efficient and reliable optimization of the control policy. This is because the gradient provides a clear direction for updating parameters, and the low variance ensures that this direction is consistent and reliable, even when data is limited. 
learning diverse walking skills on challenging terrains: 
 We design a more difficult task: learning diverse locomotion skills over challenging terrains. Specifically, we design four different gait patterns, including trot, pace, bound, and gallop. Additionally, we vary the gait frequency from 1 Hz to 4 Hz. The robot receives a high-level velocity command and is required to track randomly commanded velocity. Figure 5 shows the learning curves using different numbers of robots for policy training. We compare the learning performance of our method with a model-free RL algorithm (PPO) [10] . Given minimal samples, e.g., only four robots, RL has a slow convergence speed and does not learn meaningful walking skills, e.g., the policy constantly falls after a few simulation steps. In contrast, differentiable simulation achieves much higher rewards and can acquire useful walking skills.As the number of robots increases, the performance of both algorithms improves. Notably, the performance improvement for RL is much more significant than our approach. This indicates that the zeroth-order gradient estimates used by reinforcement learning are generally inaccurate and require many more samples to achieve stable training. On the contrary, the first-order gradient estimates used by our differentiable simulation can have very stable and accurate gradients, even given very limited simulation samples. Figure 5 demonstrates diverse walking skills over challenging terrains using a blind policy trained via differentiable simulation.We demonstrate the performance of our policy in the real world using Mini Cheetah [21] . Mini Cheetah is a small and inexpensive, yet powerful and mechanically robust quadruped robot, intended to enable the rapid development of control systems for legged robots. The robot uses custom backdriveable modular actuators, which enable high-bandwidth force control, high force density, and robustness to impacts [21] . The control policy runs at 100 Hz during deployment. Figure 5 shows several snapshots of the robot's walking behavior over different terrains using different gait patterns. We trained a blind policy in simulation using 64 robots and then transferred the policy directly to Additionally, we compare the total training wall-clock time and the final performance of the resulting policies. Differentiable simulation often involves the creation of a complex computational graph to facilitate gradient computation. As the simulation horizon extends, the length of the computational graph can increase proportionally, leading to a substantial increase in the total training time. This increase can diminish its benefits in situations where gathering a large number of samples is both cheap and fast. Therefore, we ask the question: Can differentiable simulation be effectively applied to large-scale settings, for example, thousands of environments? To ensure a fair comparison, we ran both PPO and our training framework for the same number of training iterations, namely 1000 iterations. We compare the total training time, which includes the time for simulating dynamics and updating policies. As shown in Table 1 , our results indicate that differentiable simulation can be as time efficient as PPO in large-scale environments despite the requirement of backpropagation through time (BPTT)-a process known for its computational demands. However, notice that the training time to reach a certain reward is much lower with our method and requires less robot and data. For example, our method requires only roughly 64 robots and 200 training iterations to solve this task, while PPO still performs less after 1000 training iterations with 1024 robots.We use the foot position loss ∥p foot -p ref foot ∥ 2 to provide a learning signal for the swing legs. This loss term is critical for the swing leg since it contains information about the motor position. Following prior works [35, 36] , the swing leg trajectory can be computed by fitting a quadratic polynomial over the lift-off p lift foot , mid-air p air foot , and landing position p land foot of each foot, where the lift-off position is the foot location at the beginning of the swing phase, the landing position p land foot is calculated using the Raibert Heuristics [34] , which is expressed as the following functionp land foot = p hip foot + v CoM T stance /2where T stance is the expected time the foot will spend on the ground, p hip foot is the location on the ground beneath the robot's hip, v CoM is the body velocity projected on the xy-plane. The desired contact state of each leg. e.g., swing or stance, is determined via a gait generator. The gait is modulated by a phase variable ϕ ∈ [0, 2π]. The phase is defined through a dynamic function ϕ t+1 = ϕ t + 2πf ∆t, where f is the stepping frequency. We can design different locomotion patterns by adapting the stepping frequency f and the phase difference between each leg. 
related work: 
 Previous efforts in leveraging differentiable simulation to robot control can be broadly categorized into two classes: improving the simulator itself and enhancing the associated optimization algorithms. The first kind has led to the development of several general-purpose differentiable simulators, including Brax [11] , Nimble [22] , NeuralSim [23] , and Dojo [24] . These simulators enable the integration of robot models directly into policy optimization, allowing neural network control policies to be trained using more mathematically precise analytical gradients. However, as noted in [18, 25] , in scenarios where the underlying system exhibits chaotic behavior-where small perturbations in initial conditions lead to significantly divergent states, such as in robotic simulations involving contact dynamics-these gradients can become unmanageable, leading to divergence in optimization processes. Our work highlights the benefits of employing well-behaved proxy dynamics as an alternative to the true robot dynamics involving contact. Such a design choice has its roots in model predictive control, which often leverages a simplified model for the optimization to control complex robotic systems [16, 17, 26] .Besides developing a better simulator, various methods have been proposed to improve policy training algorithms. For example, [25] proposes an α-order gradient estimator that aims to combine the efficiency of first-order estimates with the robustness of zeroth-order methods. SHAC [13] addresses local minima issues by learning a smooth value function and mitigates vanishing/exploding gradients through truncated backpropagation. Gradient norm clipping strategies have also proven effective in managing the exploding gradients issue [27] . Furthermore, since differentiating through chaotic dynamics is challenging, it is often useful to learn an approximation of these functions and use the approximation for gradient computation. This approach commonly has a connection with model-based reinforcement learning [28, 29, 30, 31, 32] , which involves learning a predictive model of the environment, followed by training a controller. Similar to [13] , our method leverages a truncated backpropagation to facilitate optimization. Our method could be improved further by combining other ideas, such as learning a critic function or clipping the gradients.We highlight one important benefit of RL compared to differentiable simulation: RL can significantly enhance its robustness by directly optimizing through non-differentiable rewards or penalties. Specifically, we use a non-differentiable value p = 200 to penalize the robot when the robot experiences termination during training, e.g., falling on the ground or lifting its legs above its body. r(x t , u t ) = -l(x t , u t ) -p if termination -l(x t , u t ) otherwise.Fig. 6 shows a study of using non-differentiable terminal penalty for both RL and differentiable simulation. The results show that adding a final penalty can greatly affect how well RL works. Without a penalty, RL might get trapped in a local minimum. However, with a large penalty at the end, RL can achieve better task rewards as well as more robust control performance. This is because RL optimizes a discounted return, which estimates "how good" it is to be in a given state. RL uses a state-value function to encode this informationV π (s) = E[G|S 0 = s] = E ∞ t=0 γ t R t+1 |S 0 = s .On the other hand, a terminal penalty has no impact on differentiable simulation since the gradient of a constant value is equal to zero, and we do not leverage a state value function. As a result, differentiable simulation requires well-defined continuous functions, e.g., a potential function or control barrier functions for robust control. 
limitations: 
 A key advantage of model-free RL lies in its ability to directly optimize task-level rewards [33] , especially non-differentiable rewards. In contrast, differential simulation depends on the backpropagation of information from its objective function, making it challenging to explore novel solutions guided by task objectives. Consequently, our system requires specifying foot positions using the Raibert heuristic [34] , as it cannot explore foot placement motion through velocity tracking loss alone. In addition, RL's flexibility allows for more robust performance enhancements, such as implementing simple termination penalties to discourage falling behaviors or employing constant survival rewards to encourage continued operation. Furthermore, differentiable simulations are often tailored to specific optimization tasks and might not generalize well across different problems. It requires specific domain knowledge in both building the simulation pipeline and formulating the problem. To generalize our approach to other tasks, additional engineering efforts are required.We conducted an ablation study to investigate the significance of the proposed state alignment mechanism. In our implementation, we use the state from IsaacGym to align the robot state within our simplified rigid-body differentiable simulation. To assess the impact of state alignment, we performed experiments where we removed the state alignment and compared the resulting learning curves to those obtained with state alignment. The results are presented in Figure 7 . The results indicate that without state alignment, the robot fails to learn any useful walking skills.  
conclusion: 
 This work proposes a new perspective on learning quadruped locomotion policies using differentiable simulation by integrating a simplified rigid-body model with a non-differentiable model for whole-body dynamics. Our key insight is that a neural network control policy can be optimized directly by backpropagating the differentiable loss function through a simplified model while simultaneously aligning the robot's state using a more accurate whole-body dynamics model. Future work should address several limitations of the proposed framework, such as removing the assumption of gait scheduling and the Raibert heuristic for foot contact sequence and foot position planning. Additionally, more sim-to-real analysis is helpful in evaluating the benefits of using a simplified model compared to the true but chaotic gradients or to the policy trained via domain randomization. Finally, we should demonstrate the true advantages of stable training introduced by the first-order gradient by tackling more challenging tasks, such as large-scale training and vision-based control. 
 
 Appendix:  


Decision: The paper presents a differentiable simulation framework for learning legged locomotion, which tries to overcome the issues of discrete dynamics. The reviewers found that the paper is well written and conveys impressive technical insights, which may have a huge impact on the follow-up works. However, there are several clarification requests from the reviewers- please refer to the individual comments.

The paper has been unanimously supported by reviewers, and the AC also recommends acceptance. Congratulations.

Reviews: review: title: title: Calculate the first order policy gradient via differentiable simulation
paper_summary: paper_summary: The authors formulate a single rigid body dynamics approximation of a legged robot which is differentiable. The reward function is differentiable with respect to the model parameters which lets us calculate the lower variance first order policy gradient. The state is aligned with a non-differentiable simulator to avoid tracking errors.
main_review: main_review: Zeroth order Policy Gradients have a high variance which is mitigated by various modifications shown in 'Proximal Policy Optimization'. In this paper, the authors formulate a differentiable formulation of rigid body dynamics which lets one calculate the first order policy gradient which has comparatively lower variance.

Strengths
1. Clear and concise mathematical formulation which lets non-experts (like me) to understand the paper. 
2. Great experiment design of increasing complexity along with hardware trials which show clear benefits of this approach over other approaches. 
3. Clear understanding of the limitations of the approach along with suggestions to mitigate those. 

Weaknesses
1. The training time seems to be the same for both the approaches in Table 1. A better understanding of it would be great.
2. Having a model-based RL baseline would have been nice to show the benefits of proposed approach as well as limitations of MBRL.
3. A small ablation study to show the effect of state alignment would be interesting to show how the states drift from the real simulator.
questions: questions: 1. It's interesting that the training time seems to be the same for both the approaches in Table 1. The authors mention "our results indicate that differentiable simulation can be as time-efficient as PPO in large-scale environments despite the requirement of backpropagation through time (BPTT)—a process known for its computational demands." It would be great if the authors could comment this is the case?

2. Was there a specific reason for choosing IsaacSim as the simulator of choice?  

3. "we utilize short-horizon policy learning to achieve stable gradient back-propagation over a short horizon." a short mathematical description of it would be nice. Wouldn't the shorter horizon impact the nature of tasks learnable in this setup?

4. would the ground reaction forces also need to be aligned? 

5. Having a model-based RL baseline would have been nice to show the benefits of proposed approach as well as limitations of MBRL.

6. A small ablation study to show the effect of state alignment would be interesting to show how the states drift from the real simulator.
limitations: limitations: 3
review_summary: review_summary: Well written paper well thought out experiments to show the clear benefits of differentiable simulation for legged locomotion.
title: title: Well-designed method and good execution
paper_summary: paper_summary: The paper proposed a novel differentiable simulation framework (DiffSim) for quadruped locomotion. To address the inherent non-differentiability of the dynamics, DiffSim decomposed it into two differentiable components. DiffSim also used a high-fidelity, non-differentiable simulator (IssacGym) for forward simulation to correct that of the simplified, differentiable simulation. DiffSim demonstrates significantly better sample efficiency compared to a model-free RL baseline, and transfers to real world zero-shot.
main_review: main_review: **Strength:** 

- Originality: I find the reformulation of dynamics into two differentiable components novel and well-designed. However, due to the poor quality of *Related Work* section, I am not certain of my evaluation here.  
- Clarity: The paper is polished and well-written, except for the *Related Work* section. 
- Quality: The paper is well-executed and the experiments are well-thought-out. I especially like the evaluation of the wall-clock time and appreciate the zero-shot real world transfer. 

**Weakness:**

- The quality of writing for the *Related Work* section is disparate from the rest of the paper. The content is generic and kind of off-the-point.  It should discuss existing works on differentiable simulator. As an example, CALIPSO also demonstrated its use on quadruped gait, albeit in simulation. 

	*Howell, Taylor A., Kevin Tracy, Simon Le Cleac’h, and Zachary Manchester. "CALIPSO: A differentiable solver for trajectory optimization with conic and complementarity constraints." In The International Symposium of Robotics Research, pp. 504-521. Cham: Springer Nature Switzerland, 2022.*
	
	
    It should also cover existing knowledge on the use zeroth vs. first-order policy gradients, such as: 
	

	*Suh, Hyung Ju, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. "Do differentiable simulators give better policy gradients?." In International Conference on Machine Learning, pp. 20668-20696. PMLR, 2022.*

- Baseline: The paper only has one PPO baseline. Aside from model-free RL, the paper should compare its tailored solution to existing general-purpose differentiable simulators. 

Minor comment: While $x$ and $\mathbf{x}$ refer to the same variable, $f$ and $\mathbf{f}$ refer to different variables. The letter $f$ refers to a third thing in Equation 7.
questions: questions: Referring to the review section:
  
- Improve the quality of writing of the *Related Work* section 
- Choice of baselines: Compare with existing differentiable simulators. How does the differentiable simulation tailored to quadruped compare with existing general purpose differentiable simulator? How does the quality of the gradients compare?

Clarifications: 
	
- Can you elaborate on the sources of misalignment between the simplified, differentiable simulator and the high-fidelity, non-differentiable one? My understanding is 1) linearization error from foot Jacobian, and 2) compounding error over time. 
- What does blind policy (line 234) mean? Does it mean the robot does not have visual perception? 
- One rollout consists of 24 time steps. Does back-propagation through time (BPTT) differentiable through all 24 time steps? Or is the computation graph detached at every k step? It would be interesting to ablate what the optimal k is. 
	
Questions: 
	
- In Table 1, the performance of DiffSim actually degrades beyond 16 robots. Why is that?
- I am curious about how much engineering effort is required for such a tailored solution. More importantly, how much additional effort would it take to apply the method to another quadruped, such as Spot or Go1?
limitations: limitations: 2
review_summary: review_summary: The paper is well-written, the method is interesting, and the experiments are good overall. But, it has a notable weakness of having a single baseline, and not comparing to existing differentiable simulators.
title: title: Review
paper_summary: paper_summary: This paper presents a method that uses a differentiable simulator to aid the gradient computation in RL. The differentiable simulator adopts a simpler dynamics model and is iteratively aligned with the full dynamics. The result shows efficient learning of high-performance policies.
main_review: main_review: **Overview**

This paper makes a pioneering attempt in using differentiable simulation (or more broadly, physical priors) to reduce the variance in reinforcement learning. While the proposed method may not fully operate at the scale of non-differentiable RL yet, it provides valuable insight to the field.

**Strengths**

* Effective dynamics model choice

I like the author's choice of the single-rigid-body (SRBD) model as the main model for differentiable simulation. Compared to the full dynamics model, this model is much simpler to implement and compute, while still being highly accurate.

* Learning framework design

The authors carefully designed many key components of the learning framework, such as the iterative alignment between differentiable and full dynamics (Eq.5-6) and the truncated backpropagation. These carefully designed components contribute to the overall effectiveness of the method.

* Clear, insightful results section

The entire paper, and the results section particularly, is very well-organized and clear. I like that the authors start with a double integrator example and compared the result of PPO, DiffSim and LQR. The following sections further demonstrates the framework's performance and addresses important concerns such as training time.

**Weaknesses**

* The proposed simulator may not operate at the scale of "massive end-to-end RL" yet

The proposed learning framework still relies on a few key "signals" to "aid" learning, such as the reference foot position and the gait phases. These guidances might limit the expressiveness of the policy, and prevents the robot from learning more complex behaviors (e.g. using legs to manipulate objects). It would be great if the authors could study some of these in future works.

* More sim-to-real analysis would be desirable

While the authors claim that the diff-sim-trained policy can be transferred directly to the real world, it would be interesting to further analyze on the sim-to-real gap beetween different policies. Would the additional gradient signal reduce, or increase the sim-to-real gap, compared to a PPO policy trained with/without domain randomization? That would be another way to extend the impact of this paper.
questions: questions: I have a few detailed questions/comments:

1. Eq.(3) seems to suggest that the desired action space is both the reference joint position/velocity, while the actual action space seems to be position only (from the appendix)? Can you clarify this in the revision?

2. Are gaits / stepping frequencies the *commands* to the policy, or are those figured out by the policy itself? Would the same framework automatically determine the contact sequences/frequencies?

3. How is contact determined in the SRBD model? That is, how does the model know which legs provide GRFs and which legs don't? Is it based on the reference gait, or the simulator's contact detection? How would this contact detection affect the learning process?

4. It would be great to discuss the "short-horizon policy training" in detail. How short is the horizon used? How does this horizon affect the training time and policy performance?

5. It seems that you are using vanilla policy gradient (Eq.(4)) for diffsim, but the PPO policy gradient for non-diff-sim baseline. I understand that diffsim provides lower variance gradients, which might make the variance-reduction techniques in PPO redundant. However, would the long chains of back-propagation lead to any vanishing/exploding gradients? In these cases, would gradient-clipping techniques in PPO help in stabilizing training?
limitations: limitations: 3
review_summary: review_summary: Overall a great paper with interesting new ideas. It should be one of the top papers this year.