Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models

Authors: jacob  levy
tyler  westenbroek
david  fridovich-keil
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:55:05

Abstract: Traditionally, model-based reinforcement learning (MBRL) methods exploit neural networks as flexible function approximators to represent $\textit{a priori}$ unknown environment dynamics. However, training data are typically scarce in practice, and these black-box models often fail to generalize. Modeling architectures that leverage known physics can substantially reduce the complexity of system-identification, but break down in the face of complex phenomena such as contact. We introduce a novel framework for learning semi-structured dynamics models for contact-rich systems which seamlessly integrates structured first principles modeling techniques with black-box auto-regressive models. Specifically, we develop an ensemble of probabilistic models to estimate external forces, conditioned on historical observations and actions, and integrate these predictions using known Lagrangian dynamics. With this semi-structured approach, we can make accurate long-horizon predictions with substantially less data than prior methods. We leverage this capability and propose Semi-Structured Reinforcement Learning ($\texttt{SSRL}$) a simple model-based learning framework which pushes the sample complexity boundary for real-world learning. We validate our approach on a real-world Unitree Go1 quadruped robot, learning dynamic gaits  -- from scratch -- on both hard and soft surfaces with just a few minutes of real-world data. Video and code are available at: https://sites.google.com/utexas.edu/ssrl

Main paper: 
 introduction: 
 Effective robotic agents must leverage complex interactions between the robot and its environment, which are difficult to model using first principles. Model-based reinforcement learning (MBRL) is a powerful paradigm for controller synthesis [1] , wherein the robot learns a generative dynamics model for the environment. The model can then be used to hallucinate synthetic rollouts [2] , providing a source of data augmentation for policy optimization algorithms [3, 4, 5] . When the model is accurate, it can generate long rollouts which extrapolate beyond the training data, accelerating policy learning substantially. However, in practice, the black-box neural network models favored in the Figure 2 : The SSRL framework. A deterministic policy is used to collect data from the real world while a stochastic policy is utilized in conjunction with the learned dynamics model to "hallucinate" short synthetic rollouts which branch from this data. The model incorporates Lagrangian dynamics and encodes previous state predictions, which are fed to external torque and noise estimators to predict future states. The synthetic data is used with a model-free RL algorithm to update the policies.MBRL literature struggle to generalize beyond the training data [6, 7, 8] , and thus do not outperform modern model-free alternatives [9, 10] . Currently, both paradigms are too inefficient and unreliable to make learning new behaviors in the real world practical for many applications.An appealing alternative is to leverage known physics to design structured model classes. This general approach has been used for efficient system identification and controller synthesis across many bodies of work, ranging from classic adaptive control techniques [11, 12, 13, 14] to more recent physics-informed neural architectures [15, 16, 17] . However, these previous approaches do not scale to the complexities of real-world learning for contact rich-systems such as walking robots (Fig. 1 ). Indeed, modeling contact remains an open problem. Moreover, prior structured approaches to learning contact dynamics make strong assumptions about what can be perceived with available on-board sensors. For example, at inference time [18] requires access to a signed-distance representation of potential contact surfaces. More generally, prior approaches generally assume access to privileged state observations which make it possible to predict exactly when and where contact is made [19, 15] . However, reliably estimating these quantities in the real-world using noisy on-board sensors is another open area of research [20, 21, 22, 23, 24] .We ask: can we build a light-weight modeling framework which leverages known structure and is implementable with onboard sensors? We answer this question in the affirmative by introducing Semi-structured Reinforcement Learning (SSRL), a simple MBRL pipeline for contact-rich control in the real-world (Fig. 2 ). For concreteness, we focus primarily on the quadruped depicted in Fig. 1 and aim to learn an effective locomotion controller entirely from scratch in the real world. We consider the case where the robot's observations only include proprioceptive measurements via joint encoders, IMU measurements, and a global velocity estimator.We inspire our approach by looking to the Lagrangian equation of motion for the robot:EQUATIONwhere q and q are the generalized coordinates and velocities of the robot, motor torques τ are distributed to the joints by the matrix B, F e represents contact forces generated by the environment with J the resulting Jacobian, τ d represents parasitic damping torques, and M , C, and G are the mass matrix, Coriolis and centrifugal forces, and gravity vector. Here, B, M, C, and G are determined by the geometry and inertial properties of the robot, which are known a priori. Thus, given access to observations, we assume that the only unknown terms are J T F e and τ d .We leverage the known Lagrangian structure of (1) by estimating these unknown terms with an estimator τ e (h t ) ≈ J T F e + τ d which is conditioned on a history h t of the available measurements, enabling the model to infer information about hidden states of the environment such as the geometry of the ground underfoot [25, 26] . We instantiate τ e as an ensemble of probabilistic models to help capture uncertainty in the predictions, as is commonplace in the MBRL literature [6, 27] . We then introduce a novel inference procedure for making forward predictions, wherein the predictions made by these semi-structured models are fed back into τ e in an auto-regressive fashion. The SSRL framework leverages these predictions as a source of data augmentation to accelerate real-world policy optimization, using a simple Dyna-style [3, 6] MBRL algorithm. Finally, we demonstrate that this light-weight approach scales to the real-world, learning to walk on different terrains within a minute and learning substantially more dynamic gates within a few minutes. This result outperforms recent state-of-the-art real-world learning results for quadrupeds [28] by both a) requiring an order-ofmagnitude less real-world data and b) reaching substantially higher walking speeds. 
preliminaries and problem formulation: 
 Notation: While the underlying dynamics of the robot evolve in continuous time according to the differential equation ( 1), the policy acts in discrete time and we will use subscripts to denote discrete time steps. The state of the robot s t ∈ S captures the available proprioceptive states q t and their velocities qt , available from joint encoders and IMU measurements. This does not include global coordinates of the robot. We capture the state of the environment (or extrinsics) e t ∈ E, which includes non-proprioceptive information (such as the distance of the ground below the robot) and the state of the ground underfoot (such as terrain deformation). The robot actions a t ∈ A are the outputs of neural network described below. The overall dynamics for the system are defined by: Robot Transitions:EQUATIONTo control this joint system, we will denote the policy the agent optimizes via a t ∼ π θ (•|s t , h t ), where h t = (s t-1 , . . . , s t-h ) bundles the histories of state measurements and θ are parameters.Control Architecture: We adopt the control architecture depicted in Fig. 3 , similar to prior works on RL-based locomotion [25, 29, 30] . We optimize a neural network policy π θ (•|s t , h t ) conditioned on previous observations which outputs (i) changes to parameters of a nominal gait generator that outputs desired foot positions and (ii) offsets to these foot positions. The resulting targets are sent to an inverse kinematics solver which computes desired joint positions. The joint targets are output at 100 Hz to low-level PD controllers for conversion to torques. Additional details are provided in Appendix A.Reinforcement Learning Problem: We formally frame the learning of a locomotion controller in the real world in terms of a partially observable Markov decision process (POMDP) [31] , defined by the tuple (X , A, p, r, Ω, O, γ). Here, X = S × E is the overall state space for the system, and p( ) depends only upon the robot states and actions, and is therefore directly measurable in the real world. We define the reward function to maximize the robot's forward velocity, maintain upright orientation, minimize angular rates, conserve energy, and avoid excessive torques.Finally, we define a termination flag d t ∈ {0, 1} where d t = 1 when body roll or pitch exceed limits. Exact definitions are found in Appendix A. Given an episode length T ∈ N, discount factor γ ∈ (0, 1), and a distribution x 0 over X of initial conditions for the system, the goal is to maximize the expected discounted total reward:max θ E[ T t=0 γ t (1 -d t ) • r(s t , a t , s t+1 )]. 
semi-structured reinforcement learning: 
 A high-level overview of our method is presented in Fig. 2 , outlined as follows: i) we learn an ensemble of deterministic external torque estimators τ e which are conditioned on h t ; ii) we then integrate the torque predictions through the Lagrangian ODE (1), add a learned noise term to generate probabilistic 1-step predictions, and feed this prediction back into h t to produce auto-regressive predictions over multiple steps iii) we fit this semi-structured representation of the dynamics using multi-step prediction losses; and finally iv) predictions from these models are used as a source of data augmentation for MBRL.We construct our approximations to the discrete-time probabilistic robot transition dynamics s t+1 ∼ p s (•|s t , a t , e t ) by building on top of the deterministic Lagrangian dynamics (1) which we rewrite as: M (q)q + C(q, q) + G(q) = Bτ + J(s, e) T F e (s, e, τ ) + τ d (s, e)EQUATIONwhere we now explicitly denote the dependence of the contact Jacobian J(s, e), contact forces F e (s, e, τ ), and dissipative terms τ d (s, e) on the state of the robot, environment, and low-level torques supplied by the motor. As discussed in Section 1, identifying precise locations where contact occurs on the robot can be extremely difficult from on-board measurements. Thus, instead of inferring J and F e separately, we directly estimate τ e = J T F e +τ d . We condition these estimates a latent encoding z t = E ϕ (h t ) of h t , which enables the network to infer information about the extrinsics e t needed for predicting future states [26, 25] . Altogether, we use an ensemble of deterministic models τ e ≈ τ e,i t = T ψi (s t , a t , z t ) with tunable parameters ψ i .We first describe how we generate a probabilistic 1-step prediction for the next state associated to each of the torques estimates τ e ≈ τ e,i t = T ψi (s t , a t , z t ). First, we let τ t = G(s t , a t ) denote a zeroorder hold estimate for the low-level motor torques applied to the robot; here, G is a known map which captures how the current state and action are processed by our control architecture (Fig. 3 ) to produce low-level joint torques. The i-th deterministic next-state prediction is then given by si t+1 = I(s t , τ t , τ e,i t ), which captures how a chosen numerical integrator for the Lagrangian dynamics (1) propagates s t , τ t , and τ e,i t . We then add zero-mean Gaussian noise to si t+1 whose varianceΣ i t = N i ψi (s t , a t , z t ) is a diagonalGaussian which is output by the same network predicting the i-th torque estimate. Altogether, this constructs a probabilistic estimate for the next state:Uncertainty-Aware State Predictions:EQUATIONThe ensemble of probabilistic models {p i ψi } is used to generate k-step predictions using Algorithm 1, which adapts the method from [6] to our auto-regressive setting where future state predictions are conditioned on the history h t . Given a state s t and state history h t , to generate a synthetic rollout we: (i) sample an action from the policy, (ii) randomly choose a model pi ψi from the ensemble, (iii) generate a next-state prediction ŝi t+1 using (4), and iv) incorporate this prediction into h t+1 and repeat the previous steps until a k-step prediction has been generated.To train {p i ψ } i , we maximize the joint-likelihood of state predictions along these synthetic rollouts under the real-world data. Due to the Markov assumption (Section 2), the joint distribution of L(ψ i ) = 1 HN e Ne-H t=h H j=0 si t+1+j -s t+1+j T Σ i t+j -1 si t+1+j -s t+1+j + log det Σ i t+j . (5)Here, N e is the size of a buffer of real-world transitions D env , mean state predictions are propagated deterministically according to si t+1 = S i (s i t , a t , h t ), and each prediction si t is used to update the state history h t . For each state s t in the buffer, (5) generates a synthetic rollout H steps long and computes a loss related to how much the propagated states differ from the experienced states.Remark 1. By optimizing the multi-step loss (5) end-to-end, we i) force the latent z t to encode information about the environment which is necessary for making state predictions over long horizons and ii) average out noisy state estimates and sudden changes in the external torques that occur when the robot makes contact with the ground (Fig. 4 ). This leads to reliable long-horizon predictions which are substantially more accurate than those generated by black-box models (Fig. 5 ).Finally, we introduce the Semi-Structured Reinforcement Learning (SSRL) in Algorithm 2. This policy optimization strategy is a Dyna-style algorithm [3, 6] which leverages model-predictions as a source of data augmentation. Specifically, this approach branches hallucinated rollouts off realworld trajectories to reduce the effects of model bias, and increases the length of these rollouts throughout training as the model becomes more accurate. We perform many rounds of data generation and policy optimization after collecting real-world samples, and as depicted in Fig. 5 our semi-structured model are able to make substantially more accurate predictions over long horizons when data is scarce. Together, these features enable SSRL to push the sample complexity limits for MBRL. We also found the following necessary for making real-world learning practical: Deterministic Real-World Rollouts and Random Hallucinations. Steps in the real environment are taken using a deterministic policy µ θ which simply outputs the mean action from the stochastic policy π θ . Black-box MBRL approaches typically need to inject random 'dithering' noise to ensure their is enough exploration to obtain accurate models. However, we found that this led to unacceptably erratic behavior in the real-world when paired with our policy class. Recent modelfree approaches for learning locomotion in the real-world [32] severely restrict the action space to make stochastic exploration tractable, and thus severely limit the performance of resulting policies. However, our semi-structured models have the right amount of inductive bias to accurately identify the dynamics when the real-world data set D env is generated with deterministic exploration and data is scarce. We do, however, use a stochastic policy π θ to generate rich, diverse synthetic data sets D model for policy optimization. Specifically, policy parameters θ are trained with soft-actor critic (SAC) [33] , using a mixture of transitions from real-world and hallucination buffers D env and D model . Take N E steps in the environment deterministically with µ θ and add transitions to D env 4:Train models pψi on D env using loss (5) 5:Increase hallucination rollout length k according to predetermined schedule 6:for K hallucination updates do Perform G updates of policy π θ using mixture of D env and D model at ratio r D 4 Experimental Results 
real-world results: 
 We demonstrate our approach through two real-world experiments where a Unitree Go1 quadruped is trained from scratch to achieve maximum speed on both hard ground and memory foam.Training is performed from scratch in the real world over N epochs = 18 epochs with N E = 1000 environment steps per epoch, totaling 18, 000 steps or 3.0 min of realworld interaction. We use an observation history length of h = 5, a multi-step loss horizon of H = 4, and hallucinate synthetic rollouts for up to k = 20 steps; see Appendix D for hyperparameters. To enhance plasticity, the model, actor, and critic are reset at 10, 000 steps [34] . Joint states are measured via encoders, IMU data provides body orientation and angular velocity, and linear velocity is acquired with a Vicon motion capture system. Neural networks are trained in JAX [35] , and low-level joint angle commands are sent via Unitree's ROS interface [36] . We compute the mass matrix, Coriolis terms, and gravity vector in (3) using the differentiable simulator Brax [37] , enabling gradient-based training of the semi-structured dynamics model with (5) .Results. Fig. 1 shows a time-lapse of rollouts as training progresses. After just 3.0 min of real-world data, the quadruped achieves an average velocity of 0.98 m s -1 on hard ground and 0.53 m s -1 on memory foam (Fig. 4 , right). Figure 4 (left) plots the reward per episode until the first termination (d t = 1). Initially, the quadruped often falls, resulting in low rewards, but after 1.5 min, the learned policy becomes robust, and rewards increase. Despite the challenge of walking on memory foam, where the robot's feet sink deeply, forward velocity improves consistently in both terrains, demonstrating our approach's versatility to differing contact dynamics. To examine model accuracy, we compare the learned external force predictions τ e,i t to real-world external force estimates τ e over one second. We estimate the real-world external force with τ e = M (q t ) qt + C(q t , qt ) + G(q t ) -Bτ t where joint accelerations are estimated by finite differencing: qt = ( qt+1 -qt )/∆t and motor torques τ t are estimated with the PD control law. Figure 4 (right) shows the learned external vertical force predicted on the base of the robot. Notably, the predictions appear as smoothed versions of the actual torque estimates. This, when combined with the accuracy of our predictions over long-horizons (Section 4.2) provides insight into why our approach enables such effective policy optimization [38] . Appendix B.1 provides plots for the force estimated on additional degrees of freedom for the robot.In addition to the results presented here, we provide extensive ablations on standard RL benchmarks in Appendix C. Here, we investigate the following hypotheses: Hypothesis 1. SSRL boosts performance and sample efficiency for MBRL in contact-rich settings. Center-With our approach, the quadruped steadily learns to walk faster. Right-Predicted and real external vertical force acting on the robot base over one second of real-world data. Real forces are estimated by finite differences. The predictions add noticeable smoothing to the real-world data.Experimental setup. The simulation setup mirrors the real-world setup (Section 4.1), except rollouts are simulated in Brax [37] ; parameters are detailed in Appendix D. To examine Hypothesis 1, we perform policy optimization with SSRL (Algorithm 2) and compare its policy performance to a baseline approach that utilizes the same optimization process but substitutes the semi-structured dynamics model with a black-box model. We also benchmark against SAC [33] , with an extended run shown in Appendix B.6. To test Hypothesis 2, we repeat the prior experiment, except we train the model with a single-step loss horizon (H = 1). To assess generalization (Hypothesis 3), we train our semi-structured models and the black-box models from scratch over 3 minutes of saved simulated data using 1and 4-step losses. We then generate new data using a stochastic policy where we lower the friction and ground contact stiffness by 25% in the simulator. We generate 20-step synthetic rollouts from 400 randomly-sampled starting states within the new dataset and average the prediction error ∥ŝ t -s t ∥/dim(s t ) over the 400 rollouts. All runs are repeated across 4 random seeds.Results. Shown in Fig. 5 (left), our semi-structured dynamics models outperform black-box models in both sample efficiency and maximum reward, supporting Hypothesis 1. Even when data is scarce, our semi-structured model generates more accurate synthetic rollouts during exploration, resulting in significantly improved policy performance. We also observe in Fig. 5 (left) that, while black-box models yield similar policy performance for both single-and multi-step losses, training our semistructured dynamics models with a multi-step loss results in improved performance over a singlestep loss, confirming Hypothesis 2. In Fig. 5 (right), we see that our semi-structured models produce predictions 20 steps into the future that are significantly more accurate than black-box models. By leveraging physics-based knowledge, our models generate synthetic rollouts that generalize better to an unseen environment, confirming Hypothesis 3; additional experiments are found in Appendix B. Right-Prediction error for 20-step synthetic rollouts in an unseen environment showcases our method's superior ability to generalize.6WHSV 5HZDUG 3ROLF\3HUIRUPDQFH 2XUVVWHS 2XUVVWHS %ODFNER[VWHS %ODFNER[VWHS 6$& 6WHSV 3UHGLFWLRQ(UURU +DOOXFLQDWLRQ$FFXUDF\ 2XUVVWHS %ODFNER[VWHS 
related work: 
 Model-based RL. Our works builds on a wealth of prior works that use general function approximators and probabilistic modeling for black-box modeling [39, 6, 27] . Model-based reinforcement learning algorithms either learn a model that is used for online planning [40, 27, 41] or Dyna-style algorithms which hallucinate imagined rollouts for direct policy optimization [6, 3, 4] . Moreover, significant attention been devoted to extending these strategies to learn latent representations for environments from available observations [2, 42] . However, these approaches do not leverage known structure to accelerate learning. In contrast, there has been a substantial line of work which leverages known structure to efficiently learn accurate models [15, 16, 17, 19, 15] . However, these approaches generally require access to the full state of the system at inference time, which is impractical for realworld learning for contact-rich with on-board perception. Thus, our semi-structured approach brings the strengths of both paradigms to bear in a single framework. Finally, we also note related work [43, 44, 38] which investigates smoothing out contact dynamics to make policy optimization more tractable for difficult contact-rich problems. We found that our learned models naturally learned smooth representations for the dynamics which generate accurate long-horizon predictions (Fig. 5 ).Model-free RL. A parallel line of work [9, 10, 28] aims to make off-policy model-free algorithms (which form the back-bone for our policy optimization strategy) more stable and efficient in lowdata regimes. These approaches introduce regularization techniques which enable the use of higher update-to-data ratios without overfitting to the available data, matching the efficiency of modelfree methods such as the MBPO [6] algorithm that we build upon. These algorithmic advances are generally orthogonal to our contribution, and thus in the future we plan to incorporate them into our framework to further accelerate real-world learning.Learning Locomotion Strategies in the Real World. Learning locomotion behaviors from scratch directly in the real-world has primarily been studied in the context of model-free reinforcement learning [45, 46, 32, 47] , with a few works using black-box models in the context of model-based reinforcement learning [48, 49] . Compared to these works, our semi-structured modeling approach enable the robot to achieve more dynamic locomotion strategies than these previous approaches, with just a fraction of the real-world samples. Specifically, our approach either achieves a significantly higher walking speed than each of these approaches, or improves on their sample complexity by approximately an order of magnitude (Fig. 4 ). Several other works investigate fine-tuning locomotion controllers trained in simulation to reduce the burden on real-world data [28, 50] -as we discuss below, we hope to investigate this direction in the near future.Direct Transfer From Simulation. There has also been recent and rapid progress directly transferring locomotion controllers from simulation zero-shot [51, 26, 25, 52, 53] , using techniques such as domain adaptation and domain randomization. In this paper we have focused on learning locomotion controllers from scratch, in an effort to demonstrate the ability of our framework to substantially adapt the behavior of the robot with small amounts of real-world data. However, in the future we plan to fine-tune policies that have been trained using extensive simulated experience, improving the performance of these policies in cases where they fail [28] but leveraging a better initialization for the policy for real-world learning. 
limitations: 
 This paper presents a novel framework for model-based reinforcement learning, which leverages physics-informed, semi-structured dynamics models to enable highly sample-efficient policy learning in the real world. However there are several key limitations. First, our method requires observability of enough proprioceptive states to propagate the Lagrangian dynamics of the robot. Additionally, relying solely on proprioception restricts the model's ability to predict changes to the environment such as the appearance of an obstacle or transitions between different ground surfaces. In the future we plan to extend the current framework to include additional perceptual modalities which can infer more about the state of the environment around the robot.In this appendix, we provide details of our implementation for the Unitree Go1 Quadruped, including the observation and action spaces, the reward function, the termination condition, and the control architecture.The observation space Ω ⊂ R 36 consists of the elements in Table 1 . The x-axis of the base is the forward direction, the y-axis is the leftward direction, and the z-axis is the upward direction. The phase variable ϕ ∈ [0, 2π) represents progression along the gait cycle and is defined as ϕ t = 2πt/T ϕ mod (2π) where T ϕ = 0.5 sec is the gait cycle period.Symbol Dimension Quaternion orientation of the base φ 4 Joint angles q j 12 Base linear velocity (local frame)(v x , v y , v z ) 3 Base angular velocity (local frame) (ω x , ω y , ω z ) 3 Joint speeds qj 12 Cosine of phase cos ϕ 1 Sine of phase sin ϕ 1 Table 1: Observation space.The action space A ⊂ R 9 outputs the change in nominal height for the gait generator and offsets to nominal foot positions from the gait generator, as defined in Table 2 Reward Function. The reward function is a weighted sum of the terms in Table 3 . We set the weights and use exponentials in most of the terms to normalize the reward such that a forward velocity of 1.0 m s -1 with maximal values for all other terms will result in a reward of approximately 1.0 for a single time step. The roll φ x , pitch φ y , and yaw φ z of the base are obtained from the base quaternion φ. We define a ⊙ b as the element-wise multiplication of vectors a and b. Actual torques output from the joint-level PD controllers are not available; we estimate the torque applied at the joint with (11) . We define the following LinearLimit function which linearly penalizes the torque applied at the j-th joint τ j when exceeding torque limits; within torque limits, the function is a decaying exponential:EQUATIONIn this appendix, we present the results of additional experiments which demonstrate our model's ability to make accurate predictions far into the future, generalized to unseen training data, and enable efficient training.Contact forces are discontinuous in nature and can be difficult to learn. To this end, we validate the learned external torque predictions τ e,i t by comparing them to estimates of the real-world external torques τ e over one second of real-world data, presented in Fig. 6 . Estimates of the real-world external torque are computed as τ e = M (q t ) qt + C(q t , qt ) + G(q t ) -Bτ t where joint accelerations are estimated by finite differencing: qt = ( qt+1 -qt )/∆t and low-level motor torques τ t are estimated with the PD control law. The external forces acting on the floating base and the external torques acting on the joints of the front-right leg are shown. Our learned external torque predictions closely align with the estimated actual external torques. Notably, the predictions appear as smoothed versions of the actual torque estimates. Finite differencing of the velocity measurement introduces noise into the actual torque estimates while our multi-step loss learning process inherently smooths the predictions. Here, we examine the accuracy of trajectories generated from learned models and the ability of the model to generalize beyond the available training data. First, we train our semi-structured models and the black-box models from scratch over 3 minutes of saved simulated data using 1and 4step losses. Using the trained models, 20-step synthetic rollouts are generated from 400 randomlysampled starting states within the data. The average prediction error ∥ŝ t -s t ∥/dim(s t ) at each time step t is averaged over the 400 trajectories. This is repeated for 4 random seeds where the best results from the 1or 4-step losses for each model type are recorded. We then repeat this experiment for the real-world dataset. Finally, the experiment is repeated again, except we generate a new simulated dataset using a stochastic policy and on altered terrain: friction and ground contact stiffness are lowered by 25%. For this final case, the saved models from the first case are evaluated on this new dataset. The results for these 3 experiments are presented in Fig. 7 . For all cases, our semi-structured models produce predictions 20 steps into the future that are significantly more accurate than black-box models. The results from the last experiment demonstrate our model's superior ability to generalize to an unseen environment where noise is added to actions. In this experiment, we demonstrate that SSRL performance is robust against errors in a priori knowledge of the robot's inertial properties which are used to construct the Lagrangian dynamics (1). To simulate these modeling errors, we randomly vary each link's mass by ±25% and each joint's damping by ±50% for the Go1 environment used for simulated data collection. The remainder of the setup is per Section 4.2 and runs are repeated across 4 random seeds. The results are presented in Fig. 8 ; even when there are errors in the a priori knowledge of the robot's inertial properties, policy performance is similar to runs with no errors. Our external torque estimators (Section 3.1) learn to predict forces to compensate for these errors, resulting in the similar performance.Figure 8 : Our approach is robust to errors in a priori knowledge of the robot's inertial properties.Here, we examine the benefit of using the noise estimator and an ensemble of models. The noise estimator prevents overfitting to noise and using an ensemble of models captures epistemic uncertainty [27] . We plot the training performance over 4 random seeds for 3 test cases: (i) using the probabilistic ensemble described in Section 3.1, (ii) removing the noise estimators, and (iii) removing the noise estimators and the ensemble. The results are presented in Fig. 9 ; the highest performance is obtained with our probabilistic ensemble. To further demonstrate the versatility of our approach on varying contact surfaces, we perform additional experiments in simulation. Within the simulator, we vary the friction coefficient and contact time constant which determines the stiffness of contact. We perform additional training runs for friction coefficients of 0.3 and 0.8, and contact time constants of 0.06 and 0.12; the friction and time constant used for all other simulated experiments in this paper are 0.6 and 0.02 respectively. As depicted in Fig. 10 , we find that our method still yields favorable performance even under varying simulated contact conditions. State-of-the-art real-world quadrupedal locomotion results where policies are trained from scratch [32] utilized SAC-like agents which directly take actions in the environment. However, highly restricted action spaces were required to obtain stable training behavior with 20 minutes of training data. In contrast, we use action spaces which are more in line with standard quadruped results [25, 29, 30] . This enables faster and more dynamic gaits, but also leads to a challenging optimization problem for SAC. Here, we repeat the experiment of Section 4.2, but we allow the SAC training to run for many more samples and present the results in Fig. 11 . Even though SAC now converges, it takes 250 times more interaction with the environment than our method. To demonstrate the versatility of our approach, we perform additional simulated experiments using standard, contact-rich, benchmark environments [54] commonly used to evaluate reinforcement learning (RL) algorithms.Experimental setup. We use the standard MuJoCo [54] environments Hopper, Walker2d, and Ant, which have been implemented as part of Brax [37] . Similar to the quadruped, each of these environments feature a floating-base robot with articulated limbs which make and break contact with the ground to produce motion. However, unlike the Go1 environment, these environments lack structured controllers. Instead, the outputs from the policy are only scaled linearly before being directly applied as torques on the joints. To test Hypothesis 1 and Hypothesis 2, we compare our semi-structured approach trained with a multi-step loss (H = 4) to the black-box approach from Section 4.2 trained with the single-step loss (H = 1). In both of these cases, the agent acts deterministically within the environment per Algorithm 2. We also benchmark against SAC [33] , allowing the agent to act stochastically in the environment for this algorithm only. The hyperparameters used for training are found in Appendix D and all runs are repeated for 4 random seeds.Results. The results of these experiments are presented in Fig. 12 . We observe a significant performance improvement when utilizing our semi-structured models trained with a multi-step loss, compared to the black-box approach trained with a single-step loss, confirming Hypothesis 1 and Hypothesis 2. These results demonstrate that our approach works not only with the Go1 environment, but also with other contact-rich environments with unstructured controllers.  
 
 Appendix:  


Decision: **Post-rebuttal metareview**:

This paper proposes an MBRL approach for locomotion, where a residual dynamics model is learned on top of Lagrangian dynamics, and a SAC policy is trained using the learned dynamics. This method is quite novel in the field of locomotion, and justified by hardware experiments. Applying the proposed techniques in more challenging tasks (e.g., complex terrain) and showing its advantage over model-free approaches will further strengthen the paper.

----------------
**Pre-rebuttal metareview**:

Strengths:
1. The idea of using physics prior knowledge for efficient MBRL in locomotion is interesting and novel. 
2. Convincing experimental results to prove the concept.

Weakness:
1. The presentation needs improvements.
2. Some claims are not accurate such as “the external contact force is the only uncertainty”.
3. Although relatively new in locomotion, the key idea is not new in robotics.
4. Need more challenging experiment scenarios such as rough terrains. 
5. Other issues mentioned by reviewers, such as generalization, definition of “3-min data”, etc.

Reviews: review: title: title: Cool paper making a point for using mechanistic structural knowledge to improve sample efficiency
paper_summary: paper_summary: The authors train a policy that has only excess to a series of past observations of joint encoders and IMU measurements. The policy outputs the parameters of repetitive motion generator and offsets to nominal foot position.  This motion generator then outputs foot positions which via an IK solver are mapped to reference joint positions that are tracked by a PID. During training the policy is used to collect data which is then used to train an ensemble of probabilistic dynamics models that is supposed to predict the external forces (in particular contact) acting on the robot. These external force predictions (being in generalized coordinate space and being samples of the model) are then inserted into the robot's Euler-Lagrange equations to hallucinate possible motion. On this hallucinated data, the policy is trained via SAC.
main_review: main_review: Currently, with the many recent papers coming out on quadrupedal locomotion, one could think that this problem is solved. However, the data-efficiency of these algorithms is still quite bad which this paper argues can be solved using mechanistic models such as the Lagrange equations. The algorithm proposed by the authors is rather simple which I perceive as strength.

**Strengths:**
- Convincing experimental results.
- Relatively simple model architecture.
- Hardware experiments on quadruped robot with two different ground materials.

**Comments:**
- **C1 - Verbose writing:** The paper reads as if the authors just started writing section after section without thinking in advance how to most effectively communicate its ideas to the reader. After all, the authors just propose to model dynamics by plugging an ensemble model into the joint torques of the Euler-Lagrange equations and adding a second head to the assemble model that adds noise to the models output. This model is than used in MBPO to hallucinate rollouts from the real-world data onto which a modified SAC is being trained. Why not first give a systematic high-level overview and then one by one explain the algorithms components?

- **C2 - Errors in Lagrange equations:** In line 56, the authors write *"$J^T F_e$ is the only unknown term in the dynamics"*. If $\tau$ just corresponds to the motor torques (typically estimated by multiplying motor currents with the motor constant), then the **dissipative torques in the robot's joints remain unmodeled**. Dissipative torques can be quite a nuisance for forward model predictions which is why "Learning agile and dynamic motor skills for legged robots" by Hwangbo et al deploys joint torque sensors in a  pre-identification step to solely identify joint torques. As the authors learn "external forces" in generalized coordinate space, the **model also learns dissipative torques** along side the end-effector torques.

- **C3 - Please share your code and data**

**Minor comments:**
 - Line 54: The force $C$ contains not only Coriolis terms, **but also centrifugal forces**. Featherstone refers to "C" in "Rigid body dynamics algorithms" as "bias term". I like this terminology, as if no external forces act on the system, then $C$ is the force one needs to apply to cause zero acceleration.
 - I would avoid refrain from using *"sophisticated"* words such as "uncertainty-aware" or "proprioceptive" if it is not needed. Just as an example, the authors use the word **"proprioceptive"** over seven times in the paper, why?? You have established in the introduction that the states are reprioceptive, why always add a word that does not help in communicating your actual ideas?
- Equation 4 is actually a table, please adjust the formatting. I am doubtful that this table serves the reader as adding the names of these model components to Figure 2 would already do the trick while providing also information on the model's graph.
- In Figure 2, it looks like the output of the ensemble model and the Lagrange equations are being added, but if I understood correctly you plug the output of the ensemble model **into the Lagrange equations**.
 - Line 77, typo "that"
questions: questions: - **Question 1:** When determining policy parameters θ with SAC, how do you choose how many transitions to pick from Denv and Dmodel?
- **Question 2:** How do you plan to extend this method to non-flat surface geometries and surfaces covered my movable objects in particular foliage?
- **Question 3:** Why do you need an encoder prior to the ensemble model? Can you add an ablation study that investigates how removing the encoder/noise network/... affects training?
- **Question 4 - Learning contact forces:** Due to the discontinuous nature of contact forces, learning these forces is extremely hard. So, I am surprised that the authors got anything tangible out of the ensemble. Clearly, the ensemble model is learning the contact force over one time step which causes some smoothing. Also the encoder might discover directions in generalized coordinate space in which your dynamics remain invariant to discontinuities (null space of $J^T$, see e.g. "Contact invariant model learning for legged robot locomotion" by Grandia et al). **Can you add another plot that shows the collected real-world data minus the Lagrange model without external torques, and then compare to this data to the predicted external joint torque that has been predicted by the ensemble?** I would be interested in seeing the absolute error of the ensemble errors on the whole data set and also seeing rollouts of the signal over time.
limitations: limitations: 2
review_summary: review_summary: Solving a difficult problem with few means is a significant feat. If the authors work on the structure of this paper, its writing style, add few additional experiments, and extend the critical discussion, than this work is **top notch**.
title: title: A model-based RL framework that integrates a first-principle model with a learned residual dynamics model to enhance quadruped robot locomotion.
paper_summary: paper_summary: This paper presents a model-based reinforcement learning framework that integrates a first-principle model with a learned residual dynamics model to enhance quadruped robot locomotion. For floating-base systems like quadruped robots, controlling the un-actuated base coordinates, such as the robot's body, requires external forces. Instead of relying on traditional methods like the soft contact model or kinematic constraints, this paper introduces an approach to learn the external forces based on the observed states of the robots. The model learning process utilizes a neural network to predict the external torques applied to the robot by its environment. The neural network is trained using approximate maximum likelihood estimation. For policy training, the authors employ the Soft Actor-Critic algorithm, utilizing both real-world and simulated data to optimize the policy. The paper demonstrates the proposed approach through real-world results and simulated experiments focused on the task of tracking a forward velocity. Real-world results indicate that the proposed framework can be used to train a neural network control policy for simple quadruped locomotion. The simulation experiments show that the hybrid robot model has better simulation results than a black-box model.
main_review: main_review: 1. The paper is very well written, with clear motivation and structure. It is easy to follow and understand. 

2. My main concern is that the proposed framework lacks novelty on the algorithmic side and fails to demonstrate impressive real-world performance. On the model learning side, it is very common to combine a first-principle model with residual model learning. Conceptually, both the model learning and the policy learning are not new. On the demonstration side, I expect the proposed framework to be demonstrated on more complex terrains. The real-world experiments only show results on simple flat ground or memory foam. What about more complex terrains or slippery ground?  

3. The title “Learning to Walk from Three Minutes of Data with Semi-structured Dynamics Models” can be misleading. The three minutes of data is used to learn the residual model, not the walking policy. Policy learning requires both simulation data and real-world data. Therefore, this framework requires significantly more data for learning to walk. 

4. Generalization is a key advantage of leveraging a prior model. I suggest that the authors focus on demonstrating how the proposed framework could be generalized to different environments. For example, is it possible to train on certain types of terrain and then generalize the policy/model to unseen terrains? To my understanding, both the policy and the model are overfitting the training environment.
questions: questions: 1. What is the key novelty of the proposed framework?

2. Can the proposed framework be applied to more difficult locomotion tasks?

3. How much data is actually required to train the walking policy? (both simulation and real-world data)

4. Is it possible to train on certain types of terrain and then generalize the policy/model to unseen terrains?
limitations: limitations: 2
review_summary: review_summary: The contribution of this work is minor. Weak reject.
title: title: Impressive hardware results and a strong direction for learning in the real world.
paper_summary: paper_summary: The paper proposes a model based reinforcement learning technique that structures the model learning problem as Lagrangian dynamics and learns just the external forces term as opposed to the whole state transition function. By exploiting the prior knowledge of the model through this structure, along with other techniques like learning an ensemble of models and using a multi-step prediction loss, allows the authors to learn locomotion skills on a quadruped from just 3 minutes of real world data. Hardware results are presented with the robot walking both hard and soft ground, as well as simulation ablation studies showing the usefulness of the semi-structured dynamics model and the multi-step loss.
main_review: main_review: Overall I think the work is good and presents a clear idea forward that can be built upon. Though there have been other learning in the real world and model-based RL for quadruped locomotion before (which the authors have appropriately referenced), I have not seen it structured through Lagrangian dynamics as it is here before. While I am not fully convinced of the full usefulness of this method yet, i.e. I am not convinced enough that I would use this method myself to train the locomotion policies I want (see below for more details), this is definitely a step in the right and interesting direction, and I am definitely interested in extension ideas that stated in the limitations (adding in external perception) and appendix (fine tuning simulation trained policies in the real world).

## Strengths:

This paper tackles an important topic highly relevant to the field: how to best incorporate real world data into the learning process and how to learn from a small amount of real world interactions. Solving this would help many a sim2real problems that I’m sure a lot of researchers face currently.

The work is overall well written and clear. The method is presented well and is easily understandable; after reading I feel like I understand the main idea enough to know how I would implement/use this myself. The theoretical side of things seem to be sound to my knowledge as well.

Ablation studies are included to justify certain design choices as well when it comes to the resulting policy performance.

The biggest strength by far are the impressive hardware results. Using only 3 minutes of hardware is highly impressive and is a real world sample efficiency record for quadruped locomotion to the best of my knowledge. This along with the relative soundness of the method makes this an accept for me regardless of the limitations described below.  

## Limitations:

The simpleness of the quadruped locomotion test case used here weakens how convinced I am of the actual usefulness of the presented method. I understand that this is just a test case to show your proposed method but it still stands that nowadays quadruped locomotion is mostly “solved” and is sort of a given capability. Nowadays our simulation models are good and fast enough that I’m sure in the wall clock time spent to learn the dynamics model and policy in your method you can just learn a policy in simulation and zero-shot transfer to the real hardware. Or even if sim2real is an issue, adaptation methods like RMA seem to be able to fix things with small amounts of real world data as well.
I am much more interested in seeing how your method would scale up to harder and more complex tasks/platforms like non statically stable bipeds, where the dynamics might be harder to model, learning on hardware is more complicated, and highly structured policy action spaces like footstep gait controllers are harder to engineer (which I think is highly important for both the sample complexity and feasibility of taking exploratory actions on hardware done in this work).

The main novelty here is relatively limited, and seems to be just the structuring of the model learning as Lagrangian dynamics. Everything else, as the authors already reference too, has been done before. Non black box model based RL has also sorta already been done before, see SimGan (Jiang et. al, ICRA 2021), where they learn modification to simulation parameters rather than a whole black box (though they admittedly use the learned model for policy refinement rather than learning from scratch).

A major part of this work is learning the semi-structured dynamics model, but there is not evaluation of the model’s performance itself, only how it affects policy performance. Some evaluation of how accurate the learned model is (or average of the ensemble of learned models) in your simulated results would be very good to add I think. Along with some model accuracy comparisons with the black-box approach and the single step loss, especially since you state that the goal is to do auto-regressive predictions “far into the future” and that your method can “produce rollouts that generalize far beyond the available training data”. There are currently no evaluations that attempt to show this, only evaluations of policy performance. 

## Formatting comments:

Line 65/66: “but is required” is repeated in the sentence.

Line 159: “... a predictive model for the howe the environment…”

Line 245: “environment” misspelled as “nvironment”

Figure 5 caption: “semi-structed dynamics models”, “structured” is misspelled
questions: questions: What’s the rationale for structuring things as Lagrangian dynamics and learning the external force component as opposed to just learning some black box modification to the simulator output, whether it be to simulator parameters or just a straight up addition to the simulator state? To me it seems like the simulator’s estimate of the next state would be better than Lagrangian dynamics (simulator’s output is probably closer to the real world next state than the Lagrangian’s estimate without the external force component, so you have to learn a less difficult problem) and would probably actually be my first thought of what to do rather than this Lagrangian setup. 

As the authors point out the method requires being able to differentiate through the mass matrix and Coriolis terms in order to optimize the probabilistic dynamics model. This work gets around this by using Brax, a differentiable physics simulator, but this requirement may be a sticking point for some setups which do not have this capability. How accurate of a gradient is required of this? Would something like finite differencing through a non-differentiable simulator work? Do you have any intuition on how this affects the accuracy of the learned dynamics model, and how does the accuracy of the dynamics model affect the learned policy?

For the simulation experiments, why does SAC not learn? I would expect it to still learn, just more slowly, but it seems like the learning actually collapses? For the other black box and 1-step loss comparisons I can understand it failing since a badly learned model can cause the policy learning to go bad, but SAC is taking actions directly in the environment. For example, as you reference (ref. 45) other works can learn to walk in the real world with SAC with 30000 samples. Why does your SAC baseline seem to underperform?
limitations: limitations: 3
review_summary: review_summary: Relatively incremental work when it comes to novelty, but I think it is working towards a good direction to explore in for doing learning from limited real world data. The results can be a benchmark for real world sample efficiency, showing successful locomotion learning from the smallest amount of real world data I have seen yet.