Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models
Authors: Jacob Levy
Tyler Westenbroek
David Fridovich-Keil
Venue: CoRL 2024 (2024)
Keywords: Model-Based Reinforcement Learning
Physics-Based Models
TL;DR: This paper presents a novel framework for model-based reinforcement learning, which leverages physics-informed, semi-structured dynamics models to enable highly sample-efficient policy learning in the real world.

Abstract: Traditionally, model-based reinforcement learning (MBRL) methods exploit neural networks as flexible function approximators to represent $\textit{a priori}$ unknown environment dynamics. However, training data are typically scarce in practice, and these black-box models often fail to generalize. Modeling architectures that leverage known physics can substantially reduce the complexity of system-identification, but break down in the face of complex phenomena such as contact. We introduce a novel framework for learning semi-structured dynamics models for contact-rich systems which seamlessly integrates structured first principles modeling techniques with black-box auto-regressive models. Specifically, we develop an ensemble of probabilistic models to estimate external forces, conditioned on historical observations and actions, and integrate these predictions using known Lagrangian dynamics. With this semi-structured approach, we can make accurate long-horizon predictions with substantially less data than prior methods. We leverage this capability and propose Semi-Structured Reinforcement Learning ($\texttt{SSRL}$) a simple model-based learning framework which pushes the sample complexity boundary for real-world learning. We validate our approach on a real-world Unitree Go1 quadruped robot, learning dynamic gaits  -- from scratch -- on both hard and soft surfaces with just a few minutes of real-world data. Video and code are available at: https://sites.google.com/utexas.edu/ssrl

Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models
Jacob Levy" University of Texas Austin jake. www.yyteveaaaa..oooo edu
Tyler Westenbrock" University Washington westenbrooktt@mail.cco
David Fridovich-Kell
University Texas at Austin dfk@utexas. edu
Abstract: Traditionally, model-based reinforcement learning (MBRL) methods exploit neural networks as flexible function approximators to represent priori unknown environment dynamics. However, training data are typically scarce practice, and these black-box models often fail to generalize. Modeling archi- tectures that leverage known physics can substantially reduce the complexity cystem-iientiiiicttinn, but break down in the face of complex phenomena such as contact. We introduce novel framework for learning semi-structured dynamics models for contact-rich systems which seamlessly integrates structured first prin- ciples modeling techniques with black-box auto-regressive models. Specifically, we develop an ensemble of probabilistic models to estimate external forces, condi- tioned on historical observations and actions, and integrate these predictions using known Lagrangian dynamics. With this semi-structured approach, we can make accurate long-horizon predictions with substantially less data than prior methods. We leverage this capability and propose Gemi-Structured Reinforcement Learn- ing (SSRL) a simple model-based learning framework which pushes the sample complexity boundary for real-world learning. We validate our approach on real- world Unitree Gol quadruped robot, learning dynamic gaits - from scratch on both hard and soft surfaces with just few minutes of real-world data. Video and code are available at: https: :/t::://siiie /stttssttsssse//////////e//eeees google. com/utexas edu/ssrl
Keywords: Model Based Reinforcement Leaminn, Physics-Basee Models
1.0
0.33
15
0.13
3.0
0.88
min.
Figure 1: Unitree Gol quadruped learning to walk from scratch using SSRL on hard ground (left) and memory foam (right).
1 Introduction
Effective robotic agents must leverage complex interactions between the robot and its environment, which are difficult to model using first principles. Model-based reinforcement learning (MBRL) is a powerful paradigm for controller synthesis [1], wherein the robot learns a generative dynamics model for the environment. The model can then be used to hallucinate synthetic rollouts [2], provid- ing a source of data augmentation for policy optimization algorithms [3. 4, 5]. When model accurate, can generate long rollouts which extrapolate beyond the training data, accelcrating pol- icy learning substantially. However, in practice, the black-box neural network models favored in the
"Thee authors contributed equally.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Seni-anuusured Dynemics
Lagrangian Dynamice
MG44 Cla.0le G(o)
Deterninteilc
Roal world
EXC. Tarque Estimattoss
Hall cinationn
Stachostic Polley
No Noioo Extimations
Pollcy
The SSRL framework. A deterministic policy is used to collect data from the real world while stochastic policy is utilized in conjunction with the learned dynamics model to "hallucinate" short synthetic rollouts which branch from this data. The model incorporates Lagrangian dynamics and encodes previous state predictions, which are fed external torque and noise estimators to predict future states. The synthetic data is used with a model-free RI. algorithm to update the policies.
Figure 2:
MBRL literature struggle to generalize beyond the training data [6, 8], and thus do not outperform modern model-free alternatives [9, 10). Currently, both paradigms are too inefficient and unreliable make learning new behaviors the real world practical for many applications.
An appealing alternative is to leverage known physics to design structured model classes. This general approach has been used for efficient system identification and controller synthesis across many bodies of work, ranging from classic adaptive control techniques [11, 12, 13, 14] to more recent physies-informed neural architectures [15, 16, 17]. However, these previous approaches do scale to the complexities of real-world learning for contact rich-systems such as walking robots (Fig. 1). Indeed, modeling contact remains open problem. Moreover, prior structured approaches to learning contact dynamics make strong assumptions about what can be perceived with available on-board sensors. For example, inference time [18] requires access to signed-distance repre- sentation of potential contact surfaces. More generally, prior approaches generally assume access to privileged state observations which make it possible to predict exactly when and where contact is made [19, 15J. However, reliably estimating these quantities in the real-world using noisy on-board sensors is another open area of research [20, 21, 22, 23, 24].
build light-weight modeling framework which leverages known structure and is implementable with onboard sensors? We answer this question in the affirmative by introducing Semi-structured Reinforcement Learning (SSRL), a simple MBRL. pipeline for contact-rich control in the real-world (Fig. 2). For concreteness, we focus primarily on the quadruped depicted in Fig. and aim to learn an effective locomotion controller entirely from scratch in the real world. We consider the case where the robot's observations only include proprioceptive measurements via joint encoders, IMU measurements, and global velocity estimator.
We ask:
We inspire our approach by looking to the Lagrangian equation of motion for the robot:
M(q) C(g. + G(g) Br J F +Td
where & and 9 are the generalized coordinates and velocities of the robot, motor torques 7 are dis- tributed the joints by the matrix B, F represents contact forces generated by the environment with J the resulting Jacobian, 7d represents parasitic damping torques, and M, C, and G the mass matrix, Coriolis and centrifugal forces, and gravity vector. Here, B, M, C,, and G are deter- mined by the geometry and inertial properties of the robot, which are known priori. Thus, given access to observations, we assume that the only unknown terms are J pe and rd
We leverage the known Lagrangian structure of (1) by estimating these unknown terms with an es- timator (h,) J 0e + 7d which is conditioned on a history hy of the available measurements, enabling the model to infer information about hidden states of the environment such the geometry of the ground underfoot [25, 26). We instantiate 7 as an ensemble of probabilistic models to help

---

Controller
Quadruped
Stepping metion
Params.
Policy
Generator
Joint angles
Inverse Kinamatics Solver
Joint PD
Torques
offsets
History (h,) of Proprioceptive Observations (sc)
Figure 3: Control architecture. The policy takes in history of observations and outputs parameters gait generator offsets to the gait. resulting foot positions are sent to an inverse kinematics solver which computes desired joint angles for joint level PD controllers.
capture uncertainty in the predictions, is commonplace in the MBRL literature [6, 27J. We then introduce a novel inference procedure for making forward predictions, wherein the predictions made by these work leverages these predictions optimization, using a simple Dyna-style [3, 6] MBRL algorithm. Finally, we demonstrate that this liight approach scales to the woold, learning to walk on different terrains within a minute and learning substantially more dynamic gates within few minutes. This result outperforms recent state-of- of-ttheart real-world learning results for quadrupeds [28] by both a) requiring an order-of- magnitude less real-world data and reaching substantially higher walking speeds.
models are fed back into f in rugo-segressive fashion. The SSRL frame-
source of data augmentation to accelerate real-world policy
Preliminaries and Problem Formulation
Notation: While the underlying dynamics of the robot evolve in continuous time according to the differential equation (I), the policy acts in discrete time and we will subscripts denote discrete time steps. The state of the robot 8, S captures the available proprioceptive states qi and their velocities 4t, available from joint encoders and IMU measurements. This does not include global coordinates of the robot. We capture the state of the environment (or extrinsics) ez E E, which includes non non-proprioceptive information (such as the distance of the ground below the robot) and the state of the ground underfoot (such as terrain deformation). The robot actions a, E A are the outputs of neural network described below. The overall dynamics for the system are defined by:
Robot Transitions: 8++1 ~ Pe( (t, ec) Environment Transitions: ex+1 ~ Pe at, Et). (2)
To control this joint system, we will denote the policy the agent optimizes via ae 7-(- h4), where 1ú = (..-1,...., 8t-n)) bundles the histories state measurements and 0 are parameters.
Control Architecture: We adopt the control architecture depicted in Fig. 3, similar prior works on RL-based locomotion [25, 29, 30]. We optimize neural network policy (-(-|s,, h2) conditioned on previous observations which outputs (i) changes to parameters of nominal gait generator that outputs desired foot positions and (ii) offsets to these foot positions. The resulting targets are sent to an inverse kinematics solver which computes desired joint positions. The joint targets are output at 100 Hz to low-level PD controllers for conversion to torques. Additional details are provided in Appendix
Reinforcement Learning Problem: We formally frame the learning of locomotion controller in the real world in terms of partially observable Markov decision process (POMDP) [31], de- p, S2 7, Here, X' = S x E is the overall state space for the system, and p(-|s,, Oz, e,) = (- 0,, 6x), Pe(: (s,,,, Ge, cx)) captures the joint robot-environment dynam- ics. The space of observations Ω consists of the states that can be measured, and the observation distribution O(-Ise, at, et) provides (noisy) estimates of the states from onboard sensors. The reward function r(t,, azs 11+1) depends only upon the robot states and actions, and therefore directly mea- surable in the real world. We define the reward function to maximize the robot's is forward velocity, maintain upright orientation, minimize angular rates, conserve energy, and avoid excessive torques.
fined by the tuple (X, A,

---

Finally, we define a termination flag de E {0, where d, = 1 when body roll pitch exceed limits. Exact definitions are found in Appendix A. Given an episode length 7 E N, discount factor Y E (0, 1), and a distribution Zy over X' of initial conditions for the system, the goal to maximize the expected discounted total reward: max, B 7' d,) r(s,, a, s(+1)]-
Semi-structured Reinforcement Learning
A high-level overview our method is presented in Fig. 2, outlined as follows: we learn an ensemble of deterministic external torque estimators 4e which conditioned hy ii) we then integrate the torque predictions through the Lagrangian ODE (I), add a learned noise term to gener- ate probabilistic 1-step predictions, and feed this prediction back into h, to produce auto-regressive predictions over multiple steps iii) we fit this semi-structured representation of the dynamics using multi-step prediction losses; and finally iv) predictions from these models are used as a source of data augmentation for MBRL.
3.1 External Torque Estimators
We construct our approximations to the discrete-time time probabilistic robot transition dynamics 8:+1 ~ P. p-(11sr, ae, ex) by building on top of the deterministic Lagrangian dynamics (1) which we rewrite as:
M ( Cla, + G(g) Br + e(,,e) J' (s,,),
(3)
r=(,,,7))
where we now explicitly denote the dependence of the contact Jacobian J(s, e), contact forces F C), and dissipative terms (s, e) on the state the robot, environment, and low-level torques supplied by the motor. As discussed in Section I, identifying precise locations where con- tact occurs on the robot can be extremely difficult from on-board measurements. Thus, instead of inferring J and Fe separately, latent encoding 2 (b(h)) of ht, which enables the network to infer information about the extrin- sics needed for predicting future states [26, 25). Altogether, use ensemble of deterministic models 7" = 7f = To. (St, a, 21) with tunable parameters t4.
directly estimate 7 J F 77d We condition these estimates
3.2 Generating Forwards Predictions
We first describe how we generate a probabilistic 1-step prediction for the next state associated to each of the torques estimates n" Ff Tv. (st, ae, 2e). First, we let T; G(st, az) denote zero- order hold estimate for the low-level motor torques applied to the robot; here, G is known map which captures how the current state and action are processed by our control architecture (Fig. 3) to produce low-level joint torques. The i-th deterministic next-state prediction is then given by 8 = [st, Te, ) which captures how chosen numerical integrator for the Lagrangian dynamics (1) propagates 8t, Tt, and 7. We then add zero-mean Gaussian noise to 8 whose variance Σ₁
No, (st, a2, 2t) is diagonal Gaussian which is output by the same network predicting the i-th torque estimate. Altogether, this constructs a probabilistic estimate for the next state:
Uncertainty-ANwe State Predictions: $ p.. (-1st, de, h4) == ((
The ensemble of probabilistic models ( } is used to generate k-step predictions using Algo- rithm 1, which adapts the method from [6] to our auto-regressive setting where future state predic- tions are conditioned on the history 14- Given a state 8, and state history ht, to generate a synthetic rollout we: (i) sample an action from the policy, (ii) randomly choose model P from the ensem- ble, (iii generate a next-state prediction 64 using (4), and iv) incorporate this prediction into h and repeat the previous steps until a k-step prediction has been generated.
3.3 Approximate Maximum Likelihood Estimation
To train (P) 10hhs hi we maximize the foint-likelißee of state predictions along these synthetic rollouts under the real-world data. Due to the Markov assumption (Section 2), the joint distribution of

---

Algorithm Auto- Retto-egreesivv State Predictions
1: Inputs hallucination buffer Dradel; models (P policy zo, start state s0, start history ho
2: for = O...k A do
3: Sample action a ~ Tp(- 8t, ht)
4: Randomly choose model i ~ [[[1, ..... P| and predict next state $/41 with (4)
5: Update state history h441 with +1 and 86+-1 #++1
6:
Compute reward r and termination d, and add transition (st. at, Ft, 8141, 1 de) to Drodel
7:
Return D Puodel
the next states, starting at state 8, ps ($ 8t, Oct+H Cc(+H)
I Ps (+++1+)
814-j artij ex+j). Taking the negative log- lillikki yields our training objective for the i-th model:
N.
H
H Σ j=0
((v)) HN.
Σ
st s
S441+5]
(Et+)) [ S4+149] + log det Eitz- (5)
Here, N₂ is the size of buffer of real-world transitions Deuvs mean state predictions are propagated deterministically according to 8/41 S (s), at he), and each prediction & is used update the state history he For each state 8, in the buffer, (5) generates synthetic rollout H steps long and computes loss related to how much the propagated states differ from the experienced states.
By optimizing the multi-step loss (5) end-to-end, we i) force the latent 2, to encode information about the environment which is necessary for making state predictions over long hori- zons and ti) average out noisy state estimates and sudden changes in the external torques that occur when the robot makes contact with the ground (Fig. 4). This leads reliable long-horizon predic- tions which are substantially more accurate than those generated by black-box models (Fig. 5).
Remark 1.
3.4 Policy Optimization
Finally, introduce the Semi-Structured Reinforcement Learning (SSRL) in Algorithm 2. This policy optimization strategy is Dyna-style algorithm [3, 6) which leverages model-predictions as a source of data augmentation. Specifically, this approach branches hallucinated rollouts off real- world trajectories to reduce the effects of model bias, and increases the length of these rollouts throughout training as the model becomes more accurate. We perform many rounds of data gen- eration and policy optimization after collecting real-world samples, and as depicted in Fig. 5 our semi-structured model are able make substantially more accurate predictions over long horizons when data is scarce. Together, these features enable SSRL push the sample complexity limits for MBRL. We also found the following necessary for making real-world learning practical:
Deterministic Real-World Rollouts and Random Hallucinations. Steps in the real environment are taken using a deterministic policy μ4 which simply outputs the mean action from the stochas- tic policy Tg. Black-box MBRL approaches typically need inject random 'dithering" noise to ensure their is enough exploration obtain accurate models. However, we found that this led to unacceptably erratic behavior in the real-world when paired with policy class. Recent model- free approaches for learning locomotion in the real-world [32] severely restrict the action space to make stochastic exploration tractable, and thus severely limit the performance of resulting policies. However, our semi-struetured models have the right amount of inductive bias to accurately iden- tify the dynamics when the real-world data set Deav is generated with deterministic exploration and data is searce. We do, however, use stochastic policy mo to generate rich, diverse synthetic data sets Dmodel for policy optimization. Specifically, policy parameters 0 are trained with soft-actor critic (SAC) [33], using mixture of transitions from real-world and hallucination buffers Dary and D Dmodel-
5

---

Algorithm 2 SSRL: : Policy Optimization with Semi-structured Dynamics Models
1: Initialize models pv., policy ne, critics Q+.
2: for Nepoche epochs do
3: Take Ng steps in the environment deterministically with μd and add transitions to Denv
4: Train models Py, on Peav using loss (5)
5: Increase hallucination rollout length &o according to predetermined schedule
6: for K hallucination updates do for M model rollouts do Sample state so uniformly from Dane and hallucinate with Algorithm 1
7:
8:
9:
Perform G updates of policy ny using mixture of Dany and D nodal at ratio rDD
4 Experimental Results
4.1 Real-world Results
We demonstrate our approach through two real-world experiments where Unitree Gol quadruped trained from scratch achieve maximum speed on both hard ground and memory foam.
Experimental setup. Training is performed from scratch in the real world over Napocha = 18 epochs with Ng = 1000 environment steps per epoch, totaling 18, 000 steps or 3.0 min of real- world interaction. We use an observation history length of h = 5, a multi-step loss horizon of H = 4, and hallucinate synthetic rollouts for up to k = 20 steps; see Appendix D for hyperparam- eters. To enhance plasticity, the model, actor, and critic are reset at 10, 000 steps [34]. Joint states are measured via encoders, IMU data provides body orientation and angular velocity, and linear velocity is acquired with Vicon motion capture system. Neural networks are trained in JAX [35], and low-level joint angle commands are sent via Unitree's ROS interface [36]. We compute the mass matrix, Coriolis terms, and gravity vector in (3) using the differentiable simulator Brax [37], enabling gradient-baat based training of the semi-structured dynamics model with (5).
Fig. shows time-lapse rollouts as training progresses. After just 3.0 min of real-world data, the quadruped achieves average velocity of 0.98 ms-1 on hard ground and 0.53 ms on memory foam (Fig. 4, right). Figure 4 (left) plots the reward per episode until the first termination (d, = 1). Initially, the quadruped often falls, resulting in low rewards, but after 1.5 min, the learned policy becomes robust, and rewards increase. Despite the challenge of walking on memory foam, where the robot's feet sink deeply, forward velocity improves consistently in both terrains, demon- strating our approach's 'ss versatility to differing contact dynamics. To examine model accuracy, we compare the learned external force predictions eee second. We estimate the real-world external force with 7" MM + C(g. qe) G(gg)) Br where joint accelerations are estimated by finite differencing: ÿ (शr+1 - At and motor torques Tr are estimated with the PD control law, Figure 4 (right) shows the learned external vertical force predicted on the base of the robot. Notably, the predictions appear as smoothed versions of the ac- tual torque estimates. This, when combined with the accuracy of our predictions over long-horizons (Section 4.2) provides insight into why our approach enables such effective policy optimization [38]. Appendix B.1 provides plots for the force estimated on additional degrees of freedom for the robot.
Results.
to real-world external force estimates 7f over one
4.2 Simulated Experiments
In addition to the results presented here, we provide extensive ablations on standard RL benchmarks in Appendix C. Here, we investigate the following hypotheses:
Hypothesis 1. SSRL boosts performance and sample efficiency for MBRI. in contact-rich settings.
Hypothesis 2. Training with a multi-step loss outperforms a single-step loss.
Hypothesis 3. Predictions from cemi-structured dynamics models demonstrate greater accuracy and improved generalization beyond training data compared to black-box models.

---

Policy Performance
Walking Performance
Prediction Accuracy
1,0
700
430
Foam
Awors Velocity (mis)
0.9
Foam
300
Reward
500
0.5
Force (N)
200
400
300
WWW
100
0.2
100
0.0
-100
Minutes Real Data
Minutes of Real Dala
Time (9)
Figure 4: Real-world results. Left-SSRL efficiently performs policy optimization, even when data is scarce. Center-With approach, the quadruped steadily learns to walk faster. Right-Prediicted and real external vertical force acting the robot base differences. The predictions add noticeable smoothing to the real-world data.
second of real-wor worldworld data. Real forces are estimated by finite
Experimental setup. The simulation setup mirrors the real-world setup (Section 4.1), except roll- outs are simulated in Brax [37]; parameters are detailed in Appendix D. To examine Hypothesis 1, we perform policy optimization with SSRL (Algorithm 2) and compare its policy performance to a baseline approach that utilizes the same optimization process but substitutes the semi-structured dynamics model with a black-box model. We also benchmark against SAC [33], with an extended run shown in Appendix B.6. To test Hypothesis 2, we repeat the prior experiment, except we train the model with a single-step loss horizon (H 1). To assess generalization (Hypothesis 3), we train stri-structured models and the black-box models from scratch over 3 minutes of saved simu- lated data using 1- and 4-step losses. We then generate new data using a stochastic policy where we lower the friction and ground contact stiffiness by 25% in the simulator. We generate 20-step synthetic rollouts from 400 randomly-samplee starting states within the new dataset and average the prediction error || sell (s:) over the 400 rollouts. All runs are repeated across random seeds.
Results. Shown in Fig. 5 (left), semi-structung dynamics models outperform black-bot models in both sample efficiency and maximum reward, supporting Hypothesis 1. Even when data is scarce, our semi-structured model generates more accurate synthetic rollouts during exploration, resulting in significantly improved policy performance. We also observe Fig. 5 (left) that, while black-box models yield similar policy performance for both single- and multi-step losses, training our semi- structured dynamics models with multi-step loss results in improved performance over a single- step loss, confirming Hypothesis 2. Fig. 5 (right), see that semi-structured models produce predictions 20 steps into the future that are significantly more accurate than black-bos box models. By leveraging physics-basee knowledge, our models generate synthetic rollouts that generalize better to an unseen environment, confirming Hypothesis 3; additional experiments are found in Appendix B.
Policy Performance
Halkuzination Accuracy
0,40
Cum, 1istpp
0.35
1.stpp
Black box, 4-slop
Blacs-hox,
Predicion Enrer
Reward
0.25
0.15
0.10
0.00
5000 10000 15000
20000 25000
50000
35000
40000
10
54
18
Stepe
Figure 5: Left SSRL achieves better policy performance compared to baseline using black-box models. Right-Prediinn Prediction enror for 20-step synthetic rollouts in an unseen environment showcases our method's supe- rior ability to generalize.

---

Related Work
Model-based RL. Our works builds on a wealth of prior works that use general function approxi- mators and probabilistic modeling for black-box modeling [39, 6, 27J. Model-based reinforcement learning algorithms either learn a model that is used for online planning [40, 27, 41] or Dyna-style algorithms which hallucinate imagined rollouts for direct policy optimization [6, 3, 4]. Moreover, significant attention been devoted extending these strategies to learn latent representations for en- vironments from available ohservations [2, 42]. However, these approaches do not leverage known structure to accelerate learning. In contrast, there has been a substantial line of work which leverages known structure to efficiently learn accurate models [15, 16, 17, 19, 15J. However, these approaches generally require access to the full state of the system at inference time, which is impractical for real- world learning for contact-rich with on-board perception. Thus, our semi-structured approach brings the strengths of both paradigms to bear in a single framework. Finally, we also note related work [43, 44, 38] which investigates smoothing out contact dynamics make policy optimization more tractable for difficult contact-rich problems. We found that our Iearned models naturally learned smooth representations for the dynamics which generate accurate long-horizon predictions (Fig. 5).
Model-free RI. A parallel line of work [9, 10, 28] aims to make off-policy model-free algorithms (which form the back-bone for our policy optimization strategy) more stable and efficient in low- data regimes. These approaches introduce regularization techniques which enable the use of higher update-tor tatte ratios without overfitting to the available data, matching the efficiency of model- free methods such as the MBPO [6] algorithm that we build upon. These algorithmic advances are generally orthogonal to our contribution, and thus in the future we plan to incorporate them into our framework to further accelerate real-world learning.
Learning Locomotion Strategies in the Real World. Learning locomotion behaviors from scratch directly in the real-world has primarily been studied in the context of model-free reinforcement learning [45, 46, 32, 47], with few works using black-box models in the context of model-based reinforcement learning [48, 49]. Compared to these works, our semi-structured modeling approach enable the robot to achieve more dynamic locomotion strategies than these previous approaches, with just a fraction of the real-world samples. Specifically, our approach either achieves a signifi- cantly higher walking speed than each of these approaches, or improves on their sample complexity by approximately an order of magnitude (Fig. 4). Several other works investigate fine-tuning loco- motion controllers trained in simulation to reduce the burden on real-world data [28, 50] as we discuss below, we hope to investigate this direction in the near future.
Direct Transfer From Simulation. There has also been recent and rapid progress directly transfer- ring locomotion controllers from simulation zero-shot [51, 26, 25, 52, 53), using techniques such as domain adaptation and domain randomization. this paper we have focused on learning locomo- tion controllers from scratch, in an effort to demonstrate the ability of our framework to substantially adapt the behavior of the robot with small amounts of real-world data. However, in the future plan to fine-tune policies that have been trained using extensive simulated experience, improving the performance of these policies in cases where they fail [28] but leveraging better initialization for the policy for real-world learning.
6 Limitations
This paper presents a novel framework for model-based reinforcement learning, which leverages dbysics-informed semi-structured dynamics models to enable highly sample-efficient policy learn- ing in the real world. However there are several key limitations. First, our method requires ob- servability of enough proprioceptive states propagate the Lagrangian dynamics of the robot. Ad- ditionally, relying solely on proprioception restricts the model's ability to predict changes to the environment such the appearance of obstacle transitions between different ground surfaces. In the future we plan to extend the current framework to include additional perceptual modalities which can infer more about the state of the environment around the robot.
8

---

Acknowledgments
The authors would like to thank Trey Smith and Brian Coltin for their helpful insights and feedback. This work was supported by NASA Space Technology Graduate Research Opportunity under award 80NSSC23K1192, and by the National Science Foundation under Grant No. 2409535.
References
[1] M. Moerland, Broekens, A. Plaat, C. M. Jonker, et al. Model-based reinforcement learn- ing: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
[2] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: (Tt1::99/2.00000 2019.
[3] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.
[4] R. S. Sutton, C. Szepesvári, A. Geramifard, and M. P. Bowling. Dyna-style planning with linear function approximation and prioritized sweeping. arXiv preprint arXiv:1206.3253 2012.
[5] H. Yao, S. Bhatnagar, D. Diao, R. Sutton, and Szepesvári. Multi-step dyna planning for policy evaluation and control. Advances in neural information processing systems, 22, 2009.
[6] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[7] Yu, G. Thomas. Yu, S. Ermon, J. Y. Zou, Levine, C. Finn, and Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14129-14142, 2020.
[8] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954 28967, 2021.
[9] X. Chen, C. Wang. Z. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without model. arXiv preprint wrXiv:2101.059:. 2021.
[10] Hiraoka, T. Imagawa, T. Hashimoto, T. Onishi, and Y. Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXlv:2110.02034. 2021.
[11] J.JJ. E. Slotine and W. Li. On the adaptive control of robot manipulators. The international journal of robotics research, 6(3):49-59, 1987.
[12] Sastry and M. Bodson. Adaptive control: stability, convergence and robustness. Courier Corporation, 2.11.
[13] T. Westenbroek, D. Keiovich-Keil, E. Mazumdar, S. Arora, V. Prabhu, S. S. Sastry, and C. J. Tomlin. Feedback linearization for uncertain systems via reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1364-1371. IEEE, 2020.
[14] G. Tao. Adaptive control design and analysis, volume 37. John Wiley & Sons, 2003.
[15] F. Djeumou, C. Neary, E. Goubault, S. Putot, and U. Topcu. Neural networks with physics- informed architectures and constraints for dynamical systems modeling. In Learning for Dy- namics and Control Conference, pages 263-277. PMLR, 2022.
[16] M. Cranmer, S. Greydanus, S. Hoyer. P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.0460 2020.

---

[17] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2019. URL.
[18) S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. In Conference on Robot Learning, pages 2279-2291 PMLR, 2021.
[19] Y. Jiang, T. Zhang, D. Ho, Y. Bai, C. K. Liu, S. Levine, and J. Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 2884-2890. IEEE, 2021.
[20] P. Fankhauser, M. Bloesch, and M. Hutter. Probabilistic terrain mapping for mobile robots with uncertain localization. IEEE Robotics and Automation Letters, 3(4):3019-3026, 2018.
[21] R. Yang, G. Yang, and Wang. Neural volumetric memory for visual locomotion control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages [430-1440) 2023.
[22] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics,
2022.
[23] L. Manuelli and R. Tedrake. Localizing external contact using proprioceptive sensors: The contact particle filter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5062-5069. IEEE, 2016.
[24] S. Haddadin, A. De Luca, and A. Albu-Schäffer. Robot collisions: survey on detection, isolation, and identification. IEEE Trunsuctions on Robotics, 33(6):1222-1312, 2017.
[25] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47)xeabe5986, 2020.
[26] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv: 107.:21077004344 2021.
[27] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[28] L. Smith, J. C. Kew, X. B. Peng, Ha, Tan, and Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference on Robotics and Automation (ICRA), pages 1593-1599. IEEE, 2022.
[29] G. Bellegarda and A. Ijspeert. Cpg-rl: Learning central pattern generators for quadruped loco- motion. IEEE Robotics and Automation Letters, 1(4):12547-1255, 2022.
[30] T.-Y. Yang, T. Zhang, L. Luu, S. Ha, J. Tan, and W. Yu. Safe reinforcement learning for legged locomotion. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2454-2461. IEEE, 2022.
[31] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
[32] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.0780 078800 2022.
[33] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
10

---

[34] E. Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pages 16828-16847. PMLR, 2022.
[35] Bradbury, Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transfor- mations of Python+NumPy programs, 2018. URL http: joagg///....co.gggg///a/.aa.a
[36] U. Robotics. Unitree ros to real. to_ to_real, 2021.
[37] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax-a differen- tiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 13281, 2021.
[38] H. J. т. Suh, T. Pang, and R. Tedrake. Bundled gradients through contact via randomized smoothing. IEEE Robotics and Automation Letters, 7(2):4000-400 40000, 2022.
[39] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference on machine learning (ICML- I)), pages 465-472, 2011.
[40] A. Tamar, Y. Wu, G. Thomas, Levine, and Abbeel. Value iteration networks. Advances in neural information processing systems, 29, 2016.
[41] S. Racanière, T. Weber, D. Reichert, L. Buesing, A. Guez, D. Jimenez Rezende, A. Puig- domènech Badia, O. Vinyals, N. Heess, Y. Li, et al. Imagination-augneen agents for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[42] N. Hansen, H. Su, and X. Wang. TD-MPC2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024. URL attps://ont /ttp:r///pperrvier. net:/oormmidddxxoooeeee
[43] T. Pang. H. T. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models. IEEE Transactions on robotics, 2023.
[44] A. Block, M. Simchowitz, and R. Tedrake. Smoothed online learning for prediction in piece- wise affine systems. Advances in Neural Information Processing Systems, 36, 2024.
[45] N. Kohl and P. Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 3, pages 2624. IEEE, 2004.
[46] R. Tedrake, T. W. Zhang, and H. S. Seung. Stochastic policy gradient reinforcement learning on a simple 3d biped. In 2004 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)IEEE Cat. No. 04CH37566), volume 3, pages 2849-2854, IEEE, 2004.
[47] G. Endo, J. Morimoto, T. Matsubara, J. Nakanishi, and G. Cheng. Learning cpg sensory feedback with policy gradient for biped locomotion for full-body humanoid. In AAAI, pages 1267-1273, 2005.
[48] S. Choi and J. Kim. Trajectory-based probabilistic policy gradient for learning locomotion behaviors. In 2019 International Conference on Robotics and Automation (ICRA), pages 1-7. IEEE, 2019.
[49] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, Tan, and V. Sindhwani. Data efficient rein- forcement learning for legged robots. In Conference Robot Learning, pages 1-10. PMLR, 2020.

---

[50] T. Westenbroek, F. Castaneda, A. Agrawal, S. Sastry, and K. Sreenath. Lyapunov design for robust and efficient robotic reinforcement learning. arXiv preprint arXiv:2208.0672), 2022.
[51] Hwangbo, C. D. Bellicoso, Fankhauser, and M. Hutter. Probabilistic foot contact esti- mation by fusing information from dynamics and differential/forward kinematics. In 2016 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3872- 3878. IEEE, 2016.
[52] T. Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, x(62))eab22222, 2022.
[53] Z. Li, X. Cheng, X. B. Peng, Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- learning for robust parameterized locomotion control of bipedal robots. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 281 2811-7 IEEE, 2021.
[54] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
12

---

A
Implementation Details
In this appendix, provide details of our implementation for the Unitree Gol Quadruped, including the observation and action spaces, the reward function, the termination condition, and the control architecture.
A.1 Observation and Action Spaces
The observation space C R consists of the elements in Table The z-axis the base is the forward direction, the y-axis is the leftward direction, and the z-axis is the upward direction. The phase variable [0, 2x) represents progression along the gait cycle and is defined as oe 2xt T mod (2x) where T 0.5 see is the gait cycle period.
Observation
Symbol (0= qa
Dimension
Quaternion orientation of the base
4
Joint angles
12
Base linear velocity (local frame)
(wo Go
v²) wy q w))
3
Base angular velocity (local frame)
Joint speeds
12
Cosine of phase
COS Φ sin Ø
1
Sine of phase
1
Table 1: Observation space.
The action space A R outputs the change in nominal height for the gait generator and offsets nominal foot positions from the gait generator, defined in Table 2.
Action
Symbol
Dimension
Min.
Max.
z-foot position changes
Ap
4
-0.15 m
0.15
y-foot position changes
Ap"
4
-0.075 m
0.075
Change in gait generator nominal height
A Gait
-0.1 m
0.0
Table 2: Action space.
A.2 Reward Function and Termination Condition
Reward Function. The reward function is weighted sum of the terms in Table 3. We set the weights and use exponentials in most of the terms to normalize the reward such that a forward velocity of 1.0 .0 with maximal values for all other terms will result in reward of approximately 1.0 for single time step. The roll s"" pitch y", and yaw e of the base are obtained from the base quaternion p. We define @ the element-win wise multiplication of vectors and Actual torques output from the joint-level PD controllers are not available; estimate the torque applied at the joint with (11). We define the following LinearLimit function which linearly penalizes the torque applied at the j-th joint 7 when exceeding torque limits; within torque limits, the function is decaying exponential:
7
if < T
min
LinearLimit(7), T Tdke))
exp
-
Tnin
if T < T <0
(6)
i < 7¹ < T
exp
max
7
if T 2 TJ
38x
13

---

Reward Term
Expression
Weight
Maximize forward velocity
27-1
0.42
Limit base yaw rate
cxp
(wf+1)2/0.2)
0.11
Limit base roll
exp
(+/t+)2/00255
(5
0.05
Limit base pitch
exp
((/++))/00255
0.05
Limit base yaw
exp
-(+4+1)/00.07
0.11
Limit base side velocity
cxp
(11+1)
0.11
Limit vertical acceleration
exp
(v841 x/02/00.22)
0.03
Limit base roll rate
exp
(Pī-1 /00000.0011
0,03
Limit base pitch rate
exp
(p -))² /0.005
0.03
Limit energy
exp
1/t1 3 T++1 /40
0.05
Penalize excessive torques
Σ, LinearLimit(7 Tuan 1/12
0.02
Table 3: Reward function terms. The reward at each time step is weighted sum of these terms.
Termination Condition. The termination flag de stops the accumulation of reward after the quadruped falls and is defined by:
de { if > x/4 0 T/4 00 otherwise.
A.3 Control Architecture
Here, give detailed specification of the control architecture introduced in Section 2. Referring to the action space definition Table 2, the policy takes in the current observation and history of observations and outputs offsets to foot positions and a nominal height for the gait generator: (Apī, Api, AAfait) ~ TT(- St, 12). The gait generator Gait [0, 1) x IR + IR' is open-loop and generates for each leg, walking-in in-acce foot positions for the quadruped by computing vertical foot position offsets from nominal standing foot positions:
Gait Ancate)
I' +
0, 0, Being 1
COS
2T oi -att Gatt
1-
))
-A
Angait
if क Gait otherwise
(eerand +
[0, 0,
A
Gait]
(8)
where Pitand E RD is the nominal standing foot position of the I-th leg, expressed in the local base frame, //Sning 0.09 is the gait peak swing height, and Gastt = 0.5 is that fraction of the time feet should remain in contact with the ground. The normalized phase ø² E [0. 1) specifies the progress of the 1-th leg along its gait cycle and is calculated with:
Φt +..5+b/) 2π
o. =
mod 1,
where b' is the phase bias for the 1-th leg; we use value of 0 for the front-right and rear-left legs, and value of 0.5 for the front-left and rear-right legs. The desired positions of the I-th foot in the local base frame are given by:
pf Appl, Abgait Ø1) = App a. Apt .0] Gait(ol; Ansait)
(10)
where Api" and App are the and y-foot positions offsets for the 1-th foot from the policy. For each foot, the desired foot positions (10) computed and sent to inverse kinematics solver to produce desired joint angles ddes E 12 The desired joint angles are sent to the joint level PD controllers, where the desired torque outputs are:
Ky q²) K,
(11)
and we proportional gain K₂ = 112 N m madd and derivative gain K, 3.5 N m msrad
14

---

B Additional Experiments
In this appendix, we present the results of additional experiments which demonstrate our model's ability to make accurate predictions far into the future, generalized to unseen training data, and enable efficient training.
B.1 External Torque Estimate Validation
Contact forces are discontinuous in nature and can be difficult to learn. To this end, we validate the learned external torque predictions 5. by comparing them to estimates the real-world external torques 7 over one second of real-world data, presented in Fig. Estimates of the real-world ex- ternal torque are computed T AM (q: C(a., qt) + G(q.) BT, where joint accelerations estimated by finite differencing: q: ($++1 /.) and low-level motor torques тr are es- timated with the PD control law. The external forces acting the floating base and the external torques acting on the joints of the front-right leg are shown. Our learned external torque predictions closely align with the estimated actual external torques. Notably, the predictions appear as smoothed versions of the actual torque estimates. Finite differencing of the velocity measurement introduces noise into the actual torque estimates while our multi-step loss learning process inherently smooths the predictions.
Base
Base
Base
200
Actual
400
150
Predicted
200
100
Force (N)
50
100
200
200
-50
100
-100
-200
-100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Front-right Hip
Front-right
Front-right
20
20
10
Tarque (N-m)
0
-10
-10
-20
20
30
0.0
0.2
0.4
0.6
0.8
1.0
0.8
1.0
0.0
0.2
0.4
0.6
0.8
2.0
(5)
Figure 6: Predicted external forces and actual external force estimates second of real-world data for the floating base and the joints of the front-right leg. Our learned external torque predictions closely align with the estimated actual external torques.
B.2 Model Rollout Accuracy
Here, we examine the accuracy of trajectories generated from learned models and the ability of the model to generalize beyond the available training data. First, we train our semi-structured models and the black-box models from scratch over minutes saved simulated data using 1- and 4- step losses. Using the trained models, 20-step synthetic rollouts are generated from 400 randomly- sampled starting states within the data. The average prediction error || sell/dim at each time step is averaged over the 400 trajectories. This repeated for random seeds where the best results from the 1- 4-step losses for each model type recorded. We then repeat this experiment for the real-world dataset. Finally, the experiment is repeated again, except generate
15

---

new simulated dataset using stochastic policy and on altered terrain: friction and ground contact stiffness are lowered by 25%. For this final case, the saved models from the first case are evaluated on this new dataset. The results for these experiments are presented in Fig. 7. For all cases, our semi-structured models produce predictions 20 steps into the future that are significantly more accurate than black-box models. The results from the last experiment demonstrate our model's superior ability to generalize unseen environment where noise is added to actions.
Simulated
Real-world
Simulated Data, Dfferent Stochastic
0.005
Ours, 4-dlap loss
Curs, 4-6lep loss
Ours, 4-5bep loss
0.004
Black bocck-bx..c
0.05
Backk-box, loss
Black 1sttp
Prediction Erro
0.04
0.003
0.03
0.2
0.002
0.02
0.001
0.01
0.000
2
4
12 Steps
10
14
16
18
20
4
6
10 Steps
12
14
16
18
20
2
8
12 Steps
10
14
16
18
20
Figure 7: Prediction error for 20-step synthetic rollouts using our semi-structured dynamics models and the black-box models where the best results from the 1- or 4- step losses are presented. Pre- diction error averaged over 400 trajectories. Plots show the mean and standard deviation over random seeds.
B.3 Robustness to Errors in the Lagrangian Dynamics
In this experiment, we demonstrate that SSRL performance is robust against errors in a priori knowl- edge of the robot's 's inertial properties which are used to construct the Lagrangian dynamics (1). To simulate these modeling errors, we randomly vary each link's mass by +255 and each joint's damp- ing by +5% for the Gol environment used for simulated data collection. The remainder of the setup per Section 4.2 and runs repeated across 4 random seeds. The results presented in Fig. 8; even when there are errors in the priori knowledge of the robot's inertial properties, policy perfor- mance is similar to runs with no erros.. Our external torque estimators (Section 3. learn to predict forces to compensate for these errors, resulting in the similar performance.
2000
No errors
Modeling errors
1500
Reward
1000
10000
20000
30000
40000
Steps
Figure 8: Our approach is robust to errors in priori knowledge of the robot's inertial properties.
B.4 Modeling Uncertainty
Here, examine the benefit of using the noise estimator and ensemble of models. The noise timator prevents overfitting to noise and using ensemble of models captures epistemic uncertainty
16

---

[27]. We plot the training performance over 4 random seeds for test cases: (i) using the proba- bilistic ensemble described in Section 3.1, 1, (ii) removing the noise estimators, and (iii) removing the noise estimators and the ensemble. The results are presented in Fig. 9; the highest performance is obtained with our probabilistic ensemble.
2003
Probob lotio Ensemble Nows, No Ersamble
1500
Roward
800
5000
10000
16000
20000
26000
30000
36000
40000
Steps
Figure 9: Training performance when removing the noise estimators and removing both the noise
estimators and ensemble.
B.5 Additional Simulated Terrain Experiments
To further demonstrate the versatility of our approach on varying contact surfaces, we perform addi- tional experiments simulation. Within the simulator, we vary the friction coefficient and contact time constant which determines the stiffness of contact. We perform additional training runs for friction coefficients of 0.3 and 0.8, and contact time constants of 0.06 and 0.12; the friction and time constant used for all other simulated experiments in this paper are 0.6 and 0.02 respectively. As depicted in Fig. 10, we find that our method still yields favorable performance even under varying simulated contact conditions.
0.3 friction, 0.06 contact
1500
0.8 friction, 0.06 contact
0.3 friction, 0.12 contact
1250
frictin, 0.12 contact
Reward
1000
750
500
250
10000
20000
30000
40000
Steps
Figure 10: Training performance of method with varying simulated contact conditions.
B.6 SAC Performance
State-of-the-art real-world quadrupedal locomotion results where policies are trained from scratch [32] utilized SAC-like agents which directly take actions in the environment. However, highly restricted action spaces were required to obtain stable training behavior with 20 minutes of training data. In contrast, we use action spaces which are more line with standard quadruped results [25, 29, 30]. This enables faster and more dynamic gaits, but also leads a challenging optimization problem for SAC. Here, repeat the experiment of Section 4.2, but allow the SAC training to for many samples and present the results in Fig. 11. Even though SAC now converges, takes 250 times more interaction with the environment than our method.
17

---

1500
Roward
500
lces
0
200,000
400,000
600,000
800,000
1,000,000
Steps
Figure 11: Training curves when running SAC to convergence; SAC requires 250 times more inter- action with the environment than our method.
C Simulated Benchmark Experiments
To demonstrate the versatility of our approach, we perform additional simulated experiments us- ing standard, contact-rich, benchmark environments [54] commonly used to evaluate reinforcement learning (RL) algorithms.
Experimental setup. We use the standard MuJoCo [54] environments Hopper, Walker2d, and Ant, which have been implemented as part of Brax [37]. Similar to the quadruped, each of these envi- ronments feature a floating-base robot with articulated limbs which make and break contact with the ground to produce motion. However, unlike the Gol environment, these environments lack structured controllers. Instead, the outputs from the policy are only scaled linearly before being directly applied as torques the joints. To test Hypothesis 1 and Hypothesis 2, compare semi-structured approach trained with multi-step loss (Hf 4) to the black-box approach from Section 4.2 trained with the single-step loss (H 1). In both of these cases, the agent acts determin- istically within the environment per Algorithm We also benchmark against SAC [33], allowing the agent to act stochastically in the environment for this algorithm only. The hyperparameters used for training are found in Appendix D and all runs are repeated for random seeds.
Results. The results of these experiments presented in Fig. 12. We observe significant per- formance improvement when utilizing our semi-structured models trained with multi-step loss, compared to the black-box approach trained with a single-step loss, confirming Hypothesis I and Hypothesis 2. These results demonstrate that our approach works not only with the Gol environ- ment, also with other contact-rich environments with unstructured controllers.
Hopper
Walker2d
Ant
4000
3000
3000
Reward
Reward
400
Reward
2000
2000
M
1000
1000
0
40000
Steps
Steps
Steps
Ours, 4 4-stop loss
Black box, 1-stop loss
SAC
Figure 12: Simulated benchmark results. Better performance achieved when using our semi- structured dynamics models and a multi-step loss. Plots show the mean and standard deviation for episodic rewards.
18

---

D Experiment Hyperparameters
Table 4 contains the hyperparameters used with our approach; these hyperparameters were also used with the approach that incorporated black-box models. Table contains the SAC hyperparameters used for our approach, the black-box approach, and standard SAC.
Hyperparameter
Gol (real world)
Gol (simulated)
Benchmarks
Epochs. N Napocha
18
40
80
Environment steps per epoch, Ng
1000
Hallucination updates per epoch, K
10
000 over epochs 0
Model rollouts per hallucination update. M
400
+ 20
1 - 45
Synthetic rollout length, k
over epochs
over epochs
- 10
0 - 15
Real synthetic data ratio, r'D
0.06
Gradient updates per hallucination update, G
40
60
20
State history length, h
Multi-step loss horizon, H
4
or
Model learning rate
x 10
Model training batch size
200
Table 4: Hyperparameters for approach and the baseline approach with black-box models. y over epochs a - b denotes a clipped linear function, i.e. at epoch i, f(t) = clip(x + i= (1/
x), v)
Hyperparameter
Gol (real world)
Gol (simulated)
Benchmarks
Learning rate
x 10
3 10
Discount factor, 7
0.99
Batch size
256
Target smoothing coefficient,
10
5 x 10-3
Actor network (MLP) width x depth
512 x 2
256 x
Critic network (MLP) width x depth
512 2
256
Table SAC hyperparameters used for our approach, the black-box approach, and standard SAC,
19


Reviews:
[{"review_id": "oZDrNXQtkL", "reviewer": "Reviewer_V9T9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors train a policy that has only excess to a series of past observations of joint encoders and IMU measurements. The policy outputs the parameters of repetitive motion generator and offsets to nominal foot position.  This motion generator then outputs foot positions which via an IK solver are mapped to reference joint positions that are tracked by a PID. During training the policy is used to collect data which is then used to train an ensemble of probabilistic dynamics models that is supposed to predict the external forces (in particular contact) acting on the robot. These external force predictions (being in generalized coordinate space and being samples of the model) are then inserted into the robot's Euler-Lagrange equations to hallucinate possible motion. On this hallucinated data, the policy is trained via SAC.", "summary_of_recommendation": "Solving a difficult problem with few means is a significant feat. If the authors work on the structure of this paper, its writing style, add few additional experiments, and extend the critical discussion, than this work is **top notch**.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- **Question 1:** When determining policy parameters θ with SAC, how do you choose how many transitions to pick from Denv and Dmodel?\n- **Question 2:** How do you plan to extend this method to non-flat surface geometries and surfaces covered my movable objects in particular foliage?\n- **Question 3:** Why do you need an encoder prior to the ensemble model? Can you add an ablation study that investigates how removing the encoder/noise network/... affects training?\n- **Question 4 - Learning contact forces:** Due to the discontinuous nature of contact forces, learning these forces is extremely hard. So, I am surprised that the authors got anything tangible out of the ensemble. Clearly, the ensemble model is learning the contact force over one time step which causes some smoothing. Also the encoder might discover directions in generalized coordinate space in which your dynamics remain invariant to discontinuities (null space of $J^T$, see e.g. \"Contact invariant model learning for legged robot locomotion\" by Grandia et al). **Can you add another plot that shows the collected real-world data minus the Lagrange model without external torques, and then compare to this data to the predicted external joint torque that has been predicted by the ensemble?** I would be interested in seeing the absolute error of the ensemble errors on the whole data set and also seeing rollouts of the signal over time.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kW3CGIKozY", "reviewer": "Reviewer_bENK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a model-based reinforcement learning framework that integrates a first-principle model with a learned residual dynamics model to enhance quadruped robot locomotion. For floating-base systems like quadruped robots, controlling the un-actuated base coordinates, such as the robot's body, requires external forces. Instead of relying on traditional methods like the soft contact model or kinematic constraints, this paper introduces an approach to learn the external forces based on the observed states of the robots. The model learning process utilizes a neural network to predict the external torques applied to the robot by its environment. The neural network is trained using approximate maximum likelihood estimation. For policy training, the authors employ the Soft Actor-Critic algorithm, utilizing both real-world and simulated data to optimize the policy. The paper demonstrates the proposed approach through real-world results and simulated experiments focused on the task of tracking a forward velocity. Real-world results indicate that the proposed framework can be used to train a neural network control policy for simple quadruped locomotion. The simulation experiments show that the hybrid robot model has better simulation results than a black-box model.", "summary_of_recommendation": "The contribution of this work is minor. Weak reject.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. What is the key novelty of the proposed framework?\n\n2. Can the proposed framework be applied to more difficult locomotion tasks?\n\n3. How much data is actually required to train the walking policy? (both simulation and real-world data)\n\n4. Is it possible to train on certain types of terrain and then generalize the policy/model to unseen terrains?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kIUx5rY2qJ", "reviewer": "Reviewer_reP2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a model based reinforcement learning technique that structures the model learning problem as Lagrangian dynamics and learns just the external forces term as opposed to the whole state transition function. By exploiting the prior knowledge of the model through this structure, along with other techniques like learning an ensemble of models and using a multi-step prediction loss, allows the authors to learn locomotion skills on a quadruped from just 3 minutes of real world data. Hardware results are presented with the robot walking both hard and soft ground, as well as simulation ablation studies showing the usefulness of the semi-structured dynamics model and the multi-step loss.", "summary_of_recommendation": "Relatively incremental work when it comes to novelty, but I think it is working towards a good direction to explore in for doing learning from limited real world data. The results can be a benchmark for real world sample efficiency, showing successful locomotion learning from the smallest amount of real world data I have seen yet.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "What’s the rationale for structuring things as Lagrangian dynamics and learning the external force component as opposed to just learning some black box modification to the simulator output, whether it be to simulator parameters or just a straight up addition to the simulator state? To me it seems like the simulator’s estimate of the next state would be better than Lagrangian dynamics (simulator’s output is probably closer to the real world next state than the Lagrangian’s estimate without the external force component, so you have to learn a less difficult problem) and would probably actually be my first thought of what to do rather than this Lagrangian setup. \n\nAs the authors point out the method requires being able to differentiate through the mass matrix and Coriolis terms in order to optimize the probabilistic dynamics model. This work gets around this by using Brax, a differentiable physics simulator, but this requirement may be a sticking point for some setups which do not have this capability. How accurate of a gradient is required of this? Would something like finite differencing through a non-differentiable simulator work? Do you have any intuition on how this affects the accuracy of the learned dynamics model, and how does the accuracy of the dynamics model affect the learned policy?\n\nFor the simulation experiments, why does SAC not learn? I would expect it to still learn, just more slowly, but it seems like the learning actually collapses? For the other black box and 1-step loss comparisons I can understand it failing since a badly learned model can cause the policy learning to go bad, but SAC is taking actions directly in the environment. For example, as you reference (ref. 45) other works can learn to walk in the real world with SAC with 30000 samples. Why does your SAC baseline seem to underperform?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-rebuttal metareview**:

This paper proposes an MBRL approach for locomotion, where a residual dynamics model is learned on top of Lagrangian dynamics, and a SAC policy is trained using the learned dynamics. This method is quite novel in the field of locomotion, and justified by hardware experiments. Applying the proposed techniques in more challenging tasks (e.g., complex terrain) and showing its advantage over model-free approaches will further strengthen the paper.

----------------
**Pre-rebuttal metareview**:

Strengths:
1. The idea of using physics prior knowledge for efficient MBRL in locomotion is interesting and novel. 
2. Convincing experimental results to prove the concept.

Weakness:
1. The presentation needs improvements.
2. Some claims are not accurate such as “the external contact force is the only uncertainty”.
3. Although relatively new in locomotion, the key idea is not new in robotics.
4. Need more challenging experiment scenarios such as rough terrains. 
5. Other issues mentioned by reviewers, such as generalization, definition of “3-min data”, etc.
Decision: Accept