Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Models
Authors: Tyler Westenbroek
Jacob Levy
David Fridovich-Keil
Venue: CoRL 2023 Poster (2023)
Keywords: Model-based Reinforcement Learning
Feedback Control
Quadrupedal Locomotion

Abstract: We focus on developing efficient and reliable policy optimization strategies for robot learning with real-world data. 
In recent years, policy gradient methods have emerged as a promising paradigm for training control policies in simulation. 
However, these approaches often remain too data inefficient or unreliable to train on real robotic hardware. In this paper we introduce a novel policy gradient-based policy optimization framework which systematically leverages a (possibly highly simplified) first-principles model and enables learning precise control policies with limited amounts of real-world data. Our approach $1)$ uses the derivatives of the model to produce sample-efficient estimates of the policy gradient and $2)$ uses the model to design a low-level tracking controller, which is embedded in the policy class. Theoretical analysis provides insight into how the presence of this feedback controller addresses overcomes key limitations of stand-alone policy gradient methods, while hardware experiments with a small car and quadruped demonstrate that our approach can learn precise control strategies reliably and with only minutes of real-world data.

Enabling Efficient, Reliable Real-World
Reinforcement Learning with Approximate Plysics-Based Models
Tyler Westenbrock
Jacob Levy Aerospace Engineering University of Texas Austin jake. levy@utexas..... edu
David Fridovich-Kei
Oden Institute
Aerospace Engineering University of Texas at Austin dfk@utexas. .dd
University of Texas at Austin vestenbbroekk@@mail.c com
Abstract: focus on developing efficient and reliable policy optimization strategies for robot learning with real-world data. In recent years, policy gradi- promising paradigm for training control policies
methods have emerged in simulation. However, these approaches often remain too data inefficient un- reliable to train on real robotic hardware. In this paper we introduce a novel pol- icy gradient-based policy optimization framework which systematically leverages a (possibly highly simplified) first-principes model and enables learning precise control policies with limited amounts real-world data. Our approach 1) uses the derivatives the model to produce sample-efficiet estimates of the policy gradient and 2) uses the model to design low-level tracking controller, which is embedded in the policy class. Theoretical analysis provides insight into how the presence of this feedback controller addresses overcomes key limitations of stand- alone policy gradient methods, while hardware experiments with small car and quadruped demonstrate that our approach can learn precise control strategies re- liably and with only minutes of real-world data. Code is available at https: // //github. com CLt/CLLeeRRootte htt.f/oootislaaaeeaannnnnniigiio thsperilliiiittSttteeeeeelllll jjl
Task
Neural Network
Task Correction
Gain Connecton
Trajectory
0-
Controller
System
Task Iritial Pollcy
Learned Pollcy
Figure 1: (Left) Schematic of the proposed policy structure, the crucial element of which is low-level bilizing controller which improves smoothness properties of the underlying problem, improving learning. (Middle) Still frames depicting the approximate paths taken by car and quadruped during test-time. (Overlaid) Top-down view of the car executing two laps of around figure-8 88 before and after training.
Introduction
Reliable, high-performance robot decision making revolves around the robot's ability to learn control policy which effectively leverages complex real-world dynamics over long time-horizons.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

This presents a challenge, as constructing a highly accurate physics- based model for the system using first-principles is often impractical. In recent years, reinforcement learning methods built around policy gradient estimators have emerged as a promising general paradigm for learning an effective policy using data collected from the system. However, in current practice these approaches are often too data-inefficient or unreliable to train with real hardware data, leading many approaches to train high-fidelity simulation environments [1, 2, 3]. However, there inevitably exists a gap between simulated and physical reality, leaving room to improve policy performance in the real world. In this paper, we demonstrate how to systematically leverage physics-based model to yield highly efficient and reliable policy optimization techniques capable of learning with real-world data.
Modern techniques for policy learning generally fall into two categories: model-free [4, 5, 6, 7] and model-based [8, 9, 10, 11, 12]. Model-free approaches learn mapping from states to inputs directly from data. These approaches are fully general and can synthesize high-performance policies, but are extremely data-inefficient, In contrast, model-based approaches use the collected data to fit predictive model to estimate how the system will behave at points not contained in the training set. While these approaches are more data-efficient, model inaccuracies introduce bias into policy gradient estimators [13, 14], limiting the performance of the learned policy.
However, due to the unstable nature of many robotic systems, both of these paradigms suffer from more fundamental challenge: minute changes to the control policy greatly impact performance over long time horizons. This "exploding gradients" phenomenon [15], [16], [17] leads the variance of policy gradient algorithms to grow exponentially with the tuee-horizon and renders the underlying policy learning problem ill conditioned, making gradient-based methods slow to converge [18]. Moreover, model bias also compounds rapidly over time, limiting the effectiveness of otherwise efficient model-based approaches [13].
As shown in Fig. 1, this paper systematically exploits approximate physics-based model and low- level feedback control to overcome these challenges in policy learning. Concretely, the contributions are:
We introduce a novel framework which uses the approximate model to simultaneously design 1) a policy gradient estimator and 2) low-level tracking controllers which we then embed into the learned policy class. Using the model to construct the gradient estimator removes the need learn about the real-world dynamics from scratch, while the low-level feedback controller prevents gradient estimation error from "exploding".
Theoretical analysis and illustrative examples demonstrate how we overcome exponential dependencies in the model-bias, variance and smoothness policy gradient estimators.
We validate our theoretical findings with a variety of simulated and physical experiments, ultimately demonstrating our method's data efficiency, run-time performance, and most importantly, ability to overcome substantial model mismatch. Overall, this paper suggests new holistic paradigm for rapidly fine-tuning controllers using real-world data.
2
Related Work
Broadly speaking, there are two possible sources bias when using a model for policy gradient estimation. The first source of error can arise if the model is used simulate or 'hallucinate' trajectories for the system which are then added to the data set [13, 19, 20, 21]. While this approach yields a larger training set, it also introduces bias the trajectories generated by the model can rapidly diverge from the corresponding weal-world trajectory. To overcome this source of error, number of works [14, 22, 23] have proposed policy gradient estimators which 1) collect real- world trajectories and 2) use the derivatives of a (possibly learned) model to propagate approximate policy gradient information along these trajectories. We adopt this form of estimator in this work, and note strong connections to the updates used in the Iterative Learning Control literature [24].
Evaluating the gradient along real trajectories removes the first source of error. However, inaceu- racies in the derivatives of the model lead to second source of error and, as we demonstrate in

---

Section 5, these errors can grow exponentially over long time horizons for unstable robotic sys- tems. Moreover, prior works have demonstrated that exploding gradients lead to a large variance for policy gradient estimators and ill-conditioning in the underlying policy optimization problem [15]. [16), [17]. We demonstrate how low-level feedback control can overcome this second source of error, while reducing variance and improving conditioning. While the use of hiearchical control architectures with embedded low-level feedback has been key ingredient in many sim-to-real rein- forcement learning frameworks [25], [26]. [1], we argue that the combination of the aforementioned pieces opens the door for a new real-worle training paradigm that fully leverages our approximate physics-based models.
3 Problem Formulation
We assume access to a simplified, physics based model of the environment dynamics of the form:
T++1 = F112, we);
(1)
where T2 E X C R is the state, Ue E u C RTm is the imput and the (potentially nonlinear) map F: X x 4 - X' determines how the state evolves over discrete time steps t E N. To make the modelling process and down-stream controller synthesis tractable, such models are necessarily built on simplifying assumptions. For example, the model we use to control the RC car in Fig. neglects physical quantities such as drag and motor time-delays. Nonetheless, such models capture the basic structure of how controller inputs will affect desired quantities (such as position) over time, and are highly useful for designing effective control architectures.
Although many reinforcement learning frameworks model the environment as a stochastic process, to aid in our analysis, we will assume that the real-world dynamics evolve deterministically, accord- ing the (possibly nonlinear) relation:
T++1 F(z,, ue).
To control the real-world system, we will optimize over a controller architecture of the form u of (ax)) where T² [= t==
T-1 represents the overall policy, T < is the finite horizon for the task
we wish to solve, 0 E θ C RP is the policy parameter, and each map nf. X + U is assumed to be differentiable in both x and 0. Thus equipped, we pose the following policy optimization problem:
T
max [(0) := BEB
where Jr(0; = Σ R(xe).
c=0
Here, D is the probability density of the initial state xo and R is the (differentiable) reward function.
4 Approximating the Policy Gradient with an Imprecise Dynamics Model
In this section we demonstrate how to calculate the policy gradient by differentiating the real-world dynamics map F' along trajectories generated by the current policy. We then introduce the estimator used in this paper, which replaces the derivatives of F with the derivatives of the frrstpprinippls model F. We will initially focus on the gradient V [r:; mn) of the reward experienced when un- rolling the policy from single initial conditon Zo X, and then discuss how to approximate the total policy gradient J (0) using batch estimator. To ease notation, for each zo & and 0 E 0 we capture the resulting woald trajectory generated by T via the sequence of maps defined by:
2 = (oo (2o). oं (of (zo))). (x0)
Structure of the True Policy Gradient: Fix an initial condition Vg E D and policy parameter 0 E O. We let {#r)== and {u (with x, = of (zo) and u, = कt (zx)) denote the correspond- ing sequences of states and inputs generated by the policy n². The policy gradient captures how changes to the controller parameters will affect the resulting trajectory and the accumulation of fu- ture rewards. The following shorthand captures the closed-loop sensitivity of the state and input to changes in the policy parameters: the derivatives of the dynamics, which we denote with:
drs 9 06 ol (It),
dus
o6 T. (0/ (za)) These terms depend on
DB
θ A dr
θ F(F,, ut). B. du
F ut),
K, dr क (2::
(4)
3

---

Proposition 1. The policy gradient is given by the following expression:
T
Oz.
(10
= v((a))
where
(5)
00
0x. 00 Σ
Out
(-1
Lie By 00
II
AP and A. = A, + B,K.
s-1+1
For proof the result see the supplementary material. The first expression in (5) calculates the gradient in terms of the sensitivities dz, while the latter expressions demonstrate how to compute this term using the derivatives of the model and policy. In (5) the term P captures how perturbation the policy at time t' and state Zt' propagates through the closed-loop dynamics to affect the future state at time > t'. As we investigate below, when the robotic system is unstable these terms can grow exponentially large over long time horizons, leading to the exploding gradients phenomenon and the core algorithmic challenges we seek to overcome.
30
Approximating the Policy Gradient Using the Model: We approximate the policy gradient VoJr(0: (0; =0) using the approximate physics-based model F in (1). Holding zo E X. O, and the resulting real-world trajectory (are)= (we)z=) fixed as above, denote the derivatives of the model along this trajectory
① flz,, uc).
A. F u.), dx
B =
(6)
du
We can then construct estimate for (0; of the form:
dz,
VoJr(0; = (.())))
where
00
t=0
Ou Σ
d Br
II
Ag
and Aç" = A. B.K..
00
e=1+1
Remark Note that this estimator can be evaluated by 1) recording the real-world trajectory which arises when policy x is applied starting from initial state To, and then 2) using the derivatives of the model F to approximate the derivatives of the real-world system along that trajectory. Effectivvey., the only upproximation here is of the form &1,y By Rs &e. Be when calculating the estimate of the
system sensitivity Ozs Ozz In Sections 5 and 6, we study what causes this approximation to break down over long time horizons, and how properly properly-structuse structured feedback controllers can help.
Remark 2. While the policy gradient approximation given by (7) will prove convenient for analysis, this formula requires numerous 'forwards passes' to propagate derivatives forwards in time along the trajectory. As we demonstrate in the supplementary material, in practice this approximation can be computed more efficiently by propappropggtting through time'.
Batch Estimation: To approximate the gradient of the overall objective VJ))), we draw initial conditions {= independently from the initial state distribution D, compute each approximate gradient V., an)) in (7), and finally compute:
1 N
VJ))) gA (0: (z)
==
(8)
We use this estimator in our overall policy gradient algorithm, which is outlined in Algorithm 1.
Exploding Gradients: Key Challenges for Unstable Robotic Systems
We now dig deeper into the structure of the policy gradient and our model-based approximation. We repeatedly appeal to the following scalar linear system to illustrate how key challenges arise:
Running Example: Consider the case with true and modeled dynamics given respectively by:
Tt+1 F(I,, uz) az, + but and T++1 F(x,, ue) âx, + buse
(9)

---

Algorithm Policy Learning with Approximate Physical Models
1: Initialize Time horizon T & N, number of samples per update N N, number of iterations N-1 and initial policy parameters 01 E θ
K E N, step sizes {ag k=0 K do
2: for iterations =
3: Sample N initial conditions {aj DM
4: for 1, , do
5: Unroll { (zt)) on (2) with o.
6: Estimate g3 (0x) using (8) and trajectories (a M==1
7:
Update 0x+1 = 02 + ax (0)
where a, à, b, > and Ty; ur RR Suppose we optimize over policies of the form u, = (xxe) Ue where B (Tu, U1..... -1) I are the policy parameters. In this case, the policy param- eters (ūx = specify sequence of open-loop control inputs applied to the system. Retaining the conventions developed above, along every choice of (o and the resulting trajectory (=)) we have A, 0, B, b, A, à, B, b and K, : = à When a, â > 1, the system (and model) are passively unstable [27, Chapter 5], and small changes to the policy compound over time, as captured by and || and || growing exponentially with the difference - t', along with the formula for the gradients (5).
and thus we have Dt,'' a'-t'-1 and
5.1 Exploding Model- Bias
Recall that the aforementioned estimator for V Jr za) only introduces error the term az. Oz.
de 09
and in particular &e,r By = &.. By along the resulting trajectory. We will seek to understand how the point-wise errors in the derivatives of the model A Ag Aç and AB, B. B. propagate over time. Towards this end manipulate the following difference:
D1. - D.rBBr Dr,e Br + Ar Br - - Br = t1. AB. + A By (10)
- B +
1 Ber
Σ 8=1+1
The last equality in (10) provides a clear picture of how inaccuracies in the derivatives of the model are propagated over time. For example, when approximating Lr. = Di the error A BB is magnified by Pr,r, while the error A is magnified by 2r,'+11
Running Example: Continuing with the scalar example, in this case we have AB, = b b and a. Moreover, using the preceding calculations, we have & Br. - e Br = a'-r'
A = à a.a.
b) + - a -ib(- a). Thus, when a, à > and the system is unstable, the errors in derivatives of the model are magnified exponentially over long time horizons when computing the sensitivity estimate
20
xs
de 20
and ultimately the gradient estimate Jrr s (
5.2 Exploding Variance
We next illustrate how unstable dynamics can lead our batch estimator OAN to explode over long time horizons 7 unless a large number of samples N are used.
Running Example: Consider the case where r(z)) = - 2 1| ll2 and the initial state distribution is D uniform over the interval [-1, 1]. Consider the case where we apply 0 = (Mz w)-1) = (0, (o....,0) ,0) so that no control effort applied. In this case, for every initial condition Fo. the resulting state trajectory is given by It = a'xo. and thus our estimate for the gradient is (er(6:zo) ET- (a'zo) Et=1 -t Moreover, by inspection we see that the average of the estimator is Eig (0; (2o) N
= E EN [r(0):0)] = and thus the variance of the estimator is 1.2 El (0; (aol E E Jr(0; 20)]1 1. EEll (0; MPP?
+T (a'xo) Σ a'-t'bl
quantity which grows exponentially with the horizon T > 0.
5

---

5.3 Rapidly Fluctuating Gradients
Let f: R + R be potentially non-convex and twice differentiable objective, such as the ones considered this paper. In this general setting, well-established results for gradient- characterize the rate of convergence to approximate stationary points of the underlying objective, namely, points 2 E IRY such that [|V](=)]2 < < for some desired tolerance e > A key quantity which controls this rate the smoothness of the underlying objective, which typically character- ized by assuming the existence of constant L > 0 such that [V) f < L|| 22 ll for each 21, #2 E IR9. When the constant L is very large, the gradient can fluctuate rapidly, and small step-sizes may be required to maintain the stability of gradient-based methods [18], slowing the rate of convergence for these approaches. Many analyses control these fluctuations using the Hessian of the objective by setting L == supzer# [ where || l|s.2 is the induced 2-norm.
methods
Below, our main theoretical results will bound the magnitude of 2 J(0), characterizing the smooth- ness of the underlying policy optimization problem and illustrating the benefits of embedded low- level controllers. We demonstrate how to derive expression for the Hessian in the Appendix, but provide here a concrete illustration of how it can grow exponentially for unstable systems:
1 311r,118 is applied to our example scalar system. For every initial condition Zo and choice of policy parameters 0 (i1,-..,, i1- by inspection we have It = a'uro + Σi= a' that the overall ob- jective is concave and given by J(zu; = Ei= Σx= - + -===oa=
Running Example: Consider the case where the quadratic reward r(æ,)
The Hes- sian of the objective can be caleulated directly; particular the diagonal entries are given by 188
lăw? la grows exponentially in time horizon. From the discussion above, this implies that policy gradient methods will be very
82
Σi ab This demonstrates that (x 0)1 2
Bu?
slow converge to optimal policies.
Embedding Low-Level Feedback into the Policy Class
We now demonstrate how we can overcome the these pathologies by using the model to design stabilizing low-level feedback controllers which are then embedded into the policy class.
Running Example: Let us again consider the simple scalar system and model we have studied thus far, but now suppose we use the model to design a proportional tracking controller of the form Ue = k(ā; æt), where {5x = represents desired trajectory we wish to track and k 0 is the feedback gain. We then embed this controller into the overall policy class by choosing the parameters to be 0 = (Zo. Zi....,... E2) so that u, = - = k(ā; zz). Here, the parameters of the control policy specify the desired trajectory the low-level controller is tasked with tracking. In this case, along each trajectory of the system we will now have A₁' = a - bk, Aç = bk, B, = b and B; = b. If the gain k > 0 is chosen such that la bkl < 1 and J - ik < 1, then the transition matrices L = (A ( and Lt,,' = (A will both decay exponentially with the difference model we have reduced the sensitivity of trajectories to changes in the controller parameters.
Thus, by optimizing through a low-level tracking controller designed with the
Remark 3. In practice, we may select a control architecture as in Fig. where our parameters are those of a neural network which corrects a desired trajectory and low-level controller. The natural generalization of the damping behavior displayed by the proportional controller above is that the low-level controller is incrementally stabilizing, which means that for every initial condition No and 0 θ will have | || Ma'-t' There are many systematic techniques for synthesizing incrementally stabilizing controllers using dynamical model from the control literature [27, 28).
We now ready to state our main result, which demonstrates the benefits using the model to design the policy gradient estimator and embedded feedback controller:
Theorem 1. Assume that 1) the first and second partial derivatives of Re. T.. F and F are bounded,
2) there exists a constant A > 0 such that for each Do X and E U the error in the model
derivatives are bounded by max (1) dz
& F F(x, u) oz F(zz, w)ll. 2 F(x, u) Du F(z, w)ll < A and

---

3) the policy class { loee has been designed such that exists constants M. a > O such that for each To E X, A E O and > t' we have: max ( 11 < Ma -t Letting 9r
Elg (#; (mb))=)) )l denote the mean of our gradient estimator, there exist scalars C, C,W, > such that the bias and variance of our policy gradient estimator are bounded as follows:
CT A CT
if > ifa = 1 ifa < 1
WT4
ifa>
V.J -or(0))
11 (0) 91 (0)11)
S
WT
i/=
CTA
WT
if < 1.
Moreover, the smoothness of the underlying policy optimization problem is characterized via:
&T
if <a>1 if==1
J (0)112 -
KT
KT 1.
Proof of the result can be found in the supplementary material. The result formalizes the intuition built with our example: when the system passively unstable (and we can have a > 1). the core al- gorithmic challenges introduced above can arise. However, embedding a (incrementally stabilizing) low-level tracking controller into the policy class can overcome these pathologies (a S 1). Note that the condition max [| stabilizing controller (which has been designed for the model) is stabilizing the real-world system. This is reasonable condition, as under mild conditions stabilizing controllers are known to be ro- bust to moderate amounts of model uncertainty [27]. However, it is an interesting matter for future work to characterize the amount of model-mismaal our approach can handle without model-bias exploding over long time horizons.
@ ||, || < Ma'' in the statement of Theorem requires that the
7 Experimental Validation
For each experiment we use policy structure per Fig. I which embeds low-level feedback that aims to stably track reference trajectories; a formal definition this structure is given in Appendix B.
NVIDIA JetRacer: We begin by hardware-testing our approach on an NVIDIA JetRacer 1/10th scale high-speed car using the following simplified dynamics model:
F++1
Ix + 2i cos (0x) Ar
y++1
+ 24 sin (dx) At
(11)
0:+1
a,At
_d+11
di Dew,
where At > O the discrete time-step, (t,, E SF(2) are the Cartesian coordinates and heading angle of the car, 2 > is the forward velocity of the car in its local frame, and (w,,we) e [0, 1] [-1, 1] are the control inputs where at the throttle input percentage and wi is the steering position of the wheels. We note that this model makes several important simplifications: (i) drag is significant
the actual car, but missing from (11); (ii) proper scaling of the control inputs we) has been omitted; (iii) the actual car has noticeable steering bias, and does not follow straight line when We = 0: and (iv) physical quantities such as time-delays in the motor are ignored.
The task consists of tracking figure-8 made up of two circles, meters diameter, with nominal lap time of 5.5 We implement backstepping- bbased based tracking controller [27, Ch. 6] for low- level control. As shown in Fig. this controller alone does not ensure accurate tracking, due to inaccuracies in the model used to design it. We train policy with 2.2 min of real world data over 8 iterations, each 16.5s long and see clear improvement in tracking performance (Fig. 1).
Next, we use a high fidelity simulator of the car to benchmark our approach against state-of satte-of-the-nn the-art art reinforcement learning algorithms in Figure All methods optimize over the feedback control ar- chitecture described above and therefore were trained using the same action space our approach. We compare the model-based approaches MBPO [13] and SVG [22] and the model-fres ap- proaches SAC [8] and PPO [9]. Each of these approaches learns about the dynamics of the system
7

---

-3300
VEPO
55C
-2000
-2500
Task
Intall Palkc
-3200
Learned
(m)
Figure 2: (Left) Training curves different algorithms applied a high-fidelity simulation model of an car. (Right) One lap of the quadruped around the figure-8 task with corrected waypoints from a neural network.
from scratch; thus, it is unsurprising that our approach converges more rapidly it exploits known physics represented by the model. The use of feedback enables us to take this approach and ob- tain a high-performing controller, even though the model we use is highly inaccurate, overcoming model-bias. Additional details for the benchmark experiment can be found in Appendix B.
We also replicate the figure-8 tracking experiment on Unitree Gol Edu quadrupedal robot demonstrate the effectiveness of our approach when using highly simplified model. The Go1 is f8-degree-offfreedom system which control in hierarchical manner. At the lowest level, joint control module generates individual motor torques to actuate the robot's limbs to desired angles and velocities. At the next layer, a kinematic solver converts desired foot placements to joint angles. A gait generation module determines trajectories of foot placements from high-level linear and angular velocity commands issued to the robot. We provide these high- level commands to the Gol via Unitree's ROS interface [29], as outputs from backstepping-basse controller that was formulated using the following simplified dynamical model of the system:
Gol Quadrupedal Robot:
T++1
Ie cos (d+) At
M++1
yı sin (dx) At
(12)
0141
di + Ai
where (Ire 31) are the Cartesian coordinates of the base of the robot on the ground plane and di is its heading. The inputs the model the desired forward velocity 1t and the desired turning rate wy, Note that this is extremely simplified model for the system, with dynamic structure similar to the model for the car used in the previous example. Setting a nominal lap time of 37.7 we trained the policy using min of real-world data over iterations, each 50.9s long. Even though we used a highly simplified model for the dynamics, we again see a clear improvement in performance after training (cf. Fig. 2).
Limitations
Our approach successfully learns high-performance control policies using limited data acquired on physical systems. A key enabler to this end is the embedding of stabilizing low-level feedback within the policy class and the use of an priori physics-based model. However, there several key limitations. First, for situations such contact-rich manipulation, it may not be clear how to design controller with the required (incremental) stability property or that can incorporate neces- sary perceptual observations. Future work may address this limitations by incorporating techniques for learning stabilizing controllers (e.g., the Lyapumov methods of [30, 31]) or by working with la- tent state representations learned from vision modules. Additionally, while our method is highly sample-efficient, it does not take advantage many established techniques from the reinforcement learning literature, such value function learning and off policy training, leaving many directions for algorithmic advances. One particularly interesting direction is to combine the proposed approach with emerging modell-basdd reward shaping techniques [32, 33].

---

References
[1] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 281 1-2817. IEEE, 2021.
[2] J. Dao, K. Green, H. Duan, A. Fern, and J. Hurst. Sim-to-real learning for bipedal locomotion under unsensed dynamic loads. In 2022 International Conference on Robotics and Automation (ICRA), pages 10449-10455. IEEE, 2022.
[3] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 1004.00333.. 2018.
[4] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv: 1803. 1347, 2018.
[5] A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018
[6] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models, 2018.
[7] M. Deisenroth and C. F. Rasmussen, Pilco: A model-based and data-efficient approach to pol-
icy search. In Proceedings of the 28th International Conference on machine learning (ICML- 11), pages 465-472, Citeseer, 2011.
[8] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep reinforcement learning. arXiv preprint 1rXiv:1812.11 2018.
[9] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR.
[10] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmlr, 2014.
[11] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
[12] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Frez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[13] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[14] P. Abbecl, M. Quigley, and A. Y. Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pages 1-8, 2006.
[15] I.. Metz, C. D. Freeman, S. Schoenholz, and T. Kachman. Gradients not all you need. arXiv preprint arXiv:2111.05803 2021.
[16] H. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? In International Conference Machine Learning, pages 20668-20696. PMLR, 2022.
[17] Parmas, C. E. Rasmussen, J. Peters, and K. Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pages 4065-4074. PMLR, 2018.

---

[18] D. P. Bertsekas and J. N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization, 10(3):627-642, 2000.
[19] R. S. Sutton. Integrated modeling and control based reinforcement learning and dynamic programming. Advances in neural information processing systems, 3, 1990.
[20] Buckman, Hafner, G. Tucker, Brevdo, and Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
[21] S. Gu, Lillierap, I. Sutskever, and S. Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pages 2829-2838. PMLR, 2016.
[22] N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa. Learning continuous control policies by stochastic value gradients. Advances in neural information processing systems, 28, 2015.
[23] N. Amann, D. H. Owens, and E. Rogers. Iterative learning control using optimal feedback and feedforward actions. International Journal of Control, 65(2):277-293, 1996.
[24] H.-S. Ahn, Y. Chen, and K. L. Moore. Iterative learning control: Brief survey and catego- rization. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 37(6):109--11211 2007.
[25] Lee, Hwangbo, I.. Wellhausen, Koltun, and Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020.
[26] T. Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, x(62)eeb 2022.
[27] S. Sastry, Nonlinear systems: analysis, stability, and control, volume 10. Springer Science & Business Media, 1999.
(28) I. R. Manchester and J.-J. E. Slotine. Control contraction metrics: Convex and intrinsic criteria for nonlinear feedback design. IEEE Transactions Automatic Control, 6246):3046,3053, 2017.
[29] U. Robotics. A1. URL https:://ww.uuiirreee htt.tttts//////...ttttcc.ooooooo
com/productss//l//
[30] J. Z. Kolter and G. Manek. Learning stable deep dynamics models. Advances in neural infor- mation processing systems, 32, 2019.
[31] H. Ravanbakhsh and S. Sankaranarayanan. Learning control Ιyapunov functions from coun- terexamples and demonstrations. Autonomous Robots, 43:275-307, 2019.
[32] Westenbroek, Castañeda, A. Agrawal, S. Sastry, and K. Sreenath. Learning min-norm stabilizing control laws for systems with unknown dynamics. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 737-744. IEEE, 2020.
[33] T. Westenbrock, F. Castaneda, A. Agrawal, S. Sastry, and K. Sreenath. Lyapunov design for robust and efficient robotic reinforcement learning. 6th Annual Conference on Robot Learning, 2022. URL
[34] K. M. Lynch and F. C. Park. Modern robotics. Cambridge University Press, 2017.
[35] R. W. Beard. Quadrotor dynamics and control. Brigham Young University, 19(3):46-50) 2008
10

---

A Proofs
This appendix contains proofs of claims that were omitted in the main document and several support- ive Lemmas. Section A.I provides the derivation for Proposition 1, Section A.2 states and formally derives the reverse-time time representation of the gradient, while Section A.3 builds on this calculation derive representation for the second variation, which is subsequently use bound the Hessian. Finally, Section A.5 contains the auxiliary lemmas.
A.1 Proof of Proposition 1
The expression for (ro, 0) follows directly from the chain rule. To obtain the expression for dhy we differentiate the dynamics It+1 = F ue) to yield:
06
0xt+1 = F(x u4)
da
Our 00
F(x,, ud)
00
dr
00
du
A dr:
dm?
Bt
00
where the second equality is obtained by noting that:
du:
Zmf dr θगß
Or. Ox 00
+ dr
+ K₁
00
00
00
00
The desired expression is then obtained by unrolling the recursion and noting that
00
A.2 Efficient Backwards Pass for Policy Gradient Computation
While the form for the policy gradient (5) and model-based approximation in (7) will prove convenient for analysis, computing the many approximate sensitivity terms du; and in particular the Lr terms-is highly complex and requires many forwards passes along the trajectory. In practice, we can more efficiently compute the approximate gradient as follows:
00
Proposition 2. For each xo E X and E θ the policy gradient can be calculated via:
T-1
dm
(1:(0:20))
Σ (per1 B, (R((
where
(13)
00
m
Pt P (A, + B.Ki) + (R())) and pr V RT(()).
(14)
Here, the recursion with the variables E R1xn performs 'back propagation through time' along the real-world trajectory using the derivatives of the model.
Proof. As before, let {xe)T=o and {uo)To) denote the state trajectory that results from applying the policy No from To.
Permitting slight abuse of notation, we can re-write the cost by moving the dynamics constraints into the cost and weighting them with Lagrange multipliers:
J(0 Σ R((x.) P1t1 F(Z,, *b(1))
(15)
1=0
Define the Hamiltonian
He P++1, Pt+1 D my (xx)) Rx(),),
(16)
and note that we may then re-write the cost as:
T-1
[0 = Rr(eT) + (pT, (T) (po, xo) +
Pett - He((s,,t41,,)
(17)

---

To reduce clutter below we will frequently omit the arguments from Hj. since it is clear that the map is evaluated at (Iz, Let 60 E Rp be a variation on the policy parameters and let da; 30
ddž 50 denote the corresponding first variation of the state. To first order, the change in the
cost corresponding to these variations is:
8J|n(60) (0)) = (VQr(ET) + pr, 8XT) + Σ (pr V V2H1, - (VoH,,
(18)
To simplity the expression, let us make the following choices for the multipliers:
pr (T((FT)
(19)
Pt
T V VIHe
(20)
Pr-1 dx
F(x, mt( (()
(21)
A (x4)) = P dx
(22)
where we have applied the short-hand from developed in Section 3 for the particular task. Plugging this choice for the multipliers into (18) causes the dæ, terms to vanish and yields:
6J)b(60) Σ Vo Voll,00 50)
(23)
OTh (.))
(ptz1 Ou
F(x, mz)
58)
(24)
00
00
T-1
0mh 80)
Σ
(rt+1 Bg + rts
(25)
a
Since this calculation holds for arbitrary 50 this demonstrates that the gradient of the objective is given by:
T-1 VoJ(O, = Σ
Ozf 00
(P41 Br + rt.
(26)
t=
A.3 Calculating the Second Variation
To calculate the Hessian of the objective we continue the Lagrange multiplier approach discussed above. Now let 5²xz, denote the second order variation in the state with respect to the perturbation 60. By collecting second order terms in (17) the attendant second-order variation to the cost is given by:
(3,1o(50) ( 22 RT (XT) 82x) (Vr((TT) + pr, pr, 8'TT)
(27)
(pt 622 (6x, V (x.), 6xx)
+ 2(S u H4, 80) V Hr 50)
(28)
By using the choice of costate introduced above, this time the second order state variations 8²x, vanish from this expression so that we arrive at:
22, ())((60) = + Σ fox 22_HHHHH H,(x,,, 6xx) 2(2x, V2 H,, 80) + (50 V H, 50).
(29)
t=
12

---

By unravelling this expression, we observe that:
dur
dUT
V Jr -T;
22
(T
08
00
T
dzy
0²
dx
+
He(
00
dr²
00
)T
+ 2
Hi Ptt,,,
00
Ord0
1=
0²
He(z,, Pt+
002
which, for the purposes of our analysis, we note does not depend on second variations of the state.
A.4 Restatement of Main Result and Proof
Theorem 1. Assume that the first and second partial derivatives of R2, " F and F are bounded. Further assume that there exists constant A > 0 such that for each xo E X and E U the error in the model derivatives are bounded by max # F(z. u)ll. ll a F(z, 4) < A. Finally, assume that the policy class of has been designed such that exists constants M, a > 0 such that for each To E X, E O, and t > we have: max ( Il. || < Ma Then we may bound the bias and variance of our policy gradient estimator as follows:
3hr
CT CT
i > i < 1,
WT 27
if 1 i
||V+(00) θr (0)1 <
E | (0)
(0)
CTA
if
i
Moreover, the smoothness of the underlying policy optimization problem is characterized via:
if "f
KT
o if 1.
KT
Proof. We first bound the bias of the gradient:
[[V Dr (0) = ||l[ Jr (0; (0; J (0;
sup (0; 2o) (0;
where the preceding expectations are over ~ D. The desired bound on the bias directly follows by applying the bound on gradient errors from Lemma 2 below.
Next, bound the variance estimate note that:
Elll (0) (0)1
1 Σ N
i=1 Elllgr 2
< N sup 0+( 2o) - (
N sup (0;
where the first expectation is over (x DA the second is with respect (xo) ~ D. The desired bound on the variance follows via direct application of Lemma which provides uniform upper- bound on the gradient estimates.
Similar to before we have:
V Jr(())
< ((z))~ (f; < sup 2 -r.(0;
(D))D
The desired bound follows from Lemma 3, which uniformly bounds the task-specific Hessians.
13

---

A.5 Supportive Lemmas
Lemma 1. Let the Assumptions of Theorem hold. Then there exists B > O independent of the parameters T' E N, M and a E R such that for each 2o E D and I E θ we have:
if
VoJr(0; 1
$1
fa=1
BT
fa<
Proof. Let the constant > be large enough so that upper-bounds the norm of the first and second partial derivatives Re and F. Fix a specific task zo and set of policy parameters and let Ar, B2, K₁ be defined along the corresponding trajectory as usual.
Recall from Section 3 that
dmp
(0;
Σ
(pet1 (e41BB+ B VR(az))
00
==
where the co-state Pr IR 1xn is given by:
T-1
Σ
R((x))
D..t
s-t+1
by inspection. Thus, we may upper-bound the growth of the co-state as follows:
T-1
Ipp:1 & T + Σ (L+
(
(30)
8=
By carrying out the summation, we observe that there exists C > sufficiently large such that
if > 1 CiTa"
||el|| v
i = 1 CTT
(31)
C
if 1,
where we have used the fact that ET-1 Ma
M 1-a for the third case. We can bound the
overall gradient follows:
(0:20))))
Σ L(Llppeall + L), 1=
(32)
which when combined with the bound on the costate above demonstrates the desired result for some constant 3 > 0 sufficiently large to cover all choices of xg.
Lemma 2. Let the Assumptions of Theorem hold. Then there exists C independent of N, we have:
M, A. AB > and a >> Osuch that for each No E D and
CT T A
fa>1 1
-o 201
CT
fa
CT²A
f
where A = min((A,, AB}.
Proof. Let the constant > be large enough so that upper-bounds the norm of the first and second partial derivatives of RG, af. F and F. Fix specific task 2n and set of policy parameters and let Ar, B,, K2 as usual.
14

---

Using equations (7) and (10) we obtain:
I
[.Jr(0;z0)) (0, z0)1 =
vR(x:)
1 Be - 21 By)l
1=
[R(x))ll
Σ
| AB +
Σ $ A 1,1) Br
x
1==0
T
Ma A+
Ma
Σ
Ma AMa
Ma
t=1
P=0
S=++1
Note that the preceding analysis holds for any choice of 0 and zo. Thus, noting that
Σ
Ma---AMo
M 1-a A
8=/+1
in the case where a < 1, leveraging the preceding inequality we can easily conclude that there exists C > 0 sufficiently large such that for each and Ty we have:
CT
ifa
VoJr(0; (r; I
CT
if
CTT2
if <
which demonstrates the desired result.
Lemma 3. Let the Assumptions of Theorrem hold. Then there exists K 0 independent of N, M and E R such that for each zo D and E O we have:
3T
ifa >
IV/JT(0;x0O)1
<
KT
ifa
KT
i <.
Proof. Let the constant >> be large enough so that it upper-bounds the of the first and second partial derivatives of RG, Nf, F and F. Fix specific xo and set of policy parameters 0. Recall from that the Hessian can be calculated as follows:
OET T. (2Rr(ET)
Oet
2 (0;
00
00
dz
0
dr.
H. Pc.
09
Or
00
dx,
82
+ 2
H (.e. 0)
3ra0
T 82
+
He P).
302
Using the assumptions of the theorem, we observe that there exists constant C > sufficiently large such that
82 Or22
8² 0r00
82
max
H. H. Pt, 0),
H (x 0)
C1 +++ (33)
0x00
and
IV Jr x0)
L||
dxr
1|2
Σ
C1 [1 + +1][1
Ox
112
+
Or:
+
(34)
00
00
00
holds for all choices of and 0.
15

---

Using our preceding analysis, we can bound the derivative as the state trajectory as follows:
dxz
1|
Be
00
00
d==0
P==0
This demonstrates that there exists C2 sufficiently large such that:
C2Ta"
if >
dx 00
S
C2T
i <1,
(35)
C2
where in the case where a < 1 we have used the fact that Er Ma''' < M 1. previous bounds (33), (31) and (34) then demonstrates the desired result.
Combining the
I-a
B Additional Experiments and Details
Here, provide additional simulation experiments and details for experiments presented in the main paper.
B.1 Policy Structure for Experiments
Per Section for each experiment, we construct our policy (Fig. 1) around a low-level controller :: X' X x I& U, which produces control inputs u, u( ddes G+) to stably track ref- erence trajectories {-d=s]T Jt= with controller gains G2 R$ A task th: R reference trajectory zdess = (t()). however tracking is poor due to mismatch in dynamic model- ing when forming the controller. To this end, a neural network NN: R x X + R x R= with parameters E θ generates corrections to the controller gains and to the reference trajec- tory (AG,, Am(e)) = NN (E(), ( encodes task objectives. Our ultimate policy class is of the form (t, z4; G) p(a,, (t) + ())) G (E((), xx)) where G is nominal of feedback gains. Unless otherwise specified, the neural network is 64 64 multilayer perceptron with tanh(-) activations. For each of the benchmarks in Appendices B.4 to B.6, all methods use the same low-level feedback con- troller (as described in their respective sections) and the policy structure described in this section. Therefore, all methods were trained using the same action space.
X' produces the
= NN(()E( 21), where ę: R + RP
B.2 The Benefit of Low-Level Feedback
In this experiment, we compare the policy class of Fig. 1 against policy class in which neural network directly determines open-loop control inputs (as in Section 5, omitting a low-level stabiliz- ing controller). We use the double pendulum model from [34], and the task requires moving the end effector to desired location, using a reward function based on Euclidean distance. The following parameters were used for training: Random seeds: 64, Episodes: 50, Episode length: 300, Policy calls per episode: 30, Epochs: 50, Batch size: 5. First experiment: We provide the true dynam- ies both approaches to observe the variance and conditioning, independent of model-mismatch. Training curves for the best learning rate for each approach are depicted in Fig. 3a, which supports the main thcoretical findings. Second Experiment: Next feed Algorithm I an approximate model that contains pendulum masses and arm lengths that are 70% and 90% of the actual values, spectively. As shown in Fig. 3b, the unstable dynamics lead to significant model bias which limited the asymptotic performance the naive controller without embedded feedback controller.
B.3 NVIDIA JetRacer Actor Network Outputs
We examine neural network outputs during single execution of the figure-eight task for the NVIDIA JetRacer hardware experiment, depicted in Fig. 4. We see that the neural network issues
16

---

600
Rewadd
1000
-1000
With inarimuel lnechorkk comsriged meveet -18
With fendhark: corvenges
Without onsleval "edbacck converged reward: -000.0
Withouu low lovel loaditeck; converged rowart 415.7
1500
-1500
Without model mismatch
model mismatch
Figure 3: Training curves for the double pendulum experiment. Embedding low-level feedback results in beller performance both with and without model mismatch.
A/X (%)
X
0.0
Ki -0.1
Akyy (84
19
time (9)
Alvv (%) CIAGO
Inital Controller
Corrected task points
time (9)
Figure 4: One lap of the car around the figure-8 task before/after training and neural network outputs.
corrections on the outside of the track, which reasonable considering the untrained car was track- ing the inside of the track. note the following controller gains adjustments from the neural network: (i) an overall negative value selected for the feedforward steering gain AK,, counteracts the car's inherent steering bias in the positive steering direction; (ii) lower values of forward veloc- ity gain A K, were selected when crossing the origin, allowing the car to more closely track at this critical point; and (iii) elevated values of AK, are selected to speed up the car for the rest of the track, increasing reward.
B.4 Parameters for Simulated Car Benchmarks
Here, we report the details for the simulated car benchmark reported in Figure 2 in the main doc- ument. For each algorithm the episode length is 300 steps of the environment, and the simulation step was 0.1 seconds. For each method, a total of 10 random seeds was run and the actor network was a 64 64 feedforward network defining the splines tracked by the low-level controller. Further details for each tested method are:
Our Method: Learning rate: 0.1, Episodes per iteration: 5. MBPO: Dynamics model: 2 layer feedforward tanh network (256 x 256), Models in ensemble: 5, Learning rate: 1 x 10 10-3 Episodes
per iteration: 10, Critic Network: layer feedforward tanh network (256 x 256). SVG: Dynamics
model: layer feedforward tanh network (256 x 256), Learning rate: 2.5 x 10 3 Episodes per
iteration: 20 Critic Network: layer feedforward tanh network (256 x 256). PPO: Episodes per
iteration: 25, Learning rate: 1 -3 Critic Network: layer feedforward tanh network (256x256)
SAC: Episodes per iteration: 5, Learning rate: 1 x 10 Critic Network: 2 layer feedforward tanh
network (256 x 256).
17

---

B.5 Cartpole Simulation Benchmark
In this benchmark presented in Figure we attempt to track a desired end effector position for the classic simulated cart-pole environment. In particular, we use linearizing controller [27] to approximately track desired positions for the end effector. For each algorithm the episode length was 100, and the simulation step was 0.1 seconds. For each method, total of 10 random seeds were run and the actor network was 64 x 64 feed-forward tanh network defining desired spline parameters for the desired trajectory tracked by the low-level controller.
Learning rate: 0.05, Episodes per iteration: 5, Simplified Model: constructed by decreasing the mass and friction parameters of the true model by 50 percent. MBPO: Dynamics
Our Method:
model: 2 layer feedforward tanh network (256 x 256), Models in ensemble: 5, Learning rate: 2.5 x
10-3, Episodes per iteration: 10, Critic Network: 2 layer feedforward tanh network (256 x 256).
SVG: Dynamics model: 2 layer feedforward tanh network (256 x 256), Learning rate: 1 X 10-3,
Episodes per iteration: 20, Critic Network: 2 layer feedforward tanh network (256 x 256). PPO:
Episodes per iteration: 25, Learning rate: 10-5, Critie Network: layer feedforward tanh
network (256 x 256). SAC: Episodes per iteration: Learning rate: 5 10
Critic Network:
layer feedforward tanh network (256 x 256).
-100
200
-300
Reward
MBPO
400
SAC
PPO
500
SVG
-700
Minutes of Data
Figure 5: Training curves for different algorithms applied the cart-pole environment.
B.6
Quadrotor Benchmark
Next we conduct simulation benchmark using the quadrotor dynamics model from [35] and present the results Figure 2. The simulator timestep 0.1s and each episode is 400 timesteps. The task is follow figure-8 pattern in the air. A total of 10 random seeds were run for each method, and for each algorithm the actor network was 128 128 feed-forward tanh network, which specified the splines and feedback gains for the tracking controller from [35].
Our Method: Learning rate: 0.1. Episodes per iteration: 10, Simplified model: constructed by
decreasing the mass and friction parameters the true model by 50 percent. MBPO: Dynamics
model: 2 layer feedforward tanh network (256 256), Models in Ensemble: 5. Learning rate: 2.5 x
10-4 Episodes per iteration: 10, Critic Network: layer feedforward tanh network (256 256).
SVG: Dynamics model: 2 layer feedforward tanh network (256 256), Learning rate: 1 10
Episodes per iteration: 20. Critic Network: 2 layer feedforward tanh network (256 256). PPO:
Episodes per iteration: 25, Learning rate: 1 x 10-5, Critic Network: 2 layer feedforward tanh
18

---

-600
-800
-1000
Ours
Reward
20
60
100
Minutes Data
Figure 6: Training curves different algorithms applied to quadrotor environment.
network (256 x 256). SAC: Episodes per iteration: 5, Learning rate: 3 x 10-5, Critic Network: layer feedforward tanh network (256 x 256).
B.7 Dynamics Mismatch Study - Simulated Car
In simulation for the car experiment, we study the performance our approach as model accuracy degrades. We use the following actual dynamics model:
P++1
ir + oa cos (dx)) A
3r+1
+ vi sin (d:) At
(36)
U441
Vt - - At
0++1
L + (B-w: at
where 8a and Bw represent control input scaling for acceleration and turn rate, respectively; is the coefficient of drag; and bw represents bias in the car's steering. The set A := {B.. Bw cu2 bw parameterizes the actual dynamics of the car. We define a mismatch coefficient, 7, which scales these parameters to cause mismatch between the actual model and the model used for training. That is, we use the set B {7,3, 79 7Co, } with Eq. (36) to form our approximate dynamics model F:
F++11
+ cos (dx. Al
ye+1
Ye + sin (0x) A/
(37)
Ue+1
We (7Baa: - 7cve() 2 At
(441
de + (YBwwr 7bu) A
Note that if 1, then the models match exactly.
We perform training with various values of over 10 random seeds, each with 15 training iterations, and present the results in Fig. 7 and Fig. 8. We find that, even in cases of large model mismatch, a policy learned which improves the performance of the car.
Training Details:
19

---

-50
10
-100
0.75
0.25
5
15
20
lteration
Figure Training curves for simulated car experiment varying degrees of model mismatch.
Task
Untrained
1.0
0.75
0.5
0.25
Figure 8: One lap of the car around the figure-8 task where training was performed with varying degrees of model mismatch.
20


Reviews:
[{"review_id": "t2fs737h2th", "reviewer": "Reviewer_wqLE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the issue of model bias and variance reduction in policy gradient estimation for robot control. They provide a solution to mitigate errors and instabilities which may arise as a result. Specifically, a model-based approach for gradient estimation is presented, where real-world rollout data is used for objective/reward evaluation over a finite time horizon. This is followed by a back-propagation scheme through the dynamics to obtain the policy gradient, where the gradients of the dynamics are approximated using an analytic model. The authors examine how modelling errors and unstable dynamics affect the gradient estimate, providing ample theoretical justification. They propose to incorporate an incrementally stabilizing feedback controller to counteract these phenomena, resulting in improved policy gradient estimation. The approach is tested in simulation, against common online-RL baselines, and on multiple real-world platforms.", "summary_of_recommendation": "Recommended accept, on the condition that the comments are addressed (particularly better simulated experiments and more baselines.)", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the points listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the paper is interesting and offers a good contribution. However, some points should be addressed, which would help bring confidence to the presented results.\n\nPros:\n- Well motivated against prior work.\n- Well written and clearly presented.\n- The analysis is thorough, and good theoretical justification is provided.\n- Solution is clear and elegant.\n- Nice physical experiments on multiple platforms.\n\nCons:\n- Unclear how novel the policy gradient estimator (Algorithm 1) is compared to prior work. Apart from the feedback controller, how does Equation (8) and Algorithm 1 differ from prior work on policy gradient estimation using dynamics derivatives ? Can you make comparisons to these other policy gradient estimators, at least in simulation?\n- Providing more results in simulation using common environments (ex. Cartpole) against baselines (ex. Stochastic Value gradients, MBPO, SAC, etc.) would help solidify the results and against prior work, and help with reproducibility.\n- Provide information on number of random seeds, episodes, etc. used for simulated experiments. This would help to ascertain statistical significance and confidence in results.\n\n\nComments & corrections:\n- Code release would be highly suggested.\n- The policy is defined with possibly time-varying parameters (line 84). In the running example used to in section 4.3, you claim the hessian of the objective is block-diagonal. Could you show this?  \n- There may be circumflex symbol missing above F in the second equation in 6, since the first-principles model is used here.\n- Typo on line 171: should be “stochastic gradient”\n- Line 184: remove redundant “the”\n- line 232: remove redundant “the”", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "ifC51YX2zs", "reviewer": "Reviewer_9dCJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to data inefficiency and numerical instability of policy gradient reinforcement learning by using the domain knowledge of a feedback controller of (possibly low fidelity) model of the robot.\nThe core aspects of the methodology are (a) the use of an--inaccurate model--of the physics of the robot to help approximate the gradient of the policy model and (b) to use a low-level feedback controller (based on the inaccurate model) as a component, together with a feed-forward neural network (that computes corrected gain for the controller from the state of the system), of a learning-based control policy.\nThe method is demonstrated in real robots using a small car and a quadruped tracking a figure-8 trajectory.", "summary_of_recommendation": "The major caveats remain (1) how difficult is it to derive a model of the system and/or how inaccurate can it be for it to still yield usable results and (2) on the flip side, how much more value/challenge there is in learning a policy for a system for which we already have a feedback controller. Nonetheless, I think there is a lot of insightful observation in this paper on how to combine non-learning-based control with policy gradient reinforcement learning and the arc from theory to real robot deployment is promising.\n\nI have read the rebuttal and appreciate the authors answer, my feedback, which is positive, is unchanged.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- In Figure 1, could you add the behavior of a policy with an exact model?\n- Are there connections between the problems highlighted in section 5 and RL's deadly triad (e.g. model-bias and function approximation, bootstrapping/off-policy learning, and leveraging—or not—a known model/feedback controller)?\n- \"The policy was trained with 2.2min.\" Is this sentence complete?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper's ideas constitute an interesting and comprehensive methodology to fuse policy-gradient/RL for continuous control methods with traditional modeling of dynamical systems and the development of model-based controllers. Although it does not address the challenges that are faced when the latter two are inevitably uncertain or impossible, it is a valuable contribution that goes beyond the mere pitching of learning-based vs non-learning-based approaches as well as \"only\" model-based learning approaches.\nI would have liked to see also comparisons with the performance of increasing (from ideal to worst) model inaccuracy.\n\n- Please clarify with an example how model-based approaches \"inevitably introduce bias\"\n- Please cross-reference from the introduction the concepts of exploding gradients and exponential dependencies that are formally defined in later sections\n- The first paragraph of section 7 contains repeated information and would be improved with a formal/formula definition of the NN input and output.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "eBwHj9kiI5-", "reviewer": "Reviewer_D3r9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the behavior of gradients in the case when the forward pass is conducted on hardware and the backwards pass is conducted using some simplified approximate mode. The authors derive the error in the resulting gradient estimate and discuss factors that influence the growth of that error (particularly closed-loop stability). They show that incorporating a default-stable low-level controller beneath a high-level RL policy improves training performance.", "summary_of_recommendation": "The paper is well executed and discusses several critical difficulties in using analytical gradients for policy learning. However, these issues with gradients are well-known in the community, so the additional contribution of this paper is not clear. If there is specific novelty in the problem setting considered here (forward pass on HW, backwards pass on an approximate model), then I could consider upgrading my recommendation (although this still seems like an incremental contribution). In addition, the recommendation to embed a stable low-level controller is already a well-known and commonly-practiced technique in robotics.\n\n**Update post-rebuttal**: The authors have sufficiently addressed my concerns. I still think that the main finding is not surprising, but I also think the community would benefit from the discussion in this paper, so I will increase my score.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please answer the questions raised in the previous section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n- The presentation is clear and the authors present their mathematical analysis well.\n- The hardware demonstrations are compelling.\n\n### Weaknesses\n- The paper's first main conclusion, that the gradients can explode or have very high variance when the system is closed-loop unstable, has been studied before. See https://arxiv.org/abs/2202.00817 and https://arxiv.org/abs/2111.05803. Can you please clarify what your contribution is beyond these papers, either in terms of problem setting or theoretical analysis?\n- The other main conclusion, that policy learning can be accelerated by embedding a lower-level feedback controller along with a deep policy, also does not seem to be new. For instance, the state-of-the-practice in RL-based locomotion seems to involve using a deep policy to predict setpoints for a stable linear joint angle tracking controller. Can you clarify your contribution beyond what is commonly done in the field today?\n\n### Major comments\n- It would seem that the main conclusion of the paper is that RL can work better when the policy is closed-loop stable. Embedding a low-level policy, as the authors suggest, is just one way of doing this. Are there other ways in which you might constrain the policy class to be stabilizing?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2iARvFUCLRW", "reviewer": "Reviewer_wfR7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper analytically clarified the destabilizing factors of the policy gradient method utilizing a simplified (imprecise) dynamics model, and showed that using a proportional feedback controller as the policy is effective for its stabilization.\nThe paper also demonstrated by using two types of robots that the policy optimization can be extremely efficiently achieved.", "summary_of_recommendation": "This is a very interesting study because it can be regarded as a method that successfully follows the conventional model-based control theory.\nThe usefulness of the method has been theoretically and practically verified.\nHowever, the proposed method might not be used for arbitrary tasks with free lunch, so additional discussion would be needed on its limitations and potential for further developments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "In Fig. 2, the performance of the proposed method seems to improve when there is a model mismatch, is this an illusion due to the different vertical axes?\nAnyway, it may be useful to include the convergence values in Fig. 2 to show that the final performance of the proposed method would not depend on the accuracy of the model.\n\nI think that the model hat F that the proposed method should prepare in advance can be prepared in the same way as in conventional model-based control methods for tasks with few external factors such as this experiment.\nHowever, what about tasks with many external factors involved, such as contact-rich robot manipulation for various objects?\nIn particular, in recent RL, the exact state in which the MDP is established is not known, and redundant observations (e.g. camera images) are often used.\nIn such cases, if the proposed method is used straightforwardly, Hessian's computational cost may be too high, and it would be very helpful to have a countermeasure to take advantage of it.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The authors theoretically analyzed the reasons for the divergence of policy gradients and revealed that the proportional feedback controller can suppress the divergence.\n- By making effective use of a simple model based on first principles, the proposed method achieved the policy optimization more efficiently than conventional SOTA reinforcement learning algorithms.\n\nWeaknesses\n- There are limitations in the applicability and scalability of the proposed method.\n- Since a true model is required to necessarily suppress the divergence of the policy gradients, the stability should not always be guaranteed in practice.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies ways to achieve efficient learning on real systems by using an approximate dynamics model. A gradient estimator is defined that makes use of the model. Instabilities that may arise due to model bias are addressed by a feedback controller. The approach is tested in simulation and compared to common online-RL baselines. The authors also apply and evaluated it on multiple real-world platforms.

Issues raised by the reviewers were able to be cleared during the rebuttal and all reviewers recommend acceptance of the paper. 

One issue that is remaining is the title:  "Feedback is all you need" 
 * is not what your paper is proposing, as there are a lot of other  aspects, so it is misleading
 * the term is already overused
So I ask the authors to change the title accordingly.
Decision: Accept (Poster)