Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control
Authors: Minjun Sung
Sambhu Harimanas Karumanchi
Aditya Gahlawat
Naira Hovakimyan
Venue: ICLR 2024 poster (2024)
Keywords: Robust control
Reinforcement learning
Primary area: reinforcement learning
TL;DR: We propose a novel framework as an add-on scheme to enhance the robustness of model-based RL algorithms against uncertainties.

Abstract: We introduce $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. Unlike model-free approaches, MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed switching law. Using the approximate model, control input produced by the underlying MBRL is perturbed by the $\mathcal{L}_1$ adaptive control, which is designed to enhance the robustness of the system against uncertainties. Importantly, this approach is agnostic to the choice of MBRL algorithm,  enabling the use of the scheme with various MBRL algorithms. MBRL algorithms with $\mathcal{L}_1$ augmentation exhibit enhanced performance and sample efficiency across multiple MuJoCo environments, outperforming the original MBRL algorithms, both with and without system noise.

Published as a conference paper at ICLR 2024
ROBUST MODEL BASED REINFORCEMENT LEARNING USING L ADAPTIVE CONTROL
Minjun Sung, Sambhu H. Karumanchi; Aditya Gahlawat, Naira Hovakimyan Department of Mechanical Science & Engineering
University of Illinois Urbana- Phaaaa-haampign Urbana, IL, 61801, USA {mjsung2, shk9, gah gahlawat,
ABSTRACT
We introduce CI-MBRL, a control-thcoretic augmentation scheme for Model- Based Reinforcement Learning (MBRL) algorithms. Unlike model-free ap- proaches, MBRL algorithms leam model of the transition function using data and use it to design control input. Our approach generates a series of approx- imate control-affine models of the learned transition function according to the proposed switching law. Using the approximate model, control input produced by the underlying MBRL is perturbed by the L1 adaptive control, which is de- signed to enhance the robustness of the system against uncertainties. Importantly, this approach is agnostic the choice of MBRL algorithm, enabling the use of the scheme with various MBRL algorithms. MBRL algorithms with L1 augmentation exhibit enhanced performance and sample efficiency across multiple MuJoCo en- vironments, outperforming the original MBRL algorithms, both with and without system noise.
INTRODUCTION
Reinforcement learning (RL) combines stochastic optimal control with data-driven learning. Recent progress in deep neural networks (NNs) has enabled RL algorithms to make decisions in complex and dynamic environments (Wang et al., 2022). Reinforcement learning algorithms have achieved
remarkable performance in wide range of applications, including robotics (Ibarz al. 1021
Nguyen & La| (019). natural language processing (Wang et al., 2018: Wu et al. (018)), autonomous
driving Milz al. 2018 Li al. 2020), and computer vision YYun et al. 2017 7 Wu et al. 2017).
There are two main approaches to reinforcement learning: Model-Free RL (MFRL) and MBRL. MFRL algorithms directly learn a policy to maximize cumulative reward from data, while MBRL algorithms first learn a model of the transition function and then use it to obtain optimal poli- cies (Moerland al. 2023). While MFRL algorithms have demonstrated impressive asymptotic performance, they often suffer from poor sample complexity (Mnih et al., 2015) Lillicrap et al.) 2016 Schulman al. 2017). On the other hand, MBRL algorithms offer superior sample complex- ity and are agnostic to tasks or rewards (Kocijan et al. 2004 Deisenroth et al.) 2013). While MBRL algorithms have traditionally lagged behind MFRL algorithms in terms of asymptotic performance, recent approaches, such as the one presented by Chua et al. (2018), aim to bridge this gap.
In MBRL, learning a model of the transition function can introduce model (or epistemic) uncertain- ties due to the lack of sufficient data data with insufficient information. Moreover, real-worle systems are also subject to inherently random aleatoric uncertainties. As a result, unless sufficient data-both in quantity and quality-is available, the learned policies will exhibit a gap between expected and actual performance, commonly referred to as the sim-to-real gap (Zhao et al. 2020).
The field of robust and adaptive control theory has rich history and was born out of need to design a controller to address the uncertainties discussed above. Given that both MBRL algorithms and classical control tools depend models of the transition function, it is natural consider exploring the consolidation of robust and adaptive control with MBRL. However, such a consolidation is far
'The authors equally contributed this work.

---

Published as a conference paper at ICLR 2024
from straightforward, primarily due to the difference between the class of models for which the robustness is considered. To analyze systems and design controllers for such systems, conventional control methods often assume extensively modeled dynamics that are gain scheduled. linear, control affine, and/or true up to parametric uncertainties (Neal et al. [2044 Nichols et al. 1993). On the other hand, MBRL. algorithms frequently update highly nonlinear models (e.g. NNs) enhance their predictive accuracy. The combination of this iterative updating and the model's high nonlinearity creates a unique challenge in embedding robust and adaptive controllers within MBRL algorithms.
1.1 STATEMENT OF CONTRIBUTIONS
We propose the -MBRLL framework as an add-on scheme to augment MBRL algorithms, which offers improved robustness against epistemic and aleatoric uncertainties. We affinize the learned model the control space according to the switching law to construct a control-affine model based on which the L1 control input is designed. The switching law design provides distinct capability to explicitly control the predictive performance bound of the state predictor within the L1 adaptive control architecture while harnessing the robustness advantages offered by the L1 adaptive control. The L1 add-on does not require any modifications to the underlying MBRL. algorithm, making it ag- nostic to the choice of the baseline MBRL algorithm. To evaluate the effectiveness of the L1 -MBRI. scheme, we conduct extensive numerical simulations using two baseline MBRL algorithms across multiple environments, including scenarios with action or observation noise. The results unequiv- ocally demonstrate that the 4 MMBRL scheme enhances the performance of the underlying MBRL algorithms without any redesign or retuning of the L controller from one scenario to another.
1.2 RELATED WORK
Control Augmentation of RL policies is of significant relevance to our research. Notable recent studies in this area, including Cheng et al. (2022) and Arevalo-Castiblanco et al. (2021), have in- vestigated the augmentation of adaptive controllers to policies learned through MFRL algorithms. However, these approaches are limited by their assumption of known nominal models and their restriction to control-affine or nonlinear models with known basis functions, which restricts their applicability specific system types. In contrast, our approach does not assume any specific struc- ture or knowledge of the nominal dynamics. We instead provide a general framework to augment an C1 adaptive controller to the learned policy, while simultaneously learning the transition function.
Robust and adversarial RL methods aim to enhance the robustness of RL policies by utilizing minimax optimization with adversarial perturbation, seen in various studies (Tobin et al. 2017 Peng et al. 2018 Loquercio et al., 2019; Pinto et al. (017]. However, existing methods often involve modifications data or dynamics in order handle worst-case scenarios, leading to poor general performance. In contrast, our method offers distinct advantage by enhancing robustness without perturbing the underlying MBRL algorithm. This allows us to improve the robustness of the baseline algorithm without sacrificing its general performance.
ML methods train models across multiple tasks facilitate rapid adaptation to dynamic variations as proposed in (Finn et al.) 2017; Nagabandi et al. 2019a b). These approaches employ hierarchical latent variable models to handle non-stationary environments. However, they lack ex- plicit provisions for uncertainty estimation rejection, which result in significant performance degradation when faced with uncertain conditions (Chen et al., 2021). In contrast, the LM-MBRL framework is purposefully designed to address this limitation through uncertainty estimation and ex- plicit rejection. Importantly, our L -MBRL method offers the potential for effective integration with meta-RL approaches, allowing for the joint leveraging of both methods achieve both environment adaptation and uncertainty rejection in non-stationary and noisy environment.
PRELIMINARIES
In this section we provide a brief overview of the two main components our C -MBRL frame- work: MBRL and L1 adaptive control methodology.
2

---

Published as a conference paper at ICLR 2024
2.1 MODEL BASED REINFORCEMENT LEARNING
This paper assumes discrete-time finite-horizon Markov Decision Process (MDP), defined by the tuple M = (X
po; 7, H). Here X' R" is the compact state space, 4 C IR is the compact action space, == X' x U - N' is the deterministic transition function, : X' x A + IR is a bounded reward function. Let {(x) be the set of probability distributions over X' and po E(X') is the discount factor and H E N is a known horizon of the
4, J,
be the initial state distribution. problem. For any time step t H, if Zt E X' or dt & U, then the episode
considered to have failed such that r(ze, ue) = O for all t' = t,t+1, ,.... H. A policy is denoted by and is specified as т ;= [T],..., -H-IJ. "I....,TT-II) I]. where Ti X + E(A) and E(A) is the set of probability distributions over A. The goal of RL is to find a policy that maximizes the expected sum of the reward along a trajectory T == (Iv. uo.
-H -H TH). or formally, to maximize [ = Me IE (=1 rfr( ue]], where T++1 22=- = f uz) (Nagabandi et al. 2018). The trained model can be utilized in various ways to obtain the policy, as detailed in Sec. 3.1
2.2 L ADAPTIVE CONTROL
The L1 adaptive control theory provides framework to counter the uncertainties with guaranteed transient and steady-stan performance, alongside robustness margins (Hovakimyan & Cao 2010). The performance and reliability of the L1 adaptive control has been extensively tested on systems in- cluding robotic platforms (Cheng et al. 2022 Pravitra et al. [020]] Wu et al. (022), NASA AirSTAR
sub-scale aircraft (Gregory et al.) 2009 [2010), and Learjet (Ackerman al. (2177). While we give
a brief description of the 2, adaptive control in this subsection, we refer the interested reader to Appendix A for detailed explanation.
Assume that the continuous- time dynamics of a system can be represented as
+(t) = g(x(t)) + h(x(t))((t)) + a(t), u(t)). ax(0) = Ju.
where (x) E R" is the system state, u(() E RT is the control input, g R - R and h RR" - Raxm are known nonlinear functions, and d(t,x(()) u(t) E R. represents the unknown residual containing both the model uncertainties and the disturbances affecting the system.
Consider a desired control input (t) E RT and the induced desired state trajectory x x1()) E I based on the nominal (uncertainty-free) dynamics
x (t) = g(x (t)) + (t))u² x² x=(0)= = Z0.
(2)
If we directly apply (t) the true system in Equation ( the presence of the uncertainty d(t, xxt), u(t) can cause the actual trajectory to diverge unquantifiably from the nominal trajectory. The L1 adaptive controller computes an additive control input un (a()) to ensure that the augmented input u(t) = u² (t) + (t) keeps the actual trajectory xxt) bounded around the nominal trajectory x (t) in a quantifiable and uniform manner.
The Ln adaptive controller has three components: the state predictor, the adaptive law, and a low- pass filter. The state predictor given by
a(1) = g(x(()) + h(x(())) (1) + (I) + +
+ A,5(1).
with the initial condition a0)) 2o. where 2(t) R" is the state of the predictor, dtt) is the estimate of d(t, st)), u(t)), a(t) = a(t) - xxt) is the state prediction error, and A, E Ruxx is a Hurwitz matrix chosen by the user. Furthermore, (t) can be decomposed as
(1) ((a(()) + h (m)()))
(4)
where (t) and dum (f) are the estimates of the matched and unmatched uncertain- ties. Here, h. (z(t)) E (n-mm is a matrix satisfying h(x(t)) h. (x(t) = 0 and rank ([)()(()), (x(())) = n. The existence of (x(()) is guaranteed, given h(x(()) is a full-
rank matrix. The role of the predictor is to produce the state estimate a(t) induced by the uncertainty estimate (t).
The uncertainty estimate is updated using the piecewise constant adaptive law given by
((1) (iT.)) - - (T.)())))((T)),, E (iT.,, (i + 1)T,), EEN, a(0)= = āo. (5)
3

---

Published as a conference paper at ICLR 2024
where (TT)) A (exp(A,TT) - In), p((T))) = =xp((A,T,)E((TT)) and T, is the sampling time. Finally, the control input is given by
(6) where [|]] denotes the Laplace transform, and the L1 input un (t) is the output of the low-pass filter C(s) in response the estimate of the matched component o. (t). The bandwidth of the low-pass filter is chosen to satisfy the small-gathe stability condition (Wang & Hovakimyan, 2011).
v(()= n (1) ua
ua(s) - -
3 THE L1-MBRL ALGORITHM
In this section, we present the L -MBRL algorithm, illustrated in Fig. We first explain a standard MBRL algorithm and describe our method to integrate Li adaptive control with it.
3.1 STANDARD MBRL
As our work aims to develop an add-on mod- ule that enhances the robustness of an exist- ing MBRL algorithm, we provide high-level overview of a standard MBRL algorithm and its popular complementary techniques.
Tmin Dynamics Model(s)
It C (xxx)
Model A Affinization
U
The standard structure of MBRL algorithms involves the following steps: data collection, model updating, and policy updating using the updated model. To reduce model bias, many recent results consider ensemble of mod- els (ff. .M. M E N. on the data set D. The ensemble model fo, is trained by min- imizing the following loss function for each Og (Nagabandi et al. 2018):
Or
Agent
c Controller
MBRL
L Add-on
Figure 1: L1 MBRRL Framework. The policy box To (- includes policy update and control input sampling for each time step. Although this figure illustrates an on-policy MBRL setting with pa- rameterized Ta to provide a simple visualization, the framework is not limited to such class and can also be applied off- off-policy algorithms without a parameterized policy.
Σ
(10441-xx)= fo, (2,, 4
(D)
(E))
(7)
In this paper, we consider 7 as the loss func- tion used train the baseline MBRL algorithm among other possibilities. This is only for conve- nience in explaining the L1 augmentation in Section 3.2, and appropriate adjustments can be readily made for the augmentation upon the choice of different loss functions.
Besides the loss function, methods like random initialization of parameters, varying model archi- tectures, and mini-batch shuffling widely used reduce the correlation among the outputs of different models in the ensemble. Further, various standard techniques including early stopping, input/output normalization, and weight normalization can be used avoid overfitting.
Once the model is learned, control input can be computed by any of the following options: I) using the learned dynamics
simulator to generate fictitious samples (Kurutach et al., 2018 Clavera et al. 2018). 2) leveraging the derivative of the model for policy search (Levine & Koltun 2013
Heess et al. 2015), 3) applying the Model Predictive Controller (MPC) (Nagabandi et al.
2018
Chua et al.) 2018). We highlight here that our proposed method is agnostic to the use of particular techniques or the choice of the policy optimizer.
3.2
L1 AUGMENTATION
Let the true dynamics in discrete time be given by
(8) where the transition function and the system uncertainty w are unknown. Let := M ΣM So. be the mean of the ensemble model trained using the loss function in Equation
Art1 f(z,, u.) w(t, u)) Aræt 2 x441 Et,
Then, we express
Art1 fo(2e, uz), AFr+ A Ft+1
Ft,
4

---

Published as a conference paper at ICLR 2024
where Tt+1 indicates the estimate of the next state evaluated with fo uz). In MBRL, such tran- sition functions are typically modeled using fully nonlinear function approximators like NNs. How- ever, as discussed in Sec. 2.2, it is necessary represent the nominal model in the control-affine form to apply L1 adaptive control. A common approach to obtain a control-affine model involves restricting the model structure to the control-affine class (Khojasteh al. 2020 TTylor al. 2019 Choi et al. 2020p. For NN models, this process involves training two NNs go and hg, such that Equa- tion becomes A8 = go (xxx he(xz))
While control-affine models are used for their tractability and direct connection to control-theoretic methods, they are inherently limited in their representational power compared to fully nonlin- ear models, and hence, their use in an MBRL algorithm can result in reduced performance. To study the level of compromise on the performance, we compare fully nonlinear models with control-affine mod- els in the Halfcheetah environment for METRPO (Kuru-) tach et al. 2018), where each size of the implicit layers of the control-afline model Go and hg are chosen to match that of the fully nonlinear fe. The degraded performance of the control-affine model depicted in Fig. _ can be pri- marily attributed to intricate nonlinearities in the environ- ment.
Halfcheetah
Nonlinear
3000
Contrel-Affine
Episode Return
2000
1000
Although using the above naive control-affine model can be convenient, it must trade in the capabilities of the un- derlying MBRL algorithm. To avoid such limitations, we adopt an alternative approach inspired by the Guided Policy Search (Levine & Koltun| 2013). Specifically, we apply a control-affine transformation the fully nonlin- ear dynamics multiple times according to the predefined switching law. Specifically, we apply the first-order Tay- lor series approximation around the operating input ū:
50
100 150
200
Environment Steps (x10²)
Figure 2: Comparison of perfor- mance between fully nonlinear and control-affine model on the Halfchee- tah environment using METRPO. The control-affine model failed learn the Halfcheetah dynamics.
fo(I..u) u) x fol 12) + ([ (Vafoloe. w)]
) (u - 8)
= ffo(z,, 2) - ([Vufolan) 2) u=u a a+
([Vufo(re (ote w) u=
) ue e (Ie, (10)
(12))
ho(())
Here, the superscript a indicates the affine approximation of fe. The term affinization in this paper is distinguished from linearization, which lincarizes the function with respect to both I; and ue such that ēt+1 Ar, + Bu, for some constant matrix and B. Since it is common to have more states than control inputs in most controlled systems, the affinized model is significantly more accurate approximation of the nominal dynamies compared the linearized model.
Indeed, the control-affine model jg is only a good approximation of fo around ü. When the control input deviates considerably from the quality of the approximation deteriorates. To handle this, produce the next approximation model when the following switching condition holds:
|lfg (1t, u4; ū) fo u)) 2 ca-
(11)
Here, ||| indicates the vector norm, and e, the model tolerance hyperparameter chosen by the user. Note that as ca + O, we make an affine approximation at every point in the input space and we retrieve the original non-affine function fo-
Remark 1. Although a more infuitive choice for the switching condition would be || ūl >
we adopt an implicit switching condition (Equation (I7) to explicitly control over the acceptable level of prediction error between f and fo by specifying the threshold ca. This approach prevents significant deviation in the performance of the underlying MBRL algorithm, and its utilization is in- strumental in establishing the theoretical guarantees of the uncertainty estimation (See Section 3.3
Given a locally valid control affine model fg. we can proceed with the design of the Cn input by utilizing the discrete-time version of the controller presented in Sec. 2.2 In particular, the state-
5

---

Published as a conference paper at ICLR 2024
predictor, the adaptation law, and the control law are given by
1i+1 Et + A&, fe (Z ue) + 8r + (A, 20 = 20, It
-t
2t.
(12a)
8, - Men
(12b)
Q: = 9t-1 + (-K--1+ + Krm,--1
= 0, Ma,t -
(12c)
where K > is an X m symmetric matrix that characterizes the first order low pass filter C(s) K (slm + +K) discretized in time. Note that Equation (3 (3)EEuationn (6) can be considered as zero- order-hold continuous-time signals of discrete signals produced by Equation (2). As such, or and [4: are defined analogously to their counterparts in the continuous-time definitions. In our setting, where prior information about the desired input signal frequency unavailable, an unbiased choice is to set K = wIm. where w is the chosen cutoff frequency. The sampling time T, is set to At. which corresponds to the time interval at which the haseline MBRI. algorithm operates. The algorithm for the Li adaptive control is presented in Algorithm
As the underlying MBRL algorithm updates its model fe. the corresponding control-affine model fğ and L1 control input ua are updated sequentially (Algorithm []]. By incorporating the Ci control augmentation into the MBRL al- gorithm, we obtain the L -MBRL algorithm, as outlined in Algorithm - Note that in this algorithm we are adding URL, instead of ue to the dataset. Intuitively, this is to learn the nominal dynamics that remains after uncertain- ties get compensated by Na,t. Similar approach has been employed previously in (Wang Ba 2020 Appendix I.
Algorithm 1: L1 ADAPTIVE CONTROL
Data: Initialize (&,, dt) 0 Set w for K in Equation 12c
Function Control (URL,, f8):
Prediction error update: 3i
5:g 3t
Uncertainty estimate à:
update: Equation 12h
Compute Wa,t (Equation 12c ue < uRr. Wa.t
Update d2z1 (Equation
12a
return
End Function
Our -MMBLL framework makes a control- affine approximation of the learned dynamics, which is itself an approximation of the ground truth dynamics. Such layers of approximations may amplify errors, which may degrade the ef- fect of the L augmentation. In this section, prove that the L1 prediction error is bounded, and subsequently, the L1 controller effectively compensates for uncertainties. To this end, we conduct continuous-time analysis of the sys- tem that is controlled via LIBMBRI, frame- work which operates in sampled-ting time (Aström & Wittenmark] 2013). It is important to note that our adaptation law (Equation (5) holds the estimation for each time interval (zero-order- hold), converting discrete estimates obtained from the MBRL algorithm into continuous- time signal. Such choice of the adaptation law ensures that the L1 augmentation is compatible with the discrete MBRL setup, providing the basis for the following analysis.
Algorithm 2: L1 MBBRL ALGORITHM
Set D 0, {ā., dr} + 0 Initialize TT. fo, 20,00
repeat
for Episodes N₂ E IN do None for Horizon N
URL. ~ (=(()x))
if a is None or Equation
77 then
7 + URI.1 compute fo via Equation 10
Me Control Algo [ Execute Me and
D + (It, URL, Ze+1)
Update model(s) fo using Update policy Te until the average return converges
3.3 THEORETICAL ANALYSIS
Consider the nonlinear (unknown) continuous-time counterpart of Equation (
i(()= = F(x((), + ((t,x((),
(13)
where F: X U4 + R" is fully nonlinear function defining the vector field. Note that unlike the system in Equation I we do not make any assumptions on (xx(t), u(t) being control-affine. Furthermore, recall from Sec. 2.1 that the sets X c R and 4 Rm, over which the MBRL
The continuous-time functions correspond the Fuler-integral of its discrete counterparts.

---

Published as a conference paper at ICLR 2024
experiments take place, are compact. Additionally, W(t, a(t)) R" represents the disturbance perturbing the system. As before, denote by Fg (xr(t)) u()) the approximation of F(x((t, u(t)), and its affine approximate as
Fg (x(t), - = + Ho (z(())))(())
(14)
Subsequently, we define the residual eπror (tt, xt)), u(t) and an affinization enror a(x(t), u(t)) as l(t, a(t) - ((x(tt,, =(t)) + ((t, (t),, u(t)) - Fo(x(t),
a(x(t), at)) - Fo(x((), u(t)) - F2 (z((), a(t)).
Note that ||((x(t), ((t) S c, in the MMMML framework by Equation
11
We pose the following assumptions.
Assumption 1.
1. The functions F ((r()), u(l)) and W (1, x(), u(()) are Lipschitz continuous over X x U and (0, bmax) X x U, respectively, for mmax S H, where H known finite time horizon of the episode. The learned model Fo(x(t), u(t)) is Lipschitz continuous in X, and continuously differentiable)(C)) in U.
2. The learned model is uniformly bounded over (1, E (max) x X U: [1 u(t)) + (tt, x(t), Fo u(t))l S a,
(15)
where the bound ej assumed be known.
See Appendix B for remarks on this assumption.
Next, we set
u(t) = u'' + uu()),
(16)
where u²(t) (t) is the continuous-timg signal converted from the discrete control input produced by the underlying MBRL,, and (() is the Li control input. As described in Section 2.2 the Li controller
estimates the uncertainty by following the piecewise constant adaptive law (Equation Now, we aim to evaluate the estimation error e(t, u(t)):
e(t,æ(t), u(t)) I(t, u(t) =F(x(t), u(t) + (tt) u(t) Go(x(t)) (o(=))))(t)-6((
u(t)) o(t)
(17)
Our interest in evaluating the estimation error is articulated in Remark 2] Appendix C where we also provide proof of the following theorem. We note here that the sets X' and U in the following result are compact due to the inherent nature of the MBRL algorithm, as described in se..
Theorem 1. Consider the system described by Equation (13), and its learned control-affine repre- sentation in Equation (14). Additionally, assume that the system is operating under the augmented feedback control presented Equation (6).. Let A, diag bb........, be the Hurwitz matrix
that is used in the definition of the state predictor (Equation (3). If Asssumppto [ holds, then the
estimation error defined Equation 177 satisfies xe((,x((), u([)]l S el + Eas VI E (0, Ts)) and
|le(t, ((),, u(())ll = 2ea + OTT.) VL E (ms;)max);
where 0 < T, < 1max S H < 00, and H is the known bounded horizon (see Sec.
2.1
4 SIMULATION RESULTS
In this section, we demonstrate the efficacy of our proposed L -MBRI framework using the METRPO (Kurutach et al.) 2018) and MBMF (Nagabandi al., 2018 as the baseline MBRL method? and we defer the Ci -MBMF derivations and details to Appendix D We briefly note here that we observed a similar performance improvement with Cn augmentation for C₁ -METRPO.
P Mooe precisely, Cl everywhere except finite of measure zero
"METRPO and MBMF conform to the standard MBRL framework but employ different strategies for con- trol optimization (refer to Section 2.1). Selecting these baselines demonstrates that our framework is agnostic various control optimization methods, illustrating its functionality versatile add-on module.
7

---

Published as a conference paper at ICLR 2024
In first experimental study, evaluate the proposed L1 -1BMRL framework five different OpenAl Gym environments (Brockman et al., 2016) with varying levels of state and action com- plexity. For each environment, report the mean and standard deviation of the average reward per episode across multiple random seeds. Additionally, we incorporate noise into the observation (a, = 0.1) action (a, 0.1) by sampling from uniform distribution (Wang al. (019). This enables us to evaluate the impact of noise on MBRL performance. The results are summarized in Table Further details of the experiment setup are provided in the Appendix D
Table I: Performance comparison between METRPO and LI-METRPO (Ours). The average perfor-
mance and standard deviation over multiple seeds are evaluated for window size of 3000 timesteps at the end of the training for multiple seeds. Higher performance cases are marked in bold and green.
Noise-free
du 0.1
Do 0.1
Env.
METRPO
MEERRO
METRPO
C1 4I-METRPO
METRPO
L: E:-METRPO
Inv.
-5133 67.8
0.00
105.22 81.6
-74.22 +
-21.3
Swimmer
309.5 = 19.3
313.8 18.7
258.7 + 113.7
322.7
30.7 56.1
79.2 2 85
Hopper
1140.1 +: 552.1
623.8 1491
609.0 + 793.5
868.7 + 735.8
-1391 266.5
186 6.6.6.2 459.9
Walker
6.6 0.3 0.3
-6.9 = 0.5
-9.8 2.2
-5.9 - 0.3
-30.3 28.2
1 0.3
Halfcheetah
2367.3 + 1274.5 2588.6 955.
1920.3 + 932.4
2515.9
1216.4
1419.0 517.2
1906.3 972.7
The experimental results demonstrate that our proposed L -MBRL framework outperforms the base- line METRPO in almost every scenario. Notably, the advantages of the Li augmentation become more apparent under noisy conditions.
4.1 ABLATION STUDY
We conduct an ablation study to compare the specific contributions of L1 control in the training and testing. During testing, Li control explicitly rejects system uncertainties and improves performance. On the other hand, during training. L1 additionally influences the learning by shifting the training dataset distribution. To evaluate the effect of Lı each phase, we compare four scenarios: 1) no L1 during training or testing. 2) Li applied only during testing, 3) L1 used only during training, and 4) C1 applied during both training and testing. The results are summarized in Fig.
Inverted Pendulum
Swimmer
Hopper
2000
Average reward
300
-50
-100
100
-150
-1000
0,
a,
&,
Walker
Halfcheetah
Average reward
-20
3000
L1 in training-testing off-off off-on on-off
1000
on-on
-60
G2 Cy
Figure 3: Contribution of L₁ in the training and testing phase. The notation L₁ on (off)-on (off) indicates Li is applied (not applied) during training-testing, respectively. The emror bar ranges for one standard deviation of performance. On-on and off-off correspond to our main result in Table I As expected, the on-on case achieved the highest performance in most scenarios.
8

---

Published as a conference paper at ICLR 2024
The results depieted in Fig. demonstrate that the influence of L1 control during training and testing varies across different environments and noise types. However, as anticipated, the highest perfor- mance is achieved when & is applied in both the training and testing phases.
In order to evaluate the effectiveness of the L1 -MBRL framework in addressing the sim-to-rrea gap, conducted secondary ablation study. First, trained the model without L1 in noise-free environment and subsequently tested the model with and without & under a noisy environment. The results are demonstrated in Table 2 This result indicates that our -MBRRL framework effec- tively addresses the sim-to-real gap, and this demonstrates the potential for directly extending our framework the offline MBRL setting, presentigg promising opportunities for future research.
Table 2: Addressing the sim-to-real gap with L1 augmentation: The original METRPO was initially trained on an environment without uncertainty. Subsequently, the policy was deployed in noisy environment that emulates real-world conditions, with and without & augmentation.
On 0.1
do 0.1
Gn 0.1 & G. = 0.1
Ein.
METRPO
4--METRPO
METRPO
4-MMERPO
METRPO
MEMMERP
Inv. P.
30.2 + 45.1
-0.0 0.0
-74.1 53.1
2.0
-107.0 + 72.4
-6.1 4.6
Swimmer
250.8 +: 130.2
330.5
337.8420
331.2 8.34
248.2 133.6
327 3 6.8
Hopper
198.9 +: 617.8
623.4 405.6
-84.5 1035.8
157.1 + 379.7
87.3
510.2
309.8 = 477.8
Walker
-.00 0.8
0.7
-6.4
-6.08 0.6
-6.3 0.4
55.2 + 1.5
Halfchectah
1845.8 600.9
1965.3 + 839.5
1265.0 440.8
1861.6 6 605.5
1355.0 + 335.6
1643.6 712.5
5 LIMITATIONS
(Performance of the base MBRL) Our LI-MBRL scheme rejects uncertainty estimates derived from the learned nominal dynamics. As result, the performance of Ci-MBRL inherently tied to the baseline MBRL algorithm, and L1 augmentation cannot independently guarantee good perfor- mance. This can be related to the role of q in Equation (15). Empirical evidence Fig. iillustrates this point that, despite meaningful improvements, the performance of scenarios augmented with L closely correlated to that of METRPO without C, augmentation.
(Trade-off in choosing 6.) In Sec. 3.2 we mentioned that as approaches zero, the baseline MBRL. recovered. This implies that for small values of cas the robustness properties exhibited by the L1 control are compromised. Conversely, if ca is increased excessively. it permits significant deviations between the control-affine and nonlinear models, potentially allowing for larger errors in the state predictor (see Section 3.3) Our heuristic observation from the experiments is to select an <a that results in approximately 0- TOO affinization switches per 1000 time steps for systems with low complexity (n < 5) and 200-500 switches for more complex systems.
CONCLUSION
In this paper, we proposed an L1 -MBRL control theoretic add-on scheme to robustify MBRL al- gorithms against model and environment uncertainties. We affinize the trained nonlinear model according to switching rule along the input trajectory, enabling the use of L adaptive control. Without perturbing the underlying MBRI algorithm, we were able to improve the overall perfor- mance in almost all scenarios with and without alcatoric uncertainties.
The results open up interesting research directions where we wish to test the applicability of Li- MBRL on offline MBRL algorithms to address the sim-to-real gap (Kidambi et al. 2020 Yu et al.) 2020). Moreover, its consolidation with a distributionally robust optimization problem to address the distribution shift is of interest. Finally, algorithms with probabilistic models (Chua et al. 20188, Wang & Ba 2020 to explore a method to utilize the covariance information in addition mean dynamics.
will also research the L1-MBRL design for MBRL

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work financially supported by National Aeronautics and Space Administration (NASA) ULI (80NSSC22M0070), NASA USRC (NNH2)ZEAO0IN-UURR)) Air Force Office of Scien- tific Research (FA-550-21-1-0411),, National Science Foundation (NSF) AoF Robust Intelligence (2133656), NSF CMMI (2135925), and NSF SLES (2331878).
REFERENCES
Kasey A Ackerman, Enric Xargay, Ronald Choe, Naira Hovakimyan, M Christopher Cotting, Rohert B Jeffrey, Margaret P Blackstun, T Paul Fulkerson, Timothy R Lau, and Shawn S Stephens. Evaluation of L1 Adaptive Flight Control Law on Calspan's variable-stability Learjet. AIAA Journal of Guidance, Control, and Dynamics, 40(4):1051-1066 2017.
Miguel F Arevalo-Castiblanee César A Uribe, and Eduardo Mojica-Nava. Model Reference Adap- tive Control for Online Policy Adaptation and Network Synchronization. In IEEE Conference on Decision and Control, pp. 4071-4076, 2021.
Karl Aström and Björn Wittenmark. Computer-Controlled Systems: Theory and Design. Courier Corporation, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl gym. arXiv preprint arXiv:1606.01540. 2016.
Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao. Context-Aware Safe Reinforcement Learning for Non-stationary Environments. In IEEE Inter- national Conference Robotics and Automation, pp. 10689-10695, 2021.
Yikun Cheng, Pan Zhao, Fanxin Wang, Daniel Block, and Naira Hovakimyan. Improving the Robustness of Reinforcement Learning Policies with L1 Adaptive Control. IEEE Robotics and Automation Letters, 7():6657--581, 2022.
Jason Choi, Fernando Castañeda, Claire J Tomlin, and Koushil Sreenath. Reinforcement Learn- ing for Safety- Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions. In Roboties: Science and Systems, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep Reinforcement Learning in a Handful of Trials Using Probabilistic Dynamics Models. Advances in Neural Infor- mation Processing Systems, 31, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-Based Reinforcement Learning via Meta-Policy Optimization. In Conference on Robot Learming. pp. 617-629, 2018
Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian Processes for Data- Efficient Learning in Robotics and Control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408 4233, 2013.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference Machine Learning, pp. 1116-11 135, 2017.
Irene Gregory, Chengyu Cao, Enric Xargay, Naira Hovakimyan, and Xiaotian Zou. C1 Adaptive Control Design for NASA AirSTAR Flight Test Vehicle. In AlAA Guidance, Navigation, and Control Conference, pp. 5738, 2009.
Irene Gregory, Enric Xargay, Chengyu Cao, and Naira Hovakimyan. Flight Test of an L1 Adaptive Controller the NASA AirSTAR Flight Test Vehicle. In AIAA Guidance, Navigation, and Control Conference, pp. 8015, 2010.
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap. Yuval Tassa, and Tom Erez. Learning Continuous Control Policies by Stochastic Value Gradients. arXiv preprint arXiv: 1rX0v.5510.09142, 2015.
10

---

Published as a conference paper at ICLR 2024
Naira Hovakimyan and Chengyu Cao. L1 Adaptive Control Theory: Guaranteed Robustness with Fast Adaptation. SIAM, 2010.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: Lessons we have learned. The International Journal of Robotics Research, 40(4-5):698-72) 2021.
Hassan K Khalil. Nonlinear Systems Third Edition. Patience Hall, 2002.
Evgeny Kharisov. Li Adaptive Output-Feedbuck Control Architectures, PhD thesis, 2013.
Mohammad Javad Khojasteh, Vikas Dhiman, Massimo Franceschetti, and Nikolay Atanasov. Prob- abilistic Safety Constraints for Learned High Relative Degree System Dynamics, In Learning for Dynamics and Control, pp. 781 781-792, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOREL: Model- Based Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 33: 20210021823, 2020.
Craig Knuth, Glen Chou, Necmiye Ozay, and Dmitry Berenson. Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constants. IEEE Robotics and Automation Letters, 6(3):5129-51366 2021.
Jus Kocijan, Roderick Murray-Smith, Carl Edward Rasmussen, and Agathe Girard. Gaussian Pro- cess Model Based Predictive Control. In IEEE American Control Conference, volume 3, pp- 2214-2219, 2004.
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning- Based Model Predictive Control for Safe Exploration. In IEEE Conference on Decision and Control, pp. 6059- 6066, 2018
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Picter Abbeel. Model-Ensemble Trust-Region Policy Optimization. In International Conference Learning Representations, 2018.
Arun Lakshmanan, Aditya Gahlawat, and Naira Hovakimyan. Safe Feedback Motion Planning: A Contraction Theory and L1 Adaptive Control Based Approach. In IEEE Conference on Decision and Control, pp. 1578-1583, 2020.
Sergey Levine and Vladlen Koltun. Guided Policy Search. In International Conference on Machine Learning, pp. 1-9, 2013.
Junxiang Li, Liang Yao, Xin Xu, Bang Cheng, and Junkai Ren. Deep Reinforcement Learning for Pedestrian Collision Avoidance and Human-Machhe Machine Cooperative Driving. Information Sciences, 532: 1 10-124, 2020.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. In International Conference on Learning Representations, 2016.
Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza. Deep Drone Racing: From Simulation to Reality with Domain Randomization. IEEE Transactions Robotics, 36(1):1-14,201
José María Manzano, Daniel Limon, David Muñoz de la Peña, and Jan-Peter Calliess. Robust Learning- -aaed MPC for Nonlinear Constrained Systems. Automatica, 117:108948, 2020.
Elliott Mendelson. Schaum's 's Outline of Calculus. McGraw-Hil Education. 2022.
Stefan Milz, Georg Arbeiter, Christian Witt, Bassam Abdallah, and Senthil Yogamani. Visual SLAM for automated driving: Exploring the applications of deep learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 247-257, 2018.

---

Published as a conference paper at ICLR 2024
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level Control through Deep Reinforcement Learning. Nature, 518(7540):529-33 533, 2015.
Thomas M Moerland, Joost Brockens, Aske Plaat, Catholijn M Jonker, et al. Model-Basee Re- inforcement Learning: A survey. Foundations and Trends in Machine Learning. 16(1):1-118, 2023.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural Network Dy- namics for Model-Based Deep Reinforcement I learning with Model-Free Fine-Tuning. In IEEE International Conference Robotics and Automation, pp. 7559-7566, 2018.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abheel, Sergey Levine, and Chelsea Finn. Learning Adapt in Dynamic, Real-World Environments through Meta- Reinforcement Learning. In International Conference on Learning Representations, 2019a.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep Online Learning Via Meta-Learning: Continual Adaptation for Model-Based RL. In International Conference on Learning Represen- tations, 2019b.
David Neal, Matthew Good, Christopher Johnston, Harry Robertshaw, William Mason, and Daniel Inman. Design and Wind-Tunnel Analysis of Fully Adaptive Aireraft Configuration. In AIAA Structures, Structural Dynamics & Materials Conference, pp. 1727, 2004.
Hai Nguyen and Hung La. Review of Deep Reinforcement Learning for Robot Manipulation. In IEEE International Conference on Robotic Computing, pp. 590-595, 2019.
Robert Nichols, Robert T Reichert, and Wilson Rugh. Gain Scheduling for H-infinity Con- trollers: A Flight Control Example. IEEE Transactions on Control Systems Technology, 1(2): 69-79, 1993.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. In IEEE International Conference on Robotics and Automation, pp. 3803-3810, 10, 2018
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Rein- forcement Learning. In International Conference Machine Learning, pp. 281 2817-2826, 2017.
Jintasit Pravitra, Kasey Ackerman, Chengyu Cao, Naira Hovakimyan, and Evangelos A Theodorou. Li-Adaptive MPPI Architecture for Robust and Agile Control of Multirotors. In IEEE International Conference on Intelligent Robots and Systems, pp. 7661-7666, 2020.
Joaquin Quinonero-Cander Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.
Dataset Shift Machine Learning. Mit Press, 2008
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Struc- tured Prediction to No-Regret Online Learning. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pp. 627 635, 2011.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In International Conference on Machine Learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal Policy 2017.
Optimization Algorithms. arXiv preprint arXiv:
Andrew Taylor, Victor Dorobantu, Hoang M Le, Yisong Yue, and Aaron D Ames. Episodic Learning with Control Lyapunov Functions for Uncertain Robotic Systems. In IEEE International Conference on Intelligent Robots and Systems, pp. 6878-6884, 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main Randomization for Transferring Deep Neural Networks from Simulation to the Real World. In IEEE International Conference Intelligent Robots and Systems, pp. 23-30, 2017.
12

---

Published as a conference paper at ICLR 2024
Tingwu Wang and Jimmy Ba. Exploring Model-based Planning with Policy Networks. In Interna- tional Conference on Learning Representations, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking Model-Based Reinforce- ment Learning. arXiv preprint
2019.
Xiaofeng Wang and Naira Hovakimyan. L1 Adaptive Controller for Nonlinear Reference Systems. In IEEE American Control Conference, pp. 594-599, 1,9,,099 2011.
Xin Wang. Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang. Video Captioning via Hierarchical Reinforcement Learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp.
2018.
Xu Wang, Sen Wang, Xingxing Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai. and Qiguang Miao. Deep Reinforcement Learning: Survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to See Physics via Visual De-animation. Advances Neural Information Processing Systems, 30, 2017.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. A Study of Reinforcement Learn- ing for Neural Machine translation. In Conference Empirical Methods in Natural Language Processing, pp. 3612-3621, 2018.
Zhuohuan Wu, Sheng Cheng, Kasey A Ackerman, Aditya Gahlawat, Arun Lakshmanan, Pan Zhao, and Naira Hovakimyan. C1 Adaptive Augmentation for Geometric Tracking Control of Quadro- tors. In International Conference on Robotics and Automation, pp. 1329-1336, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-Based Offline Policy Optimization. Advances in Neural Information Processing Systems, 33:14129-1441 2020.
Sangdoo Yun, Jongwon Choi, Youngjoon Yoo, Kimin Yun, and Jin Young Choi. Action-Decision Networks for Visual Tracking with Deep Reinforcement learning. In IEEE Conference on Com- puler Vision and Pattern Recognition, pp. 2711-2720, 2017.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-Real Transfer in Deep Re- inforcement Learning for Robotics: A Survey. In IEEE Symposium Series on Computational Intelligence, pp. 737-744, 2020.
Ruijic Zheng, Xiyao Wang. Huazhe Xu, and Furong Huang. Is Model Ensemble Necessary? Model- based RL via Single Model with Lipschitz Regularized Value Function. In The International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXTENDED DESCRIPTION OF L1 ADAPTIVE CONTROL
In this section, we provide detailed explanation of L₁ adaptive control. Consider the following system dynamics:
-(1 +(1) - + - x(),, u(t));
x00) = 20:
(18)
where xtt) E R" is the system state vector, uut) E RT is the control signal, R" - E and h IR - R=xm are known functions that define the desired dynamics, both of which are socally-ipsschin continuous functions. Furthermore, d(t,, x((), u(t)) E R"" represents the unknown nonlinearities and is continuous in its arguments. We now decompose d(t,x(t), u(t)) with respect to the range and kernel of h(((()) to obtain
=(() -(x(()))+++))+))))())))))))))))) + (((()))(((( +
m(())) + (m (l)))()) x(0) 20, (19)
where h( (t, xtt), (t.) u(t)) + h (z(t)) (t, xzt), utt) = d(t, u())) Moreover, h (x(()) E (n-m) m) denotes a matrix whose columns are perpendicular to h(x(t)) E R=xm such that ((((t)) (x)())
for any æ(t) E R... The existence of (x())) is guaranteed if is a full-rank matrix. The terms o. (()),, u(t)) and a Jum (t, x(t,, ut)) are commonly referred to as matched and unmatched uncertainties, respectively.
Consider the nominal system in the absence of uncertainties
x (t) (4) - h(x (1)) x²(0)= = To
where (t) is the baseline input designed so that the desired performance and safety requirements are satisfied. we pass the baseline input to the true system in Equa- tion (8). the actual state trajectory =t)) can diverge from the desired state tra- jectory (t) in an unquantifiable man- ner due to the presence of uncertainties d(t, ((t), u(t)). To avoid this behavior, we employ L, adaptive control, which aims to compute an input ua(() such that, when combined with the nominal input u (I),, forms the augmented input
Baseline Controller
Uncertain System
C(s)
State Predictor
8
0
Adaptation Law
Adaptive Controller
Figure 4: The architecture of L1 adaptive controller.
u(t) = (t) + (t).
(20)
The objective of this approach is to ensure that the true state sr()) remains uniformly and quantifiably bounded around the nominal trajectory x² (t) under certain conditions.
Next, we explain how the L, adaptive controller achieves this goal. The L, adaptive control has three components: the state predictor, the adaptation law, and a low-pass filter. The state predictor for Equation 18) is given by
a(() = g(x(t)()) (((()))u))))))) + +(() + A,ū(t),
(21)
where (t) A h(((tt) (t) h- (=())))) (t). Moreover, 2m (t) and oun. (f) the estimates
of the matched and unmatched uncertainties om (t,, (t)) and o am (t, (t)), respectively. The initial conditions are given by (0) zo. (0) 0, and 6um (0) 0. Here, =()) E R is the state of the predictor, u(t) E Rm is the augmented control input in Equation (00). (t) (t) - xtt) denotes the prediction error, and A, S" a chosen Hurwitz matrix. The state predictor produces the state estimate i-t) induced by the adaptive estimates 5 (t) and @mma
Following the true dynamics in Equation (18) and the state predictor in Equation 21). the dynamics of the prediction enror ((t) = aat) - xxt) is given by
a(1) A,F(() (a(1) d(t, u()), a(0)= 0.
(22)
14

---

Published as a conference paper at ICLR 2024
Here, refer to (t) - d(t,, xxt), u()) the uncertainty estimation enror. Moreover, since A, Hurwitz, the system Equation (22 governing the evolution ă(t) is exponentially stable in the absence of the exogenous imput (t) d(t,, 2x)), u(t)). Therefore, =(t) serves as a learning signal for the adaptive law, which we describe next.
We employ piecewise constant estimation scheme (Hovakimyan & Cao 2010) based the solu- tion of Equation (22 which can be expressed using the following equation:
t(t) exp (A,t)ā(0) L'" exp (l(tt-))( - d(T, u(T))
(23)
For given sampling time T, > 0, we use a piecewise constant estimate defined as
(tt) (iTs), t (T,, (i + 1)T,), e N [0).
When the system is initialized (i - 0), we set ((t) 0 which implies ( = 0 for t E [0,T,). Now consider time index -N and the time interval [iT', (i+ 1)T.). Over this time interval, the solution of Equation written as
23 obtained by applying the piecewise constant representation, can be
(tt) exp (A, - (TT)))
+ *f""ex exp (A.(1 ( (A.(1-
())( - ((T) LE [TT, (1 ()TT....
Now, note that over the previous interval E [[i 1),,,, the system was affected by uncertainty d(t, (t),, u(t)), which resulted in .((T,) + 0. At the end of the time interval [iT,, (i + 1)T.), we obtain
(i+1)T.
a((i + 1)T.) exp (A.T.)
(xp(A.(i++1)) - -i))(()))) + Ra+IJT
VT,
exx (A.T.) ((TT))
AT (exp (A.T.) 11.)) (iT)) R(s+1y..
(24)
where
(1+1))
R(++YT, A
JIT,
exp (Ai + -
- -))dd)))((T,z(((( ut)),,,((())) u(T))dr
the residual term that captures the uncertainty entered during [iT,,, 1)T.).
Let
(25) and (iTT,) exp (ATT.) for 1, Sub- stituting this into Equation (24 removes the first two terms, leaving us only with the residual term, which will appear as the initial condition of the next time interval. In other words, the adaptation law
((TT,) = - -T-((T))((((T))) 6(0)= =
where (T,,) (exp (A,T.)
attempts to remove the effect of the uncertainty introduced in the current time interval by addressing at the start of the subsequent interval. Interested readers can refer (Kharisov) 2013 Ch. 2) for further details on the piecewise constant adaptive law.
Note that small sampling time T, results in small prediction error (( for each 2,
Therefore, it is desirable to keep T, small up to the hardware limit. However, setting small T. and/or having large eigenvalues of A, can lead to a high adaptation gain ($-1 in Equation 25 This can result in high-frequency uncertainty estimation, which can reduce the robustness of the controlled system if we directly apply ua (t) = - (1) to cancel the estimated matched uncertainty. Therefore,
low-pass ww-ppas filter in the controller decouple the fast estimation from the control loop, allowing us to employ an arbitrarily fast adaptation while maintaining the desired robustness. To be specifc, the input u. (t) is given by
(s) = CC() (t),,
where E[.] denotes the Laplace transform, and the C(s) is the low-pass filter. The cutoff frequency of the low-pass filter is chosen to satisfy small-gain gain stability conditions, examples which can be found in (Wang & Hovakimyan, 2011 Lakshmanan et al. 2020). We refer the interested reader to (Pravitra et al. [2000] Wu et al. |022| Cheng et al. 2022) for further reading on the design process of the 21 adaptive control.
15

---

Published as a conference paper at ICLR 2024
B REMARKS ON ASSUMPTION I
is evident from Equation 10 that our method relies on the continuous differentiability of Fo(x(t), u(t)) with respect to u(t) to ensure the continuity of Fg (ax(t), u(t)). Such a requirement readily satisfied when using C (or higher order continuously differentiable) activation functions for (x()),), u(t)), such as sigoid. tanh, or swish. sh. For MBRL algorithms that use activation functions that are not C², we can skip the switching law (Equation (I1) to avoid making affine non-differentiable points (e.g., the origin for ReLU).
approximations
The first part of Assumption [ on the regularity of F(x(t), u(t)) and Wtt, is standard to ensure the well poseddosssssss (uniqueness and existence of solutions) for Equation (13) (Khalil 2002
Theorem 3.1). Furthermore, as stated above, since (xx())), u(t)) is C in its arguments, its derivative (x(())), u(t)) is continuous and hence, satisfies the local Lipschitz continuity over the compact sets X' and U trivially.
The assumption in Equation 15 is satisfied owing to the Lipschitz continuity of the function in its respective arguments. If the bound is unknown, it is possible collect data by interacting with the environment under a specific policy and initial condition, and then compute a probabilistic bound. However, such bound is applicable only to the chosen data set and may not hold for other choices of samples, which is the well-known issue of distribution shift Ceiinonero-Candele et al. 2008). This ussumption, which explicitly bounds the unknown component of the dynamics, although cannot be guaranteed in the real environment, is commonly made in assessing theoretical guarantees of error propagation when using learned models, as seen in previous papers (Knuth et al. 2021 Manzano
et al. 2020 Koller et al., 2018).
C PROOF OF THEOREM
Proof of Theorem
7
We begin by applying the triangle inequality for the estimation emror:
lle(t, [le(t
-IF((x(() u()) + ((),=(),,(0)) Go(a(1)) (((()))(((()))))) a()I
= PPoo ((t,))((( - (((()) u(t) - a()]
(26)
- < -
[[(t, ((t, -
I <a + [1(t, x(),, - iE (0) UN,
where we used Equation T1..
For the case when 0, due to the initial conditions a(0) 0, â a0) 0, and the assumption in Equation (15). we get that
[ttt, u(t)) - a (0)1I = [[t (t), u (f)) S a Vt E T.).
Substituting this expression into Equation (66 proves the stated result for 1 E [0, Tz).
Next, we bound the term [[((,, (x),, - in Equation (6) for all E [Ta, mmax). Consider an i E N that corresponds to E [T'. i.c. {I..... fax [T]} 2 I. For any such i, substituting the adaptation law from Equation (25 into Equation (24) for the interval [(i 1)T,, iT,) produces the following expression
1
fm
iT,
i(TT.)
u(7))d ei
(27)
1)T,
Replacing d(r,,(7)) u(T)) |di (T,) u(T))
dn((, 2 u(T)]]] in Equation 27p and,
by the definition of A, in the theorem statement, we obtain
exp(Ai (T)
f(TT.)
dr,
exp(An (T)
(1-1YT,
for all E T. For brevity, we denote d,(r) = (T, u(7)) for = 1,2, ..........
16

---

Published as a conference paper at ICLR 2024
Since d, is continuous due to Assumption I and exp (xT, - -)) is positive semi-definite invoke the Mean Value Theorem [Mendelson] 2022, Sec. 24.1) element wise. We conclude that there exist te E [i 1)T,, iT,) for each J {I, {L..... n such that
PT. (1-1)1. cxp(A1 - r))d1 (te,)dr
B(TT))
-(-nr exp(An (iT, - (ten)dd
(28)
+ - (x)
A. (1 expn,,n,
Substituting Equation (28 into Equation
25
gives
@(t) = (iT)))
(29)
exp( (n (be.,)
which is the piece-wise constant estimate of the uncertainty for / E [iT,, (i+ 1)T.).
Next, using the piece-wise constant adaptation law from Equation 29 we bound (Ittt, u(()
(t) |]. for t E [i,,, (i + 1)T'). Since A, is Hurwitz and diagonal, its diagonal elements satisfy Ag O, for I,.... n. Thus, we have
exp(AiTa)dd(te))
(ft) a()]
(tt, u(t)
exp(An, T.)da (lo..)
[1((,x((t), u(t)) d((t) + (In exp (A,T, (d)t)))
S ((t,, a(t)) - -((te]l + ||I,- - exp (A,T,) [|((t)l S [((tt,((t), u(t)) - -d(ta)l +
exp( (11ee((()xm)))) 2t) [((())l
(30)
where d(l.) = [di (lag)
d. (t..).))) d(() = -I(T, =(T), u(()) + a(x(T), u(T)), and Amin de- notes the eigenvalue of A, that has the minimum absolute value. Using the triangle inequality, get [d((t))] [|((te, (tc), u(te) + a(x(te),
& + ca. Thus, Equation (30) can be written
(1),, w()) - a(01 < (1((,,((),, u(() - d(())l + +(1 -1 e1-exp(Amin (ipp(Ammnn.))((rre +ea)
(31)
Next, we obtain the upper bound for [lt(t, u(t) - d(ta)ll
(t),, u(t)) - d(t.)ll = [I((t,=(t,
x(t)) u(to)) - a(x(tt), u(t.))ll
(I((,,((),,u(())
((ee, (lc)) [|a(x((c)) a(te)))l
< (l((,,(()), u()) - I(le, (lo),, u(())) +
(32)
where we used the triangle inequality and Equation [Tp. Due to the Assumption ((t, x(t) u(t)) is Lipschitz over the domain of its arguments. Hence, there exist positive scalars Lt,e, L1,2, Lt such that
[M(t, (ft) (tt)), u(ta))ll S Le (1,1==()) (x(e)) |+ L1, |u(t) - u(ta)ll (33)
Furthermore, due the compactness X and U. there exist Lu,e, Lu,e such that the following inequalities hold:
|| - ((t))) S Dn,e/ lel, [| u((o))l S Lu,elt bel-
Substituting these bounds into Equation (33). we get
[1(t,, xx)), u(t)) - 1(t, (t.). u(t.))ll S t tal S LT,,,
(34)
where A Li,t + Le, + Li,,Lu,e < oc. We proceed by sequentially applying the derived bounds, starting with the substitution of Equation (34) into Equation (32). and then employing the resulting bound in Equation (3T). The proof is then concluded by incorporating the final bound into Equation (6) and noting that
(1 exp(Amin T.)) + ca) + LT, (T.))
17

---

Published as a conference paper at ICLR 2024
We provide some insights into the interpretation of this theorem. The theorem serves to quantify the predictive quality of the state predictor in the L1 add-on scheme in terms of the model approximation errors E₁ and Ca. and the parameters governing the L1 add-on scheme (T Amin). As the control imput is computed by low-pass fittering the uncertainty estimate, the performance of the Lı augmentation is inherently tied to its predictive quality. Theorem establishes that the error in state prediction, induced by estimated uncertainty, can be reduced down to 2e, by reducing the sampling time T,. In other words, we can accurately estimate the learning error 1(t, (tt))(t), u(t)), with the predictive accuracy being bounded only by the tunable parameter Co.
Remark 2.
D EXTENDED SIMULATION RESULTS
D.1 EXPERIMENT SETUP
We provide the dimensionality of the selected environments for our simulation analysis in Table For Li -METRPO, the number of iterations for each environment was chosen to obtain asymptotic performance, whereas for L1 -MBMF we fixed the number of iterations to 200K. Such setup for MBMF is due to the unique structure of MBMF to use MBRL only to serve as the policy initial- ization, with which MFRL is executed until the performance reaches the asymptotic results. See Appendix D.2.2 for a detailed explanation of MBMF
Table 3: Dimensions of state and action space of the environments used in the simulations.
Environment Name
State space dimension (n)
Action space dimension (m)
Inverted Pendulum
4
Swimmer
8
Hopper
11
Walker
17
Halfcheetah
17
We adopted the byerrparameters that have been reported to be effective by the baseline MBRL, which in our case are METRPO and MBMF (Wang et al. 2019, Appendix B.4, B.5). Additional hyperparameters introduced by the LM-MBL scheme are the affinization threshold c, the cutoff frequency w, and the Hurwitz matrix A,. Throughout all experiments, identity matrix -I,- For the Inverted Pendulum environment, we set 1 and for Halfcheetah selected cutoff frequency
fixed A, negative
= 3, while for other environments, we chose e 0.3. Additionally, of = 0.35 T,, where T, represents the sampling time interval of the environment. It is important to note that the C1 controller has not been redesigned or retuned through all the experiments.
D.2 TECHNICAL REMARKS
If the baseline algorithm employs any data processing techniques such as input/output normalization, as discussed briefly in Section 3.1 our state predictor and controller (Equation (T2a). Equation (1c)) must also follow the corresponding process.
D.2.1 METRPO
METRPO trains an ensemble model, from which fictitious samples are generated. Then, the policy network is updated following the TRPO (Schulman et al. 2015) in the policy improvement step. The input and output of the neural network are normalized during the training step, and consequently. calculation of the Jacobian in L1 -METRPO must unnormalize the result. Specifically, this process is carried out by applying the chain rule, which includes multiplying the normalized Jacobian matrix (J') by the standard deviations of the inputs and outputs given by Equation (35):
D D
(35)
where DA. = diag ... and Dr.,. = diag .z. Oups............................
This unnormalized Jacobian (J) is subsequently utilized to generate the L1 adaptive control output.
18

---

Published as a conference paper at ICLR 2024
D.2.2 MBMF
In the MBMF algorithm (Nagabandi al. 2018), the authors begin by training Random Shooting (RS) controller. This controller is then distilled into a neural network policy using the supervised framework DAgger (Ross et al. 2011), which minimizes the KL divergence loss between the neu- ral network policy and the RS controller. Then, the policy is fine-tuned using standard model free algorithms like TRPO (Schulman et al. 2015) or PPO (Schulman al. 2017). We adopt a similar approach to what was done for METRPO. The Jacobian matrix of the neural network is unnormal- ized based on Equation (35). The adaptive controller is augmented to the RS controller based on the latest model trained.
D.3 EXPERIMENT RESULTS
In this section, we first present the results of L1 -MBMF in comparison to MBMF without L1 augmentation. The corresponding tabular results are summarized in Table 4 Noticeably, Ci aug- mentation improves the MBMF algorithm in every case uniformly.
Table 4: Performance comparison between MBMF and LIMMBMF (Ours). The performance is averaged across multiple random seeds with window size of 5000 timesteps at the end of the training. Higher performance is written in bold and green.
Noise-free
NN 0.1
Ga
0.1
Env.
MB-MF
2y-MB-MF
MB-MF
CI-MB-MF
MB-MF
CI-MB-MF
Inv. P.
-100.5 4.3
-10.5 d 3.7
-7.4 1.5
-4.8 1.9
-10.2 à: 2.4
-5.09 1.6
Swimmer
284.9
25.1
314.3 3.3
304.8 1.9
314.5 0.6
292.8
294.3 4.3
Hopper
1047.4
1098.7
$50.1
465.2
877.9 383.4
285.4 65.8
999699 + 206.0
171.5 :: 317.3
Walker
1773.77 +: 233.3
1481 7 :: 322.9
2962.2 : 178.6
-2447.4
329.7
-3348.8 8 +: 210.1
-2281.43 381
Hallcheetah
126.9 72.7
804.5
56.0
184.0 + 148.9
299.
61.0
146.1
235.2 19.2
Additionally, we provide detailed tabular values corresponding to the results shown in Fig. Table 5 provides a summary of the scenarios where Li control augmented only during either training or testing. The application of L1 control during the testing phase clearly benefits from the explicit rejection of system uncertainty, leading performance improvement. On the contrary, when L control is applied during the training phase, it not only mitigates uncertainty along the trajectory also implicitly affects the training process by inducing shift in the distribution of the training dataset. This study compares these two types of impact brought about by the Li augmentation.
Table 5: Comparison of L1 augmentation effects during training and testing. Li -METRPO (Train) refers to the application of 2₁ augmentation solely during training, whereas L1 -METRPO (Test) indicates training without L augmentation and the application of L1 only during testing.
Noise-free
Ox 0.1
G. 0.1 METRROOTTI METRPOTTT))
Eny.
C--METRPO (Train) 4, METRPOTTTs)
4 TrTERPOTTTais CET Test)
Inv. P.
-8.50 :: 20.75
199.36 4: 22.3
33.52 8.08
-9..7 1: 47.34
-41.63 4 19.11
-37.00 :: 39.07
Swimmer
332.6 =
296.9
Hopper
1201.2 = 90.8
1269.9 +: 752.9
771.1 1: 19.8
818 1: 394.2
-931.7 2 15.4
-976.8 2: 73.1
Walker
-7.0
0.8
7.5
6..3
-00.4
Halfcheetal
2706.2 +: 1170.4 1921.56 4 821.34
1834 4: 434.87
1957.5
581.6
987.90 4: 435.90
1022.1 619.8
Notably, there is consistent trend regarding whether Li control has greater impact during testing or training phases. The primary conclusion drawn from this ablation study in conjunction with 3- is that L, augmentation yields the greatest benefits when applied to both training and testing. One possible explanation for this observation is that such consistent augmentation avoids shift in the policy distribution, leading to desired performance.
Next, in Fig. 5] we report the learning curves of the main result.
19

---

Published as a conference paper at ICLR 2024
Inverted Pendulum
Inverted Pendulum 0.1
Inverted Pendulum de 0.1
-50
M
Episode Return
Episode Return
MM
-100
-100
-100
Episode
-150
-150
-150
-200
200
200
METRPO
L1
50
100
150
200
50
100
150
200
50
100
150
200
Steps
(x10
Environment Steps (x10=))
Environment Steps (x10*)
Swimmer
Swimmer o, 0.1
Swimmer de =- 0.1
400
200
300
Episode Return
Episode Return
200
100
Episode
100
50
550
50
100
150
200
50
100
150
200
100
150
200
Enviranment (x(DFF)
Enviranment Steps (x1DF)
Environment Steps (x10
Hopper
Hopper ds = 0.1
Hopper = 0.1
Episode Return
Episode Retur
Episade Retur
-500
1000
1000
1000
-1500
-1000
-1000
-000
-2000
2000
2000
-300
50
100
200
100
150
200
100
150
Environment Steps (x10²)
Environment Steps (x10*)
Environment Steps (x10*)
o
da
-5
Episode Reaurn
-5
Episode Retum
-10
-10
-10
-15
Episode
-15
-15
-20
-25
-20
-20
-25
-25
-35
50
200
150
Enviranment Steps (x107)
Environment Steps (x10²)
Environment Steps (x10")
Halfcheetah
Halfcheetah
4000
4000
2500
2000
Episade Raturn
Episode Return
Episode Return
1500
2000
2000
1000
1000
-500
Environment Steps (x:1D))
Environment Steps (x110))
Environment Steps (x10*)
Figure 5: Plots of L1 -METRPO learning
function of episodic steps. The performance averaged across multiple random seeds such that the solid lines indicate the average return at the corresponding timestep, and the shaded regions indicate one standard deviation.
20

---

Published as a conference paper at ICLR 2024
Inverted Pendulum
Inverted Pendulum 0.1
Inverted Pendulum da 0.1
Episode Return
Episode Return
-100
-100
-100
Episode
-150
-150
-150
-20
MBNF
-200
200
MBNF
L.
50
100
150
200
50
100
150
200
0
50
100
150
200
Environment Steps (x10=)
Environment Steps (x100))
Environment Steps (x10²)
Swimmer
Swimmer o, 0.1
Swimmer oo= 0.1
400
400
300
Episode Return
300
Episode Return
200
200
Episode
100
100
-100
50
100
150
200
50
100
150
200
100
150
200
Enviranment Steps (x(1FF)
Enviranment Steps (x1D=)
Environment Steps (x10*)
Hopper
Hopper d, = 0.1
Hopper de 0.1
1000
1000
1000
Episode Retur
Episode Retur
Epicade Return
-1000
-1000
-1000
-2000
-2000
-2000
-3000
-3000
-3000
50
100
200
50
100
150
200
100
150
200
Environment Steps (x (x10P)
Environment Steps (x10²)
Environment Steps (x10*)
Walker a
Walker aa
Episode Retur
Epicode Retum
-2000
-2000
-2000
Episode
3000
3300
-4000
-4000
-4000
-5000
5000
-5000
5.0
200
100
150
Enviranment Steps (x107)
Environment Steps (x10")
Environment Steps (x10")
Halfchectah
Halfcheetah d,
Halfcheetah
1600
1000
1000
500
Episode Return
500
Episade Raturn
500
Ep.sode
-500
-500
-500
1100
1000
-1000
50
150 Environment Steps (x10*)
Environment Steps (x(10=)
Environment Steps (x102)
Figure 6: Plots of L1 -MBMF learning curves as function of episodic steps. The evaluation of the performance is identical to L -METRPO.
21

---

Published as a conference paper at ICLR 2024
D.4 COMPARISON WITH PROBABILISTIC MODELS
DETS-L2 PETS
Eptende Rtaumm
-100
-150
-200
Steps
Figure 7: Plots of L1 -DETS vs PETS learning curves as function of episodic steps.
Probabilistic models, as discussed in (Chua et al. 2018 Wang & Ba 2020), offer common ap- proach in Reinforcement Learning (RL) to tackle model uncertainty. contrast, our approach, cen- tered robust controller, shares similar spirit but differs in architecture. While previous works directly integrate uncertainty into decision-making, for example, through methods like sampling- based Model Predictive Control (MPC) (Chua al. 2018), approach takes unique path by decoupling the process. We address uncertainty by explicitly estimating and mitigating it based on the learned deterministic nominal dynamics, allowing the MBRL algorithm operate intended.
Recently, the authors in (Zheng et al. 2022) emphasized that the empirical success of probabilistic dynamic model ensembles is attributed their This observation led to the hypothesis that the ensemble's key functionality is to regularize the Lipschitz constant of the value function, not in its probabilistic formulation. The authors have shown that the predictive quality of deterministic models does not show much difference with probabilistic (ensemble) models, leading the conclusion that deterministic models can offer computational efficiency and practicality for many MBRL scenarios. In this context, our work exploits the practical advantages of using deterministic models, while L, adaptive controller accounts for the randomness present in the environment.
heguehitt-regulariiingg aspect the value functions.
In Fig. 7 we conducted supplementary experiments comparing PETS and its deterministic coun- terpart, DETS, with L1 augmentation. The results were obtained using multiple random seeds and 200,000 timesteps in the Inverted Pendulum environment with an action noise of o, = 0.3, demonstrating that the deterministic model with L, augmentation (C1 -DETS) outperform the probabilistic model approach (PETS). However, it's important to note that this comparison is spe- eific to environment. We refrain from making broad claims regarding DETS's superiority over conclusion, we express our intent
PETS without further in-depth analysis and experimentation. to explore the development of & -MBRL that effectively work alongside probabilistic models, recognizing the potential advantages of both approaches.
22


Reviews:
[{"review_id": "tLqhiwd1JR", "reviewer": "Reviewer_NiYe", "summary": "This paper proposes a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. This method is designed to enhance the robustness of the MBRL system against uncertainties, using MBRL perturbed by the L1 adaptive control.", "questions": "1. What is the extension over:\n\nAnnaswamy, A. M., Guha, A., Cui, Y., Tang, S., Fisher, P. A., & Gaudio, J. E. (2023). Integration of adaptive control and reinforcement learning for real-time control and learning. IEEE Transactions on Automatic Control.\n\n2. Please carefully show differences with the following paper. There also seems to be a big overlap with:\n\nKim, J. W., Park, B. J., Yoo, H., Oh, T. H., Lee, J. H., & Lee, J. M. (2020). A model-based deep reinforcement learning method applied to finite-horizon optimal control of nonlinear control-affine system. Journal of Process Control, 87, 166-178.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper attempts to balance theory and empirical validation. It aims to integrate Model-Based Reinforcement Learning with control.", "weaknesses": "The paper has flaws both with theory and experiment.\n\n  1. In theory, there is so much technically that is incorrect.  Some issues are highlighted below.\n\n  2. The evaluation does not compare the proposed method with recent competitors, such as:\n\nAnnaswamy, A. M., Guha, A., Cui, Y., Tang, S., Fisher, P. A., & Gaudio, J. E. (2023). Integration of adaptive control and reinforcement learning for real-time control and learning. IEEE Transactions on Automatic Control.\n\nKim, J. W., Park, B. J., Yoo, H., Oh, T. H., Lee, J. H., & Lee, J. M. (2020). A model-based deep reinforcement learning method applied to finite-horizon optimal control of nonlinear control-affine system. Journal of Process Control, 87, 166-178.\n\nThe paper is littered with inconsistencies and half-truths that make it difficult to follow. \n\nA. The paper has serious misunderstandings concerning the relationships among non-linear or affine models and switching. The paper makes claims that mis-represent the following:\n\n1. In control theory, using a switching law is independent of whether the system is non-linear or affine.\n2. An affine model can be tuned to be a close approximation to a non-linear model.\n3. A well-tuned affine model can be compatible with the underlying MBRL algorithm.\n\nThe text around eq. 10-11 is standard control theory: we tune an affine model for a set of operating conditions, and have a switching function that switches to the best-approximation affine model as appropriate.\n\nIt is confusing to try to follow the logic of p. 5, starting with \"Although using the above naive control-affine model can be convenient, it must trade in the capabilities of the underlying MBRL algorithm.\"\n\nThere are many overblown claims. such as: \n\n  (i) \"Classical control tools rely on extensively modeled dynamics that are gain scheduled, linear, and/or true up to parametric uncertainties. An example is prohibitively expensive wind-tunnel modeling for designing flight control systems (Neal et al., 2004; Nichols et al., 1993).\"\n\nThis is not true. Many modern approaches use nonlinear methods. Your example makes no sense.\n\n  (ii) \"MBRL algorithms often use highly nonlinear models (often NNs) that do not have true parameters corresponding to the ground truth dynamics, only optimal from a predictive sense, which makes consolidating MBRL with control theoretic tools challenging.\"\n\nThe following is over 10 years old and covers many topics in (ii):\n\nWang, X., & Hovakimyan, N. (2012). L1 adaptive controller for nonlinear time-varying reference systems. Systems & Control Letters, 61(4), 455-463.\n\n\np.4: \"it is necessary to represent the nominal model in the control-affine form.\"\n\nIt is not required to have affine form, even for NNs:\n\nPadhi, R., Unnikrishnan, N., & Balakrishnan, S. N. (2007). Model-following neuro-adaptive control design for non-square, non-affine nonlinear systems. IET Control Theory & Applications, 1(6), 1650-1661.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJdH56G7B2", "reviewer": "Reviewer_QoUP", "summary": "This paper presents an improvement to existing model based RL algorithms where an approximate dynamics model is used in an L1 adaptive control scheme to contribute to the robustness of the algorithms. By relying on an \"affinized\" a trained nonlinear model using the switching rule, L1 adaptive control can be used in tandem with MBRL algorithms. This method is shown to improve both the sample efficiency and robustness of MBRL algorithms.", "questions": "Why L1 adaptive control? What were some of the other options from the adaptive control community considered in this work?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The statement of contributions is nice for understanding which parts of the proposed work are original and which already exist.\n2. There is a nice literature review in the related work and preliminaries sections to introduce L1 adaptive control to people less familiar with it. \n3. The approach was shown to work well on a wide variety of control tasks.", "weaknesses": "1. Without a solid adaptive control background, I found this paper very hard to follow. \n2. Some of the figure captions weren't descriptive enough, more detail would be helpful. \n3. This paper seemed to be very heavy on math in place it made it hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0fW3HMWMBf", "reviewer": "Reviewer_26Jm", "summary": "The paper proposes robust control method combining existing MBRL algorithms and $\\mathcal{L}_1$ adaptive control method. MBRL algorithm provides the reference control and $\\mathcal{L}_1$ adaptive control is applied to reject the uncertainty.\n \nThe difficulty in combining MBRL and adpative control method is how to represent the model. The authors have chosen control-affine dynamics using Taylor approximation and switching law, and it seems to work well.", "questions": "- What is the importance of Theorem 1 or what does it have to do with MBRL? What is the difference with the existing stability result in $\\mathcal{L}_1$-adaptive control theory?\n\n- In equation (9), how is $\\nabla_u \\hat{f}_{\\theta}(x_t,u)$ implemented?\n\n- What does it mean to skip the update in non-differentiable points? Then should the update is over when it the agorihm meets the non-differntiable point?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The idea to use MBRL as reference control and then applying $\\mathcal{L}_1$ adaptive control seems to straightforward and clear. The presentation is clear and well-organized.\n\n- The method is applicable to wide range of MBRL algorithms.\n\n- The experimental results show the effectiveness of the proposed method.", "weaknesses": "- Using output of the RL algorithm as reference input for $\\mathcal{L}_1$ adaptive control has been explored in Cheng et al., therefore the proposed method does not seem to be novel.\n\n- Since the suggested method is an augmentation framework, the performnace of the method highly depends on existing MBRL algorithms. \n\n- Since the true model is assumed to be deterministic, it highly restricts the range of application.\n\n- Comparison with existing uncertainty dealing methods seems to insufficient. I would expect several more explanations and experiments comparing with probabilistic models as in Section D.4.\n\n\nCheng, Yikun, et al. \"Improving the Robustness of Reinforcement Learning Policies With ${\\mathcal {L} _ {1}} $ Adaptive Control.\" IEEE Robotics and Automation Letters 7.3 (2022): 6574-6581.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4V0FpTFVib", "reviewer": "Reviewer_QEpj", "summary": "The authors propose an $\\mathcal{L}_1$ adaptive control \"add-on\" that can be applied to existing model-based RL algorithms in order to improve robustness to model uncertainties. This scheme is agnostic to the choice of model-based RL algorithm; it transforms/filters said algorithm's control outputs before they are input to the dynamical system or simulator. The authors provide some theory behind this approach as well as experimental validation in MuJoCo environments.", "questions": "As a sanity check, how does this perform on linear systems?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is written very well, which is rarely a given.\n- The authors spend an appropriate amount of space on each section of the paper and do not belabor introductory material or time over-explaining things.\n- The (meta-)algorithm is a nice idea that can be tested fairly easily.\n- Code to replicate experiments was provided, which is rarely a given.", "weaknesses": "Some presentation issues:\n- The single-column \"wrapped\" figures are awkward, especially on pg. 6.\n- Tables (esp. numbers) should have larger fonts--shouldn't be below footnote size.\n- In Table 1 and Fig. 3, what the bounds/error bars represent isn't stated.\n#\n- I would have liked to see on-off comparisons for more model-based RL algorithms as in Figure 3 (especially in the main body of the paper), as it's one of the paper's main selling points.\n- While I appreciate the installation README, having one of the steps being \"go get a MuJoCo license\" isn't very user-friendly (not sure why it couldn't have been included?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)