Enforcing robust control guarantees within neural network policies
Authors: Priya L. Donti
Melrose Roderick
Mahyar Fazlyab
J Zico Kolter
Venue: ICLR 2021 Poster (2021)
Keywords: robust control
reinforcement learning
differentiable optimization

Abstract: When designing controllers for safety-critical systems, practitioners often face a challenging tradeoff between robustness and performance. While robust control methods provide rigorous guarantees on system stability under certain worst-case disturbances, they often yield simple controllers that perform poorly in the average (non-worst) case. In contrast, nonlinear control methods trained using deep learning have achieved state-of-the-art performance on many control tasks, but often lack robustness guarantees. In this paper, we propose a technique that combines the strengths of these two approaches: constructing a generic nonlinear control policy class, parameterized by neural networks, that nonetheless enforces the same provable robustness criteria as robust control. Specifically, our approach entails integrating custom convex-optimization-based projection layers into a neural network-based policy. We demonstrate the power of this approach on several domains, improving in average-case performance over existing robust control methods and in worst-case stability over (non-robust) deep RL methods.

Published as a conference paper at ICLR 2021
ENFORCING ROBUST CONTROL GUARANTEES WITHIN NEURAL NETWORK POLICIES
Priya L. Donti', Melrose Roderick', Mahyar Fazlyab², J. Zico Kolter Carnegie Mellon University, JJohns Hopkins University, "Bosch Center for Al (pdenti, mroderick] Bemu.edi mahyarfaz jau. edu, zkolterêcs cmu, edu
ABSTRACT
When designing controllers for safety-critical systems, practitioners often face a challenging tradeoff between robustness and performance. While robust control methods provide rigorous guarantees on system stability under certain worst-case disturbances, they often yield simple controllers that perform poorly in the aver- age (non-worst) case. In contrast, nonlinear control methods trained using deep learning have achieved state-of-the-art performance on many control tasks, but often lack robustness guarantees. In this paper, we propose technique that com- bines the strengths of these two approaches: constructing generic nonlinear con- trol policy class, parameterized by neural networks, that nonetheless enforces the same provable robustness criteria as robust control. Specifically, our approach en- projection layers into neural
tails integrating custom convex-optimize sonvee-optimization-bsss network-basee policy. We demonstrate the power of this approach on several do- mains, improving in average-case performance over existing robust control meth- ods and in worst-case stability over (non-robust) deep RL methods.
INTRODUCTION
The field of robust control, dating back many decades, has been able provide rigorous guarantees on when controllers will succeed or fail in controlling a system of interest. In particular, if the uncer- tainties in the underlying dynamics can be bounded specific ways, these techniques can produce controllers that are provably robust even under worst-case conditions. However, as the resulting policies tend to be simple (i.e., often linear), this can limit their performance in typical (rather than worst-case) scenarios. In contrast, recent high-profile advances in deep reinforcement learning have yielded state-of-the-art performance on many control tasks, due to their ability to capture complex, nonlinear policies. However, due lack of robustness guarantees, these techniques have still found limited application in safety-critice domains where an incorrect action (either during training or at runtime) can substantially impact the controlled system.
In this paper, we propose a method that combines the guarantees of robust control with the flexibility of deep reinforcement learning (RL). Specifically, we consider the setting of nonlinear, time-varying systems with unknown dynamics, but where (as common in robust control) the uncertainty on these dynamics can be bounded in ways amenable to obtaining provable performance guarantees. Building upon specifications provided by traditional robust control methods these settings, we construct a new class of nonlinear policies that parameterized by neural networks, but that nonetheless provably robust. In particular, we project the outputs of a nominal (deep neural newwork-bssedd controller onto space of stabilizing actions characterized by the robust control specifications. The resulting nonlinear control policies are trainable using standard approaches in deep RL, yet are guaranteed be stable under the same worst-case conditions as the original robust controller.
We describe our proposed deep nonlinear control policy class and derive efficient, differentiable projections for this class under various models of system uncertainty common in robust control. We demonstrate our approach on several different domains, including synthetic linear differential inclusion (LDI) settings, the cart-pole task, quadrotor domain, and microgrid domain. Although these domains are simple by modern RL standards, we show that purely RL-based methods often produce unstable policies in the presence of system disturbances, both during and after training. In contrast, we show that our method remains stable even when vorst-case disturbances are present, while improving upon the performance of traditional robust control methods.

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
We employ techniques from robust control, (deep) RI., and differentiable optimization to learn prov- ably robust nonlinear controllers. We discuss these areas of work in connection to our approach.
Robust control is concerned with the design of feedback controllers for dynamical systems with modeling uncertainties and/or external disturbances (Zhou and Doyle 1998 Bayar and Bernhard 2008), specifically controllers with guaranteed performance under worst-case conditions. Many classes of robust control problems in both the time and frequency domains can be formulated using linear matrix inequalities (LMIs) (Boyd et al.) 1994 Kothare et al. 1996); for reasonably-sing problems, these LMIs can be solved using off-the-shell numerical solvers based on interior-poin or first-order (gradient-based) methods. However, providing stability guarantees often requires the use of simple (linear) controllers, which greatly limits average-cass case performance. Our work seeks to improve performance via nonlinear controllers that nonetheless retain the same stability guarantees.
Robust control.
Reinforcement learning (RL). In contrast, RL (and specifically, deep RL) is not restricted to simple controllers or problems with uncertainty bounds on the dynamics. Instead, deep RL seeks to learn an optimal control policy, represented by neural network, by directly interacting with an unknown environment. These methods have shown impressive results in a variety of complex control tasks (e.g., Mnih al. (2015); Akkaya al. (2019) ); see Buçonnu et al. (2018) for survey. However, due to its lack of safety guarantees, deep RL has been predominantly applied to simulated environments or highty-controlled real-world problems, where system failures are either not costly or not possible.
Efforts to address the lack of safety and stability in RL fall into several main categories. The first tries to combine control-theoretic ideas, predominantly robust control, with the nonlinear control policy benefits of RL (e.g.. Morimoto and Doya (2005); Abb-KKhllat
al. (2006); Feng al. (2009); Liu ct al. )2013); Wu and Luo (2013); Luo et al. (2014) Friedrich and Buss] (2017): Pinto et al. (2017); 7); Jin and Lavaei 2018); ; Chang al. (2019); Han et al. (2019); Zhang et al. (2020)). For example, RL has been used to address stochastic stability in Ho control synthesis settings by jointly learning Lyapunov functions and policies in these settings (Han al. 2019). As another example, RL has been used to address Hoo control for continuous-time systems via min-max differential games, in which the controller and disturbance are the "minimizer" and "maximizer" (Morimoto and Doya, 2005). We view our approach as thematically aligned with this previous work, though our method able to capture not only 3 settings, but also a much broader class of robust control settings.
Another category of methods addressing this challenge is safe RL, which aims to learn control poli- cies while maintaining some notion of safety during or after learning. Typically, these methods attempt to restrict the RL algorithm to safe region of the state space by making strong assump- tions about the smoothness of the underlying dynamics, c.g., that the dynamics can be modeled as Gaussian process (GP) Turchetta et al. 2016 Akkametauu al. (014) are Lipschitz continu- ous (Berkenkamp et al. 2017 Wachi et al. 2018). This framework is in theory more general than our approach, which requires using stringent uncertainty bounds (e.g. state-control norm bounds) from robust control. However, there are two key benefits to our approach. First, norm bounds or polytopic uncertainty can accommodate sharp discontinuities the continuous-time dynamics. Second, con- vex projections (as used in our method) scale polynomially with the state-action size, whereas GPs in particular scale exponentially (and are therefore difficult to extend high-dimensional problems).
A third category of methods uses Constrained Markov Decision Processes (C-MDPs). These meth- ods seek to maximize a discounted reward while bounding some discounted cost function (Altman 1999 Achiam et al. 2017, Taleghan and Dietterich 2018 Yang et al.) (020). While these methods do not require knowledge of the cost functions a-priori, they only guarantee the cost constraints hold during test time. Additionally, using C-MDPs can yield other complications, such as optimal policies being stochastic and the constraints only holding for a subset of states.
Differentiable optimization layers. A great deal of recent work has studied differentiable op- timization layers for neural networks: e.g., layers for quadratic programming {Amos and Kolter 2017]. SAT solving (Wang et al. 2019), submodular optimization (Djolonga and Krause 2017 Tschiatschek et al. 20118), cone programs (Agrawal et al. 2019), and other classes of optimization problems (Gould et al. 2019). These layers can be used to construct neural networks with useful inductive bias for particular domains to enforee that networks obey hard constraints dictated by the settings in which they are used. We create fast, custom differentiable optimization layers for the latter purpose, namely, to project neural network outputs into set of certifiably stabilizing actions.
2

---

Published as a conference paper at ICLR 2021
3 BACKGROUND ON LQR AND ROBUST CONTROL SPECIFICATIONS
In this paper, our aim is to control nonlinear (contimuous-time) dynamical systems of the form
#(() E A())((() + B(I)((() + G())w(():
where x(t) E Iº denotes the state at time t; u(t) E R is the control input; w(t) I2 captures both external (possibly stochastic) disturbances and any modeling discrepancies; (tt) denotes the time derivative of the state x: at time I; and A()) R*xs BB()= R G(l) E R This class of models is referred to as linear differential inclusions (LDis); however, we note that despite the name, this class does indeed characterize nonlinear systems, as, e-g., w(() can depend arbitrarily on ar(t) and u(t) (though we omit this dependence in the notation for brevity). Within this class of models, it is often possible construct robust control specifications certifying system stability. Given such specifications, our proposal is to learn nonlinear (deep neural network- based) policies that provably satisfy these specifications while optimizing some objective of interest. We start by giving background on the robust control specifications and objectives considered in this work.
3.1 ROBUST CONTROL SPECIFICATIONS
In the continuous-time halinite-horizon settings we consider here, the goal of robust control is often to construct a time-invariant control policy u(t) π (xx(t)), alongside some certification that guar- antees that the controlled system will be stable (i.e., that trajectories of the system will converge to an equilibrium state, usually x 0 by convention; see Haddad and Chellaboina (2011) for a more formal definition). For many classes of systems this positive definite Lyapunov function V : R + R. with (()) 0 and V(x) >> 0 for all x o 0, such that the function decreasing along trajectories for instance,
this certification typically the form of a
V(x(()) S aaV(((())
for some design parameter > 0. (This particular condition implies exponential stability with rate of convergence a. For certain classes of bounded dynamical systems. iinerinvariant linear control policies u(t) Kr (t),), and quadratic Lyapunov functions V(z) = xx" Pz, it is possible to construct such guarantees using semidefinite programming. For instance, consider the class of norm-bounded L.IS (NLDIs)
a Ax(() + Bu(l) + Gw((), [|w(()]22 < ||Cx(() Du,
where E R B G E Rx C Rxx and D E kx are time-invariant and known, and the disturbance w() is arbitrary (and unknown) but obeys the norm bounds above." For these systems, possible to specify set of stabilizing policies via a set of linear matrix iequulltiies (LMIs, [oyyd et al. (1944))
AS + SAT + PGGT +B +YTBB +as SCT -TTDT CS DY
3 0, S 0, #>0, 0,
(4)
-ul
where S' E Raxs and Y E Baxx For matrices S and Y satisfying (4). K = YS-1 and P = S-1 then a stabilizing linear controller gain and Lyapunov matrix, respectively. While the LMI above is specific to NLDI systems, this general paradigm of constructing stability specifications using LMIs applies to many settings commonly considered in robust control (e.g., settings with norm- bounded disturbances or polytopic uncertainty, or H control settings). More details about these types of formulations are given in, e-g., Boyd et al. (1994); in addition, we provide the relevant LMI constraints for the settings we consider in this work in Appendix A
In this work, we consider sub-classes of system (T) that may indeed be stochastic (e.g., due to stochastic external disturbance w(f)), but that can he bounded so to be amenable to deterministic stability analysis. However, other settings may require stochastic stability analysis; please see Astromt (197I).
"See, e.g.. Haddad and Chellaboina (2011) for a more rigorous definition of (local and global) exponential stability. Condition (2) comes from Lyapunov's Theorem, which characterizes various notions of stability using Lyapunov functions.
A slightly more complex formulation involves an additional term in the norm bound, i.c., Ca(() + Du(t) + Hw((), which creates quadratic inequality in w. The mechanics of obtaining robustness specifications in this setting are largely the same as presented here, though with some additional terms in the equations. As such, as is often done, we assume that H = 0 for simplicity.
3

---

Published as a conference paper at ICLR 2021
3.2 LQR CONTROL OBJECTIVES
In addition to designing for stability, it is often desirable to optimize some objective characterizing controller performance. While our method can optimize performance with respect any arbitrary cost or reward function, to make comparisons with existing methods easier, for this paper we con- sider the well-known infinite- horizon "linear quaeertt regulator" (LQR) cost, defined as
J (z(1)) + u(t) Ru(()) dt,
(5)
for some Q E S ž 0 and R E gaxa > 0. If the control policy is assumed to be invariaat and linear as described above (i.e., u(t) (t)).. minimizing the LQR cost subject to stability con- straints can be cast as SDP (see, e.g., Yao et al. (200T)) and solved using off-the-shelf numerical solvers- - fact that we exploit in our work. For example, to obtaín an optimal linear time-invariant controller for the NLDI systems described above, we can solve
minimize tr(QS) + t R!/2) S. t. Equation S,Y
holds.
(6)
4 ENFORCING ROBUST CONTROL GUARANTEES WITHIN NEURAL NETWORKS
We now present the main contribution of our paper: A class of nonlinear control policies, poten- tially parameterized by deep neural networks, that is guaranteed to obey the same stability condi- tions enforced by the robustness specifications described above. The key insight of our approach is as follows: While it is difficult to derive specifications that globally characterize the stability of generic nonlinear controller, if we are given known robustness specifications, we can create a suf- ficient condition for stability by simply enforcing that our policy satisfies these specifications at all 1. For instance, given a known Lyapunov function, we can enforce exponential stability by ensuring that our policy sufficiently decreases this function (e.g., satisfies Equation (2) at any given (:))
In the following sections, we present our nonlinear policy class, as well as our general framework for learning provably robust policies using this policy class. We then derive the instantiation of this framework for various settings of interest. In particular, this involves constructing (custom) differentiable projections that can be used to adjust the output of a nominal neural network to satisfy desired robustness criteria. For simplicity of notation, we will often suppress the t-dependence of x, u, and w. but we note that these are continuous- time quantities as before.
4.1 PROVABLY ROBUST NONLINEAR POLICY CLASS
Given dynamical system of the form D and quadratic Lyapunov function V(x) = =x Px, let
C(x) = {u E IR" (x)) -aV(x) Vi E A(I)= + B(I)u G(I)w}
(7)
denote a set of actions that, for fixed state x E R', are guaranteed to satisfy the exponential stability condition (2) (even under worst-case realizations of the disturbance w). We note that this
"safe" set is non-empty if P satisfies the relevant LMI constraints (e.g., system (4) for NLDIs)
characterizing robust linear time-invariant controllers, as there is then some K corresponding to P such that Kr E C(x) for all states ..
Using this set of actions, we then construct a robust nonlinear policy class that projects the output of some neural network onto this set. More formally, consider arbitrary nonlinear (neural network- based) policy class मे : I²= > R² parameterized by 0, and let Pe. denote the projection operator for some set (.). We then define our robust policy class as Te : R - Raa where
(π()) = (
We note that this policy class is differenniable if the projections can be implemented in a differen- tiable manner (e.g., using convex optimization layers (Agrawal al., 2019), though construct efficient custom solvers for our purposes). Importantly, as all policies in this class satisfy the stabil- ity condition 2 for all states and all times t, these policies are certifiably robust under the same conditions as the original (linear) controller for which the Lyapunov function V(x) was constructed.
Given this policy class and some performance objective (e.g., LQR cost), our goal is to then find parameters 0 such that the corresponding policy optimizes this objective - i.e., to solve
minimize 100 e(x, To(x)) ) dt s.t. i EE A(t)x + B(t)To(z) G(t)..
(9)
4

---

Published as a conference paper at ICLR 2021
Algorithm Learning provably robust controllers with deep RL
1: input performance objective
I/ e.g. LQR cost
2: input stability requirement
I/ e.g. V(x) S (a)(x
3: input policy optimizer A
e.g. a planning or RI. algorithm I/ e.g. by optimizing (6)
4: compute P. K satistying LMI constraints
5: construct specifications ((x) using P
// as defined in Equation
6: construct robust policy class me using C
Il as defined in Equation
7: train ne via A to optimize Equation |
8: return n
Since Ty is differentiable, we can solve this problem via a variety of approaches, e.g., a model- based planning algorithm if the true dynamics are known, or virtually any (deep) RL algorithm if the dynamics are unknown.
This general procedure for constructing stabilizing controllers is summarized in Algorithm I While seemingly simple, this formulation presents powerful paradigm: by simply transforming the output of neural network, while ensuring the resultant policy will stabilize the system during both training and testing.
employ expressive policy class to optimize objective of interest
We instantiate our framework by constructing "safe" sets C(x) and their associated (differentiable) projections Po(e)) for three settings of interest: NLDIs, polytopic linear differential inclusions (PLDIs), and H so control settings. As an example, describe this procedure below for NLDIs, and refer readers to Appendix B for corresponding formulations for the additional settings we consider.
4.2 EXAMPLE: NLDIS
In order to apply our framework to the NLDI setting B we first compute a quadratic Lyapunov function Vxx) x Px by solving the optimization problem (6) for the given system via semidef- inite programming. We then use the resultant Lyapunov function to compute the system-specing "safe" set C(x), and then create fast, custom differentiable solver project onto this set.
4.2.1 COMPUTING SETS OF STABILIZING ACTIONS
Given P. compute (xL)(()) the set of actions R that, for each state R", satisfy the stability condition 2 at that state under even a worst-cas realization of the dynamics (i.e., this case, under worst-case disturbance w). The form of the resultant set is given below.
Theorem 1. Consider the NLDI system 33, some stability parameter & > O, and a Lyapumov function ((x) = x. Px. with P satisfying Equation Assuming P exists, define
-FTPB x² 12PA+aaPja
COLDI(
E TR ||Cx Dull2 |G
2| Pxl12
for all stutes E Rs. For all CNLI((a)) is non-emply set of actions that satisfy the exponential stubility condition 2 Further, (NL is a convex set in u. Proof. We seek to find a set of actions such that the condition trajectories of A of actions satisfying this condition a given is given by
2 is satisfied along all possible
CNLDI( = L R
sup Ca+
v(x) ((x) -aV
Let S = (w: ||wl|2 I ||Cr + Dull2). We can then rewrite the left side of the above inequality as sup wES
V(x)
sup T Px + x TPP = 2x P(Ax + Bu) + sup
2x PGw
wES
weS
= 2xx P(AT Bu) + 2 Prl12l|Cr + Dull2:
by the definition of the NLDI dynamics and the closed-form minimization of linear term over an L2 ball. Rearranging yields an inequality of the desired form. We note that by definition of the
While this problem is anfinie-hhoizzn and continuous in time, in practice, one would optimize it in discrete time over large finite time horizon.
5

---

Published as a conference paper at ICLR 2021
specifications 4
there is some K corresponding to P such that the policy Kw satisfies the exponential stability condition (2): thus, Kr E CNLDI. and CNLDI is non-empty. Further, as the above inequality represents second-order cone constraint in u, this set is convex in u.
M
We further consider the special case where D 0, i.e., the norm bound on w does not depend on the control action. This form of NLDI arises in many common settings (e.g., where characterizes lincarization error in a nonlinear system but the dynamics depend only linearly on the action), and is one for which we can compute the relevant projection in closed form (as described shortly).
Corollary 1.1. Consider the NLDI system (3) with D = O, some stability parameter a > 0, and Lyapunov function ((x) xx Pr with P satisfying Equation (4). Assuming P exists, define
CNLDI-O(I)) = {uu E Rd 2Bu S - (2PA+ aP)x 2/|G" Pal12l|C:1112
for all states R. For all CNLDIO(Z) is non-empty set of actions that satisfy the exponential stability condition 2 Further, CNLDIO( 3) convex set in u.
Proof. The result follows by setting D = 0 in Theorem | and rearranging terms. As the above inequality represents a linear constraint in u, this set convex u.
4.2.2 DERIVING EFFICIENT, DIFFERENTIABLE PROJECTIONS
For the general NLDI setting B) we note that the relevant projection Pexu(()) (see Theorem represents a projection onto a second-order cone constraint. As this projection does not necessarily have a closed form, must implement it using differentiable optimization solver (e.g.. Agrawal et al. ). For computational efficiency purposes, we implement custom solver that employs an accelerated projected dual gradient method for the forward pass, and employs implicit differentiation through the fixed point equations of this solution method to compute relevant gradients for the backward pass. Derivations and additional details are provided in Appendix C
In the where D (see Corollary 1.1 note that the projection operation Pesmaaaa does have a closed form, and can in fact be implemented via single ReLU operation. Specifically, defining n² : 2x PB and Ç := -x" (2PA+ aaP) 2 Pxxla||Car we that
Panaoca(s) (f(x)) = (xx)
*n) *(x) ReLU ( n (n"(()) (10)
(xx) n ]Ta(s)) otherwise
5 EXPERIMENTS
Having instantiated our general framework, we demonstrate the power of our approach on a variety of simulated control domains In particular, we evaluate performance on the following metrics:
Average-case performance: How well does the method optimize the performance objec- tive (i.e., LQR cost) under average (non-worst case) dynamics?
Worst-case stability: Does the method remain stable even when subjected to adversarial (worst-case) dynamics?
In all cases, we show that our method is able to improve performance over traditional robust con- trollers under average conditions, while still guaranteeing stability under worst-case conditions.
5.1 DESCRIPTION OF DYNAMICS SETTINGS
We evaluate our approach on five NLDI settings: two synthetic NLDI domains, the cart-pole task, quadrotor domain, and microgrid domain. (Additional experiments for PLDI and H. control settings are described in Appendix !}
For each setting, we choose time discretization based on the speed at which the system evolves, and run each episode for 200 steps over this discretization. In all cases except the microgrid setting, we use a randomly generated LQR objective where the matrices 11/2 and R1/2 are drawn i.i.d. from standard normal distribution.
Coode for all experiments
is available at https: thub. com / bab
robust-nn-coo co

---

Published as a conference paper at ICLR 2021
Non-robust Methods
Robust Methods
10s
(D = 0)
NLDI
Loss
10
10
(D * 0)
NLDI
Loss
101
10
Microgrid Quadrotor Cartpole
Loss
NA
10
10a
Loss
10
102
Loss
10
250
500
750
1000
250
500
750
1000
Training epochs
Training epochs
MBP
PPO
RARL
Robust MBP
Robust PPO
Setting:
Original
Adversarial
Figure Test performance over training epochs for all learning methods employed in our experi- ments. For each training epoch (10 updates for the MBP model and 18 for PPO), we report average quadratic loss over 50 episodes, and use "X" to indicate cases where the relevant method became unstable. (Lower loss is better.) Our robust methods (denoted by *), unlike the non-robust methods and RARL, remain stable under adversarial dynamics throughout training.
Synthetic NLDI settings. We generate NLDIs of the form 3 with k = by generating matrices A, B, G, C and D i.i.d. from normal distributions, and producing the disturbance w(() using a randomly-initiaizzed neural network (with its output scaled to satisfy the norm-bound on the disturbance). We investigate settings both where D * 0 and where D = 0. In both cases, episodes are run for seconds a discretization of 0.01 seconds.
a 3, and d
Cart-pole. In the cart-pole task, our goal is to balance an inverted pendulum resting on top of a cart by exerting horizontal forces the cart. For experiments, linearize this system NLDI with * 0 (see Appendix D). and add a small additional randomized disturbance satisfying the NLDI bounds. Episodes
for 10 seconds at discretization of 0.05 seconds.
Planar quadrotor. In this setting, our goal stabilize quadcopter in the wo--dimensional plane by controlling the amount of force provided by the quadcopter's ' right and left thrusters. We linearize this system as an NLDI with = 0 (see Appendix [F), and add small disturbance as the cart-pole setting. Episodes are run for seconds at discretization of 0.02 seconds.
Microgrid. In this final setting, aim to stabilize microgrid by controlling storage device and a solar inverter. We augment the system given in Lam et al. (2016) with LQR matrices and NLDI bounds (see Appendix Fb. Episodes run for 2 seconds at discretization of 0.01 seconds.

---

Published as a conference paper at ICLR 2021
5.2 EXPERIMENTAL SETUP
We demonstrate our approach by constructing robust policy class (8) for each of these settings, and optimizing this policy class via different approaches. Specifically, we construct nominal nonlinear control policy class as Ee (z) = Kx + TT(2), where K' is obtained via robust LQR optimization (6), and where (x) is a feedforward neural network. To construct the projections Pc, we employ the value of P obtained when solving for K. For the purposes of demonstration, we then optimize our robust policy class To (2) = Pe (fo(()) using two different methods:
Robust MBP (ours): A model-based planner that assumes the true dynamics are known.
Robust PPO (ours): An RL approach based on PPO (Schulman et al. 2017) that does not assume known dynamics (beyond the bounds used to construct the robust policy class).
Robust MBP is optimized using gradient descent for 1,000 updates, where each update samples 20 roll-outs. Robust PPO is trained for 50,000 updates, where each update samples 8 roll-outs; we choose the model that performs best on a hold-out set of initial conditions during training. We note that while use PPO for our demonstration, our approach is agnostic to the particular method of training, and can be deployed with many different (deep) RL paradigms.
We compare robust neural network-based method against the following baselines:
Robust LQR: Robust (linear) LQR controller obtained via Equation 6
Robust MPC: robust model-predictive control algorithm (Kothare et al. 1996 based on state-dependent LMIs. (As the relevant LMIs are not always guaranteed to solve, our implementation temporarily reverts the Robust LQR policy when that occurs.)
RARL: The robust adversarial reinforcement learning algorithm (Pinto et al. (017). which trains an RL agent the presence of an adversary. (We note that unlike the other robust methods considered here, this method is not provably robust.)
LQR: A standard non-robust (linear) LQR controller.
MBP and PPO: The non-robust neural network policy class (a(x)) optimized via model- based planner and the PPO algorithm, respectively.
In order to evaluate performance, train all methods on the dynamical settings described in Sec- tion 5.I and evaluate them on two different variations the dynamics:
Original dynamies: The dynamical settings described above ("average case").
Adversarial dynamics: Modified dynamics with an adversarial test-time disturbance w(t) generated to maximize loss ("worst case"). We generate this disturbance separately for each method described above (see Appendix for more details).
Initialization states randomly generated for all experiments. For the synthetic NLDI and mi- crogrid settings, these are generated from a standard normal distribution. For both cart-pole and quadrotor, because our NLDI bounds model linearization error, we must generate initial points within region where this linearization holds. In particular, the linearization bounds only hold for a specified Loc ball, BNLDI, around the equilibrium. We use simple heuristic to construct this ball and jointly find smaller T. ball, Bmit such that there exists level set L. of the Robust LQR Lyapunov function with Bmm C L, S BNLDI (details Appendix HH. Since Robust LQR (and by extension our methods) guaranteed to decrease the relevant Lyapunov function, this guarantees that these methods will never leave BNLDI when initialized starting from any point inside Binit - i.c., that our NLDI bounds will always hold throughout the trajectories produced by these methods.
5.3 RESULTS
Table 1 shows the performance of the above methods. We report the integral of the quadratic loss over the prescribed time horizon on a test set of states, or indicate cases where the relevant method became unstable (i.e., the loss became orders of magnitude larger than for other approaches). (Sam- ple trajectories for these methods are also provided in Appendix H
These results illustrate the basic advantage of our approach. In particular, both our Robust MBP and Robust PPO methods show improved "average-case" performance over the other provably robust methods (namely, Robust LQR and Robust MPC). As expected, however, the non-robust
8

---

Published as a conference paper at ICLR 2021
Robust LQR
Robust
Robust MBP
Robust
Environment
LQR
MBP
PPO
RARL
MPC
PPO
Generie NI.I
0
373
16
21
253
253
27
69
33
(D = 0)
A
unstable
1009
873
unstable
1111
2321
Generic NLDI
0
278
15
82
199
199
147
69
80
(D *
A
unstable
1900
1667
unstable
1855
1669
0
36.3
3.6
7.2
10.2
10.2
8.3
9.7
8.4
Cart-pole
A
unstable
172.1
42.2
47.8
41.2
50.0
16.3
0
5.4
2.5
7.7
13.8
13,8
12.2
11.0
8.3
Quadrotor
A
unstable
545.7
137.6
64.8
unstable"
63.1
25.7
26.5
0
4.59
0.60
0.61
0.73
0.73
0.67
0.61
0.61
Microgrid
A
unstable
0.99
0.92
2.17
7.68
8.91
Table 1: Performance of various approaches, both robust (right) and non-robust (left). We report average quadratic loss over 50 episodes under the original dynamics (O) and under an adversarial disturbance (A). For the original dynamies (O), the best performance for both non-robust methods and robust methods is in bold (lower loss is better). Under the adversarial dynamics (A), we seek to observe whether or not methods remain stable; we use "unstable" to indicate cases where the relevant method becomes unstable (and to denote any instabilities due to numerical, rather than theoretical, issues). Our robust methods (denoted by ") improve performance over Robust LQR and Robust MPC in the average case while remaining stable under adversarial dynamics, whereas the non-robust methods and RARL either go unstable or receive much larger losses.
LQR, MBP, and PPO methods often perform better within the original nominal dynamics, they are optimizing for expected performance but do not need to consider robustness under worst-case scenarios. However, when we apply allowable adversarial perturbations (that still respect dis- turbance bounds), the non-robust LQR, MBP, and PPO approaches diverge or perform very poorly. Similarly, the RARL agent performs well under the original dynamies, but diverges under adversar- ial perturbations in the generic NLDI settings. In contrast, both of our provably robust approaches (as well Robust LQR) remain stable under even "worst-case" adversarial dynamics. (We note that the baseline Robust MPC method goes unstable in one instance. though this is due to numerical instability issues, rather than issues with theoretical guarantees.)
Figure [ additionally shows the performance of all neural network- network-based methods on the test set over training epochs. While the robust and non-robust MBP and PPO approaches both converge quickly to their final performance levels, both non-robust versions become unstable under the ad- versarial dynamics very early in the process. The RARL method also frequently destabilizes during training. Our Robust MBP and PPO policies, entire optimization process, i.e., do not destabilize during either training or testing. Overall, these results show that our method is able to learn policies that are more expressive than traditional robust methods, while guaranteeing these policies will be stable under the same conditions as Robust LQR.
the other hand, remain stable throughout the
6 CONCLUSION
In this paper, we have presented a class of nonlinear control policies that combines the expressive- ness of neural networks with the provable stability guarantees of traditional robust control. This policy class entails projecting the output of a neural network onto a set of stabilizing actions, param- eterized via robustness specifications from the robust control literature, and can be optimized using a model-based planning algorithm if the dynamics are known or virtually any RL algorithm if the dynamics are unknown. We instantiate our general framework for dynamical systems characterized by several classes of linear differential inclusions that capture many common robust control settings. In particular, this entails deriving efficient, differentiable projections for each setting, via implicit differentiation techniques. We show over variety of simulated domains that our method improves upon traditional robust I.QR techniques while, unlike non-robust LQR and neural network methods, remaining stable even under worst-case allowable perturbations of the underlying dynamics.
We believe that our approach highlights the possible connections between traditional control meth- ods and (deep) RL methods. Specifically, by enforcing more structure in the classes of deep net- works we consider, it is possible to produce networks that provably satisfy many of the constraints that have typically been thought of as outside the realm of RL. We hope that this work paves the way for future approaches that can combine more structured uncertainty or robustness guarantees with RL, in order to improve performance in settings traditionally dominated by classical robust control.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work was supported by the Department of Energy Computational Science Graduate Fellowship the Center for Climate and Energy Decision Making through cooper- ative agreement between the National Science Foundation and Carnegie Mellon University (SES- 00949710), the Computational Sustainability Network, and the Bosch Center for AL. This material based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE1745016. Any opinions, findings, and conclusions recommenda- tions expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
We thank Vaishnavh Nagarajan, Filipee de Avila Belbute Peres, Anit Sahu, Asher Trockman, Eric Wong, and anonymous reviewers for their feedback on this work.
REFERENCES
Kemin Zhou and John Comstock Doyle. Essentials of Robust Control, volume 104. Prentice hall Upper Saddle River, NJ, 1998.
Tamer Başar and Pierre Bernhard. -op-imai Control and Related Minimax Design Problems: A Dynamic Game Approach. Springer Science & Business Media, 2008
Stephen Boyd, Laurent EI Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear Matrix Inequalities in System and Control Theory, volume 15. Siam, 1994.
Mayuresh V Kothare, Venkataramanan Balakrishnan, and Manfred Morari. Robust constrained model predictive control using linear matrix inequalities. Automatica, 32(10):1361-137, 1996.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529-533 2015.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving Rubik's Cube with Robot Hand. arXiv preprini 1910.00110 2019.
Lucian Buşoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control, 46:8-28, 2018
Jun Morimoto and Kenji Doya. Robust Reinforcement Learning, Neural Computation, 17(2):335- 359, 2005.
Murad Abu-Khalaf Frank L Lewis, and Jie Huang. Policy Iterations on the Hamilon--Jacoii-ssacc Equation for Ho State Feedback Control With Input Saturation. IEEE Transactions on Automatic Control, (12):198991999 2006.
Yantao Feng, Brian DO Anderson, and Michael Rotkowitz. A game thcoretic algorithm to compute local stabilizing solutions to HJBI equations in nonlinear Hos control. Automatica, 45(4):881- 888, 2009.
Derong Liu, Hongliang Li, and Ding Wang. Neural-network-basee zero-sum game for discrete-time nonlinear systems via iterative adaptive dynamic programming algorithm. Neurocomputing. 110: 92-100, 2013.
Huai-Ning Wu and Biao Luo. Simultancous policy update algorithms for learning the solution of linear continuous-time Hov state feedback control. Information Sciences, 222:472-485, 2013.
Biao Luo, Huai-Ning Wu, and Tingwen Huang. Off-Policy Reinforcement Learning for Hoc Control Design. IEEE Transactions Cybernetics, 45(1):65-76 2014.
Stefan R Friedrich and Martin Buss. A robust stability approach robot reinforcement learning based on parameterization of stabilizing controllers. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3365-3372. IEFF, 2017.
10

---

Published as a conference paper at ICLR 2021
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Rein- forcement Learning. In Proceedings of the 34th International Conference on Machine Learning, pages 2817-2826. JMLR. org, 2017.
Ming Jin and Javad Lavaei. Stability-certffiedd reinforcement learning: A control-theoretic perspec- tive. arXiv preprint arXiv: 1505, 2018. 8.
Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Lyapunov Control. In Advances in Neural Information Processing Systems, pages 3245-32544 2019.
Minghao Han, Yuan Tian, Lixian Zhang, Jun Wang, and Wei Pan. Ho Model-free Reinforcement Learning with Robust Stability Guarantee. CoRR, 2019.
Kaiqing Zhang, Bin Hu, and Tamer Basar. Policy Optimization for H2 Linear Control with Hoo Ro- bustness Guarantee: Implicit Regularization and Global Convergence. In Learning for Dynamics and Control, pages 179-190. PMLR, 2020.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe Exploration in Finite Markov Decision Processes with Gaussian Processes. In Advances in Neural Information Processing Systems, 2016.
Anayo K. Akametalu, Shahab Kaynama, Jaime F. Fisac, Melanie Nicole Zeilinger, Jeremy H. Gillula, and Claire J. Tomlin. Reachability-base safe learning with Gaussian processes. In 53rd IEEE Conference on Decision and Control, CDC 2014, 2014.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe Model-based Reinforcement Learning with Stability Guarantees. In Advances in Neural Information Process- ing Systems, 2017.
Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Eitan Altman. Constrained Markov Decision Processes, volume 7. CRC Press, 1999.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained Policy Optimization. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Majid Alkaee Taleghan and Thomas G. Dietterich. Efficient Exploration for Constrained MDPs. In 2018 AAAI Spring Symposia, 2018.
Tsung-Yes Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-Basse Constrained Policy Optimization. In International Conference on Learning Representattinss 2020.
Brandon Amos and Zico Kolter. OptNet: Differentiable Optimization Layer in Neural Net- works. In Proceedings of the 34th International Conference on Machine Learning, pages 136- 145, 2017.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. SATNet: Bridging deep learning and logical reasoning using differentiable satisfiability solver. In Proceedings of the 36th Interna- tional Conference on Machine Learning, pages 6545-6554, 2019.
Josip Djolonga and Andreas Krause. Differentiable Learning of Submodular Models. In Advances in Neural Information Processing Systems, pages 1013-1023, 2017.
Sebastian Tschiatschek, Aytune Sahin, and Andreas Krause. tion. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2731-2738, 2018
Differentiable Submodular Maximiza-
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable Convex Optimization Layers. In Advances in Neural Information Processing Sys- tems, pages 9558-9570, 2019.
11

---

Published as a conference paper at ICLR 2021
Stephen Gould, Richard Hartley, and Dylan Campbell. Deep Declarative Networks: A New Hope.
arXiv preprint 1909: 190.009669, 2019.
Wassim M Haddad and VijaySekhar Chellaboina. Nonlinear Dynamical Systems and Control: A Lyapunov-Based Approach. Princeton University Press, 2011.
Karl Astrom, Introduction to Stochastic Control Theory. Elsevier, 1971.
David Yao, Shuzhong Zhang, and Xun Yu Zhou. A primal-dual semi-definite programming
approach to linear quadratic control. IEEE Transactions Automatic Control, 46(9)) 2001.
Quang Linh Lam, Antoneta luliana Bratcu, and Delphine Riu. Frequency Robust Control in Stand- alone Microgrids with PV Sources: Design and Sensitivity Analysis. 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017.
Hassan K Khalil and Jessy W Grizzle. Nonlinear Systems, volume 3. Prentice Hall Upper Saddle River, NJ, 2002.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course, volume 87. Springer Science Business Media, 2013.
Heinz Bauschke. Projection algorithms and monotone operators. PhD thesis, Dept. of Mathe- matics and Statistics, Simon Fraser University, 1996.
Russ Tedrake. Underactuated robotics: Learning, planning, and control for efficient and agile ma- chines course notes for MIT 6.832. Working draft edition, 3, 2009.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques E. Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction- regularization. The Interna- tional Journal of Robotics Research, page 0278364920949931, 2020.
Roman Kuiava, Rodrigo A Ramos, and Hemanshu R Pota. A New Method to Design Robust Power Oscillation Dampers for Distributed Synchronous Generation Systems. Journal of Dynamic Sys- tems, Measurement, and Control, 135(3), 2013.
12

---

Published as a conference paper at ICLR 2021
A DETAILS ON ROBUST CONTROL SPECIFICATIONS
As described in Section 3.1 for many dynamical systems of the form ① it is possible to specify a set of linear, time-invariant policies guaranteeing infinite-horizon exponential stability via a set of LMIs. Here, we derive the LMI 4 provided in the main text for the NLDI system (3), and additionally describe relevant LMI systems for systems characterized by polytopic linear differential inclusions (PLDIs) and for H control settings.
A.1 EXPONENTIAL STABILITY IN NLDIS
Consider the general NLDI system 3). We seek to design a time-invariant control policy u(t) Kx(tt) and quadratic Lyapunov function V(z) x² Pr with P > 0 for this system that satisfy the exponential stability criterion Vix) < (z) vt. We derive LMI characterizing such controller and Lyapunov function, closely following and expanding upon the derivation provided in Boyd al. (T994).
Specifically, consider the NLDI system
3 reproduced below:
d Az Bu Gw, ||l||2 s |CCr + Dull2-
(A.1)
The time derivative of this Lyapunov function along the trajectories of the closed-loop system is
(x) = :² Px - P² På (Ax Bu + P + Bu Gw)
((A+ BK)x Gw)) + x²' P((A+ + BK)x + Gw)
(A.2)
[x]"
[A+ BKY p P(A+ BK)
PG
w
G
0
w
The exponential stability condition V(x) < -aV(z) is thus implied by inequality
I
M1
(A+ BK T P P(A+ BK) aa
PG
x
0.
(A.3)
w
GTP
w
Additionally, the norm bound on can be equivalently expressed as
M2
[C + DKY - DK)
0
x w
(A.4)
0
Using the S-procedure, it follows that for some 2 0, the following matrix inequality a sufficient condition for exponential stability:
M1 + AM₂ 3 0. Using Schur Complements, this matrix inequality is equivalent
(A.5)
(A + BK) P(A+ BK) oP + (C PGG P3 (A.6)
Left- and right-multiplying both sides by P-1, and making the change of variables S = p-1,
Y KS, and μ 1/2, we obtain
SAT AS + +YTB + BY +as (SC +Y"D") (CS DY) pGG 20. (A.7)
Using Schur Complements again on this inequality, obtain our final system of linear matrix inequalities as
AS + SAT PGGT BY + BT +as SCT DT CS+DV
30, S 0. p> 0. (A.8)
where then K Y and P S-1 Note that the first matrix incquality is homogencous; we can therefore assume μ 1 (and therefore, 1), without loss of generality.
13

---

Published as a conference paper at ICLR 2021
A.2 EXPONENTIAL STABILITY PLDIs
Consider the setting of polytopic linear differential inclusions (PLDIs), where the dynamics are of the form
f(t) = (t))x(t) + B(t)u(t), (A(t), B(t) E B.).
(A.9)
Here, A(t) * and (tt) R2xa can vary arbitrarily over time, as long as they lie in the convex hull (denoted Conv) of the set of points above, where A; E R B, R for 1..... L.
We seek design a time-invariant control policy u(t) Kx(t) and quadratic Lyapunov function
V(x) = x² Par with > for this system that satisfy the exponential stability criterion V(x) <
-oV(x), At. Such a controller and Lyapunov function exist if there exist S E IRex= > 0 and
Y E Maxs such that
A,S + B, + SA? + +Y + a 3 0, Vi Wi=1,... I,..,..,,
(A.10)
where then = YS- and P S The derivation of this LMI follows similarly to that for exponential stability in NLDIs, and is well-described in Boyd et al. (1994).
A.3 Ho CONTROL
Consider the following Ho control setting with linear time- lime-invariant dynamics
+(1) = Ar(I) + Bu(l) + Gwi(), w E L2,
(A.11)
where A, B, and G are time-invariant as for the NLDI case, and where we define La as the set of time-dependent signals with finite C2 norm G
In cases such as these with larger more unstructured disturbances, it may not be possible to guarantee asymptotic convergence to an equilibrium. In these cases, our goal is to construct a robust controller with bounds the extent to which disturbances affect some performance output (e.g.. LQR cost), as characterized by the La gain of the disturbance- cattaanncettoouuuul map. Specifically, consider the stability requirement that this L2 gain be bounded by some parameter > 0 when disturbances are present, and that the system be exponentially stable in the disturbance-free case. This requirement be characterized via the condition that for all and some a 2
E(x, è. u) (x) + (z + a (x" to==ooo==0
+ ue Ru 7 l2) S 0.
(A.12)
We note that when E(r(t), :(t), u(t)) < 0 for all f, both of our stability criteria are met. To see this, note that integrating both sides of (A.12) from 0 to oc and ignoring the non-negative terms on the left hand side after integration yields
Io (x()) Ru(t)) S 7²2 Ru(()))
{ [Ix(t]] + (z(O))
(A.13)
This is precisely the desired bound on the La gain of the disturbance-to-ourpe map (see Khalil and Grizzle (2002)).
)). We also note that in the disturbance-free case, substituting w = into A. yields
-oV ( Qz Qu+ vf Ru) <aV(r,,
(A.14)
where the last inequality follows from the non-negativity of the LQR cost; this is precisely condition for exponential stability.
We now seek design lime-invariant control policy u(t) Kx(t) and quadratic Lyapunov func-
tion V(x) = x Px with P > 0 that satisfies the above condition. In particular, we can write
(t))) ((())
xx()) w((t
& (a((), (A BK)x(() - Gw((), Kx(())=
Mi
(A.15)
where
Mi = (A+ P(A + BK) + aP + + + KT RK)
PG
(A.16)
GTP
-2201
The La norm of time-dependent signal w(t): 10, oc) + Rd is defined as / /500 (t)
14

---

Published as a conference paper at ICLR 2021
Therefore, seek find P E R* > 0 and K Rexa that satisfy M1 3 0, for some design
parameters a > and σ > Using Schur complements, the matrix inequality M₁ N 0 is equivalent
to
(A+ BK)*P + P(A + BK) +aP+ + PGG" P/( 3 o.
(A.17)
As in Appendix A.1 we left- and right- mult-mll both sides by p-1 and make the change of variables $ P- y KS. and 1/a to obtain
SA ASS + BY +as a 1 (QQ2)) + (Y"
( + + //² 30.
Using Schur Complements again, obtain the LMI
SA AS YT D" + BY + aS + pGG //² [SQU/2 RTR
Q 1
30, 8> 0, p 0,
pl
R! R!/3y
(A.18)
where then K= YS-1, P 5-1, and a 1/p.
B DERIVATION OF SETS OF STABILIZING POLICIES AND ASSOCIATED PROJECTIONS
We describe the construction the set actions C(x), defined Equation 7 for PLDI sys- tems (A.9) and Ho control settings (A.II). (The relevant formulations for the NLDI system (3 described in the main text.)
B. EXPONENTIAL STABILITY IN PLDIS For the general PLDI system by the following theorem.
A.9 relevant sets of exponentially stabilizing actions CPLDI are given
Theorem B.I. Consider PLDI system
A.9
some stability parameter a > 0, and a Lyapunov Assuming P exists, define
function (œ) = x²x Px with P satisfying A. 10
2x PBi
<a +PPA
2x PB2
x + +2PA2)
CrInI(1: =
E R
us
2x Pr.
xF Ar.)
for all states E R. For all x, CrLn(()) is a non-empty set of actions that satisfy the exponential stability condition 2 Further, COLDI( a convex set in u.
Proof. We seek to find a set of actions such that the condition 2 is satisfied along all possible trajectories of (A.9). i.e., for any allowable instantiation of (A(t), B())). A set of actions satisfying this condition at given 2 is given by
(ILDI(>)) (u RR ((x) < (v) V(A(), B()) Conv ((A1, Bi), ... (AL, Br.)}-
Expanding the left side of the inequality above, we see that for some coefficients 74 R 2 0, =
1,..., L satisfying DL, (i(t) 1,
((x = 4T Pr+ + Pi = 2x 20 P(A())+++++ B()))
= 2a P
Σ +
Σ x (2x P (A,r + Bru)
1=1
by definition of the PLDI dynamics and of the convex hull. Thus, if we can ensure
2x (A,r Bru) < -aV(x) -ox
Ter Px, =i=1...,,
then we can ensure that exponential stability holds. Rearranging this condition and writing it in matrix form yields an inequality of the desired form. We note that by definition of the specifica- P such that the policy
tions (.. there is some K corresponding the above incqualities; thus, Kr e Cp),i(Z),, and CPIDI( non-empty. Further, as the above inequality represents linear constraint in u, this set convex in u.
Kr satisfies all of
n
15

---

Published as a conference paper at ICLR 2021
We note that the relevant projection Crmí()) represents a projection onto intersection of halfs- paces, and can thus be implemented via differentiable quadratic programming (Amos and Kolter 2017).
B.2 Ho CONTROL
For the Ho control system (A.II relevant sets of actions satisfying the condition (A.12) are given by the following theorem.
Theorem B.2. Consider the system (A.11. some stability parameter a >> O, and Lyapunov func- tion ((x) = x² Pæ with P satisfying Equation (A.18). Assuming P exists, define
CH (I) = {u E R2 E Ru Ru (2B* Px) I (PA-A'P+aP+Q+E
P) 50}
for all states I E R.. For all x. Ch (x) a non-empty set of actions that guarantee condi- tion A.12). i.e., that the L2 gain of the disturbance tor-ncc-----ouuutt- map is bounded by and that the system is exponentially stable in the disturbance-free case. Further, Ct.,, (x) is convex in a.
Proof. We seek to find a set of actions such that the condition E(x, ct u) S O is satisfied along all possible trajectories of (A.IT). where & is defined in (A.12). A set of actions satisfying this condition at given is given by
CH (I) = {u E R"!
sup u) 50, i Ar + Bu + Gw). WEL2
To begin, we note that
Exx Ar Bu Gw, x- P(A Bu Bu+ Gw) (Ax + Bu Gw) Px + ax Pr
o (x Qx + Ruu - 72
We then maximize over wt
w- arg max E(x, Az Bu + Gw, v) G Px/(on²)
(B.1)
to
Therefore,
CH (x) = {u E(x, Ar + Bu + Gw" u, w²) s 0}.
(B.2)
Expanding and rearranging terms, this becomes
CH (x)= (u u (allj+ (2BT Tu u + TT (PA-AAPPaP--QQPPCCPP//T
0}. (B.3)
We note that by definition of the specifications A..18.. there is some K corresponding to P such that the policy u Kz satisifies the conditions above (see (A.17)): thus, Kr CH and Ch is non-empty, We note further that CH is an ellipsoid in the control action space, and is thus convex u.
We rewrite the set CH.,, (x) such that the projection Pen (w) can viewed as a second-order cone projection, in order to leverage our fast custtom solver (Appendix Co. In particular, defining P = a R. )) x, we can rewrite ellipsoid
q = Bd Px, and F = x ( (PA=A""*taPtoQ+PPCCCC above
CH (x) = {u Pu 27 0}.
(B.4)
We note that as 5 > and - q p-1ę < O, this ellipsoid is non-empty (see, e.g., section B.1 in Boyd and Vandenberghe (2004)) ). We then rewrite the ellipsoid
CHa (x) = fu
Au + b 11
(B.5)
where À V P-P-ー and P g 9.
The constraint ||Au + bll2 < is then
second-order cone constraint in u.
16

---

Published as a conference paper at ICLR 2021
C A FAST, DIFFERENTIABLE SOLVER FOR SECOND-ORDER CONE PROJECTION
In order to construct the robust policy class described Section 4 for the general NLDI system (3) and the Ho setting (A.II) we must project nominal (neural seworrk-bssed policy onto the second-order cone constraints described in Theorem [ and Appendix [3.2 respectively. As this
projection operation does not necessarily have a closed form, we implement it via custom differ-
entiable optimization solver.
More generally, consider a set of the form
C = (x E R"
||Az + 6112 c d}
(C.1)
for some A E Rm*x", e R", C R", and d E R. Given some input y E R",, we seek to compute the second-order cone projection Po(g) by solving the problem
minimize xEX"
|| -112
2
(C.2)
subject to ||Ax + bll2 S c² x + d.
Let F. denote the &g norm cone, i.e., =
= [ww,t t) w 22 < t}. Introducing the auxiliary variable
E [R++1 we can then rewrite the above optimization problem equivalently
1
minimize zER", #
1112 -112 + +x(22)
2
(C.3)
subject to == Ar +
=: Ga +
c² x x d
AA] c
b and h and where 1y denotes the indicator function
where for brevity define G = for membership in the set F.
We describe fast solution technique for computing this projection, well as method for obtaining gradients through the solution.
C.1 COMPUTING THE PROJECTION
We construct a fast solver for problem Specifically. define μ I m as the dual variable on the equality constraint in Equation (C.3. The
C.3 using an accelerated projected dual gradient method.
Lagrangian for this problem can then be written as
1 |10
L 2,
v12 15 1²
Gr h),
(C.4)
2
and the dual problem given by max, ming, (x, 2, p). To form the dual problem, we minimize the Lagrangian with respect to and 2 as
inf Lt 2, x,z
inf x
{||= v11 nd= Gz) + inf 1x(2)) - pTh h.
(C.5)
We note that the first term on the right side minimized at xw) = + GT H. Thus, we see that
inf 2 [11= v1l3 nr Gx]
2 = G n² Gy..
(C.6)
For the second term, denote μ (fi, s) and (). We then rewrite this term
inf
inf inl (L 8 n T 120
|| < 1).
(C.7)
For fixed t 2> the above objective is minimized for t < 0.) Substituting this minimizer into (C7 and minimizing the result over t 2 0 yields
1-ti //1l2 (The problem is infeasible
inf 2 1=(=)) inf t(s ( = 11
(C.8)
120
17

---

Published as a conference paper at ICLR 2021
where the last identity follows from definition of the second-orde cone F. Hence the negative dual problem becomes
minimize 2
GG (+GG++*+p (Gy + + 1z(m).
(C.9)
We now solve this problem via Nesterov's accelerated projected dual gradient method (Nesterov 2013 For notational brevity, define f(u) == Žn² "G" μ + n² (Gy + h). Then, starting from arbitrary ,(-1), E mm=+1 we perform the iterative updates
((k) = (1(()) + -.(0) (1111))
((+)) = Py (w()) 1 ((((l))) Lg
(C.10)
where L Danx((GT) is the Lipschitz constant of and Py is the projection operator onto F (which has closed form solution; see Bauschke (1996). Letting m, Amia Amin(GG denote the strong convexity constant of f, the momentum parameter is then scheduled as (Nesterov 2013
k
if if = 0
k + 2
(C.11)
VLs VLg + mmf
if > 0.
After computing the optimal dual variable ,r, i.c. the fixed point of C.10 the optimal primal variable can be recovered via the equation x= y + G n² (as be observed from the first-order conditions of the Lagrangian (C.4)
C.2 OBTAINING GRADIENTS
In order to incorporate the above projection into our neural network, we need to compute the gradi- ents of all problem variables (i.e., G, h, and y) through the solution x². In particular, note that x* has a direct dependence on both G and y, and an indirect dependence on all of G, h, and through '4*
To compute the relevant gradients through μ², we apply the implicit function theorem to the fixed point of the update equations (C.T0). Specifically, these updates imply that fz U² their fixed
point can be written as
μ² = P= (1t I, +.VS(>)
(C.12)
Define M (Pr(- 8() (-==a- 2, V)(r)) and note that Vf(x²) GGT +Gy The differential
of the above fixed-point equation is then given by
du M (dy' (dy"-I, Ls
1 +, (dGGT p² GdGT p² GGT du- + +GGy + Gdy + dhi) (C.13)
Rearranging terms separate the differentials of problem outputs from problem variables, that
MGGT dy² T (T-M+ Lj
dy-= =
M (dGGT)a 4z + GdG n² + dGy + Gdy + dh), (C.14)
Lj
where / the identity matrix of appropriate size.
As described in e.g. Amos and Kolter (2017), we then these equations to form the Jacobian of f* with respect to any the problem variables by setting the differential of the relevant problem variable to and of all other problem variables 0; solving the resulting equation for du* then yields the value of the desired Jacobian. However, as these Jacobians can be large depending on problem size, we rarely want to form them explicitly. Instead, given some backward pass vector at IR (m=+1) with respect to the optimal dual variable, want to directly compute the gradient By
18

---

Published as a conference paper at ICLR 2021
of the loss with respect to the problem variables: e.g., for y, want to directly form the result of the product ae 0u. Jμ by E RIx=. We do this via similar method presented in Amos and Kolter 2017). and refer the reader there for a more in-depth explanation of the method described below. C. 14
Define J =
M
MGGT to represent the coefficient of dpt on the left side of Equa- we then compute the intermediate term
L,
tion
Given
0t By
de
T
du ==
(C.15)
0um
We can then form the relevant gradient terms directly
de 0p-
T
1
M
+ (G" + +d,d²)
0pr 0G
L,
de 0p')
)
1
Md,
(C.16)
0pr 0h
L
ae 0p")
T
1
G Md
On" Oy
L
In these computations, we note that as our solver returns x², the backward pass vector we are given actually
dz E Rixn; thus, we compute Dr"
at 81 8n 0t
GT for use in Equation (C.15).
Dz* Oy
Accounting additionally for the direct dependence of some of the problem variables on x² (recalling that y + GT u²), the desired gradients are then given by
de r
de Ox*
de Ox Ou" T O +
1 - (") + + (GT d,)" +d,7²) Li
+
= Ox
0G
Oxt DG
dr du 0G
00)
0L 0r
de Ox Ou-
Mdd
0h
D5 0h
Ox Ou 0h Ls
de Ox
de Or Ou"
)
T
de
)
T
M
0y
Ox
Oy
Or Ou Öy
Or.
Ls
(C.17)
D WRITING THE CART-POLE PROBLEM AS AN NLDI
In the cart-pole task, our goal is to balance an inverted pendulum resting on top of a cart by exerting horizontal forces on the cart. Specifically, the state of this system is defined as [Pz, Pes : where Pr. is the cart position and & is the angular displacement of the pendulum from its vertical position; we seek to stabilize the system at x Ö by exerting horizontal forces For pendulum of length and mass Mp, and for cart of mass me, the dynamics of the system are (as described Tedrake (2009)):
on the cart.
Wh sin pitl? cos sp)
mh T. sin u
(D.I)
(me ma yo yp m cos
tm sin? up)
where g = 9.81 m/s² is the acceleration due to gravity. We rewrite this system as an NLDI by defining & = fæ, u) and then linearizing the system about its equilibrium point
t 3,,000
+ T,w,
||w||| S |C + Dull,
(D.2)
2
19

---

Published as a conference paper at ICLR 2021
where y is the Jacobian of the dynamics, f(x, u) J,, ir u/" is the linearization error, and 7 is the n. x n identity matrix. We bound this linearization error by numerically obtaining the matrices C. and D, assuming that and this process in more detail below. As a note, while we employ an NLDI here to characterize the linearization error,
within neighborhood of the origin. We describe
also possible to characterize this error via polytopic uncertainty (see Ap- pendix D we choose to use an NLDI here as it yields a much smaller problem description than PLDI this case.
D.I DERIVING J, 0) For à (2, u), that
T
0
0
8yj 1i//0e lae
0 /op
0pi
Jy u)
(D.3)
0
0
0
03/04
03/04
where
Op:
Ma c08 9 (69t goos p) + gmp sin" 4p
2mp sin yp cos yp (m, sin (49 g cos up) + tu)
de
m. + map sin" 2 yp
(me + my sin? u))
0p
2plmp sin y me my sin?
06
0p
Ou
me + ma sin"
g(me + mp) cos &p + x3 Im sin? Im, cos 4o + u sin
2m, sin yp cos yp @g(mm + mp) sin Im, sin cos a cos
0G
0ep
(m, mp sin? s))
(me my sin? v)
06
2:mmy sin yp cos vp
06
mp sin?
03
yp [(m. + mp sin ))
0u
We thus see that
0 0
1/m. 0
00
(D.4)
0
g(me tmp
1/m.
D.2 OBTAINING C AND
We then seek to construct matrices C and that bound the linearization error w between the true dynamics and our first-order linear approximation (, 0)
To do so, we bound the error of
this approximation entry-wise: that is, for each entry i = 1, l.... all x in some region I I x 5 I, and all u in some region u S u
we want to find F; such that for S U.
T
3 = v f
F
(D.5)
W
Then, given the matrix
T
M FI T/2 F/T/2
FT//2
T/2
F
F T/2
(D.6)
we can then obtain C' = -M and = Maxx+m (where the subscripts indicate column tiiun--wiie index- ing).
20

---

Published as a conference paper at ICLR 2021
We solve separately for each F; to minimize the difference between the right and left sides of Equa- tion (D.3) (while enforcing that the right side larger than the left side) over a discrete grid points within x I Σ and linear program the upper triangular entries F..
I i. By assuming that F; is symmetric,
able to cast this
To obtain the matrices C and D used for the cart-pole experiments in the main paper, we let == [1.5 2 0.2 1.5)", u 10, Σ -T, and μ uu. As each entry-wise difference in Equation (.5) contained exactly three variables (i.c., total of three entries from and u), solved each entry-wise linear program over a mesh grid of 50 points per variable.
E WRITING QUADROTOR AS AN NLDI
In the planar quadrotor setting, our goal to stabilize quadcopter the two-dimensional plane by controlling the amount of force provided by the quadeopter's 's right and left thrusters. Specifically, the state of this system is defined
[Px Ps t Ps pi: "]" where (Px. Ps) the position of the quadcopter in the vertical plane and yp is its roll (i.e., angle from the horizontal position); we seek stabilize the system
0 by controlling the amount of force u [ttr: u)" from right and left thrusters. We assume that our action is additional to a baseline force of [mg/2 mg/2/" provided by the thrusters by default to prevent the quadcopter from falling. For quadrotor with mass moment-arm for the thrusters, and moment of inertia J about the roll axis, the dynamics of this system are then given by (as modified from Singh et al. (2020)):
Pz cos p== sin P Pz sin y + p2a cos 4 4 P.p g ip PuG - g cos y +
0
0
0
0
0
0
+
u,
(E.1)
1/m
1/m
E/J
-E/J
where y 9.81 m/s². We linearize this system via a similar method for the cart-pole setting, ie., as in Equation (.2)). We describe this process more detail below. We note that since the dependence of the dynamics cart-pole, while we employ an NLDI here to characterize the linearization error, it is also possible to NLDI here
is linear, have that D 0 for our resultant NLDI. As for
characterize this error via polytopic uncertainty (see Appendix []; we choose as it yields much smaller problem description than a PLDI in this case.
E.1 DERIVING J (0,
For = (ax u), we see that
0
0
Pu sin 4 - pb: cos
cos yp sin yp
sin
0
Po cos yp - p. sin 4p
COS Pp
O
0
0
0
( u)
(E.2)
g cos
P=
0
0
g sin 4
-
P
0
0
0
and thus
0
0
0
0
0
07
0
0
0
1
0
0
0
0
0
0
1
0
0)
(E.3)
0
0
0
0
0
0
0
0
0
0
0
0
E.2 OBTAINING C AND D
We obtain the matrices and D via similar method as described in Appendix though in practice only consider the linearization error with respect (i.e., since the dynamics are linear with respect to u, we have cart-pole, each entry wise difference in the equivalent of Equation (D.5) contained exactly three variables (i.c., a total three entries from æ and u), and each entry-wise linear program was solved mesh grid of 50 points per variable.
0). We let 5 = [1
0.15 0.6 0.6 1.3] and z = - As for
21

---

Published as a conference paper at ICLR 2021
F DETAILS ON THE MICROGRID SETTING
For our experiments, we build upon the microgrid setting given in Lam et al. (2016). In this system, the state x E & captures voltage deviations, frequency deviations, and the amount of power gener- ated by diesel generator connected to the grid; the action u E I² describes the current associated with a storage device and solar PV inverter; and the disturbance w E IR describes the difference between the amount of power demanded and the amount of power produced by solar panels on the grid. The authors also define a performance index E IR² which captures voltage and frequency deviations (i.e., two of the entries of the state z))
To construct an NLDI of the form 33 for this system, we directly use the A, B, and G matrices given in Lam al. (016)) We generate C i.i.d. from a normal distribution and let D 0, to represent the fact that the disturbance w and the entries of the state are correlated, but that w is likely not correlated with the actions u. Finally, we let Q and R be diagonal matrices with in the entries corresponding to quantities represented in the performance index y, and with 0.1 in the rest of the diagonal entries, to emphasize that the variables y are the most important in describing the performance of the system.
G GENERATING AN ADVERSARIAL DISTURBANCE
In the NIDDI settings explored in our experiments, we seek to construct an "adversarial" disturbance wt) that obeys the relevant norm bounds [|w(()]l2 S ||Cx(()) + Du(t)lla while maximizing the loss. To do this, we use a model predictive control method where the actions taken are w()). Specifically, for each policy roll-out, we optimize w(t) through gradient descent to maximize the loss over a horizon of 40 steps, subject to the constraint [|wu(t]lla S [|Cx((t) 122
we model wt) as neural network specific to that policy. Every 10 steps of
H ADDITIONAL EXPERIMENTAL DETAILS
Initial states. To pick initial states in our experiments, for the synthetic settings, we sample each attribute of the state i.i.d from standard Gaussian distribution. For cart-pole and planar quadrotor, we sample uniformly from bounds chosen such that the non-robust LQR algorithm (under the orig- inal dynamies) did not go unstable. For cart-pole, these bounds were chosen to be Pa.
E [-
E 0-0.1, 0.1]. p= o = 0. For planar quadrotor, these bounds were Pa: Ps E - 1-1,1] p (-005 0.05], Ps p. = O.
Constructing NLDI bounds. Given these initial states, for the cart-pole and quadrotor settings, we needed to construct our NLDI disturbance bounds such that they would hold over the entire trajectory of the robust policy; if not, the robustness specification (A.8) would not hold, and our agent might in fact increase the Lyapunov function. To ensure this upproximately, we used a simple heuristic: we ran the (non-robust) LQR agent for full episode with 50 different starting conditions, and constructed an L ball around all states reached in any of these trajectories. We then used these Loo balls the states construct the matrices C' and D for our disturbance bounds, using the procedure described in Appendices D and B
Computing infrastructure and runtime. All experiments were run on an XPS 13 laptop with an Intel i7 processor. The planar quadrotor and synthetic NLDI experiment with D = 0 took about day to run (since the projections were simple half-space projections), while all the other synthetic domains and cart-pole took about 3 days to run. The majority of the run-time was in computing the adversarial disturbances for test-time evaluations.
Hyperparameter selection. For our experiments, we did not perform large parameter searches. The learning rate we chose for our model-based planner, (both robust and non-robust) remained
constant for the different domains; we tried learning rates of 1 10 1, 10 1 x 10-5 and found 1 x 10 worked best for the non-robust version and 1 x 10 worked best for the robust version. For our PPO hyperparameters, we simply used those used in the original PPO paper.
One parameter we had to tune for each environment was the time step. In particular, we had to pick time step high enough that we could run episodes for reasonable total length of time (within which the non-robust agents would go unstable), but low enough to reasonably approximate continuous- time setting (since, for our robustness guarantees, we assume the agent's actions evolve in continu- ous time). Our search space was small, however, consisting of 0.05, 0.02, 0.01, and 0.005 seconds.
22

---

Published as a conference paper at ICLR 2021
2.0
1.5
18
1.0
0.0
-05
-15-
-1.5-
-20
10
(a) LQQ
(b) Robust I.QR
2.0
2.0
1.5-
05-
0.5-
-05
10
-1.0
-20
-2.0
(c) MBP
(d) Robust
15
1.5
1.0
1.0-
0.
0.0-
-0.
-0.5
-1.0
-15
-1.5
10
10
(e) PPO
(I) Robust PPO
Figure H.1: Trajectories different methods on the cart-pole domain under adversarial dynamics.
Trajectory plots. Figure H.I shows sample trajectories of different methods in the cart-pole do- main under adversarial dynamics. The non-robust LQR and model-based planning approaches both diverge and the non-robust PPO doesn't 't diverge, but doesn't 't clearly converge after 10 seconds. The robust methods, on the other hand, all clearly converge after 10 seconds.
Runtime comparison. Tables H. and H.2 show the evaluation and training time of our methods and the baselines over 50 episodes run in parallel. In the NLDI cases where D = O, i.e., Generic NLDI (D 0) and Quadrotor, our projection adds only very small computational cost. In the other cases, the additional computational cost is more significant, but our method is still far less expensive than the Robust MPC method.
23

---

Published as a conference paper at ICLR 2021
Robust
Robust
Robust
Robust
Environment
LQQR
MBP
PPO
RARL
LQR
MPC
MBP
PPO
Generie NL.II (7 = 0)
0.63
0.61
0.84
0.57
718.06
0.71
0.73
0.94
Generic NLDI (D + 0)
0.64
0.62
0.83
0.58
824.86
0.81
15.13
25.38
Cart-pole
0.55
0.67
0.84
0.53
646.90
0.84
10.12
13.37
Quadrotor
0.95
1.19
0.88
3348.68
1.14
1.15
T.30
Microgrid
0.58
0.61
0.79
0.57
601.90
0.74
8.14
10.25
Generic PLDI
0.57
0.54
0.76
0.51
819.24
0.73
69.35
64.03
Generic Ho.
0.84
0.80
1.03
0.76
N/A
1.00
47.81
63.67
Table H.1:
Time (in seconds) taken run each method on the test of every environment for 50
episodes run in parallel.
Robust MBP
Robust PPO
Environment
MBP
PPO
RARL
Generic NLDI (D
0)
26.36
101.77
102.37
30.78
114.60
Generic NLDI (D + 0)
26.46
100.79
82.53
221.35
1158.28
Cart-pole
25.49
87.04
98.90
146.34
689.93
Quadrotor
41.24
131.48
112.95
46.13
159.06
Microgrid
23.03
112.52
87.71
113.61
436.64
Table H.2: Time (in minutes) taken train each method in every environment.
I EXPERIMENTS FOR PLDIS AND H& CONTROL SETTINGS
In addition to the NLDI settings explored in the main text, we test the performance of our method on PLDI and Ho control settings. As for the experiments in the main text, we choose time discretization based on the speed at which the system evolves, and run each episode for 200 steps over this discretization. In both cases, we use randomly generated LQR objective where the matrices 1/2 and R1/2 are drawn i.i.d. from standard normal distribution.
Synthetic PLDI setting. We generate PLDI instances (A.9) with 5, 3. and L = 3. Specifi- cally, we generate convex hull matrices (A1, Bi)...,, (Ag, Ba) i..dd from normal distributions, and generate (A(t), B(t)) by using randomly inntiaiie neural network with softmax output weight the convex hull matrices. Episodes were rum for seconds at a discretization of 0.01 seconds.
Synthetic Ho setting, We generate Ho control instances (A.II) with 5. = 3, and d = by generating matrices A, and G.i.d. from normal distributions. The disturbance w()) was produced using randomly-initialzzn
neural network, with its output scaled to satisfy the Le bound on the disturbance. Specifically, we scaled the output of the neural network to satisfy an attenuating norm- bound on the disturbance; at time t, the norm-boume was given by 20 x f(2 x t/T)) where T is the time horizon and f is the standard normal PDF function. Episodes were run for T 2 seconds at discretization of 0.01 seconds.
Results are given in Figure 1.1 and Table L.]
Robust LQR
Robust MPC
Robust MBP
Robust
Environment
LQR
MBP
PPO
RARL
PPO
0
96.3
3,3
8.0
19.2
19.2
15.8
18.6
10.2
Generic PLDI
A
unstable
43.3
44.1
unstable
21.9
16.1
181
114
165
N/A
115
125
Generic Hoc
A
219
112
143
206
N/A
145
147
158
Table I.1: Performance various approaches, both robust (right) and non-robust (left), on domains
of interest. We report average quadratic loss over 50 episodes under the original dynamics (O) and under an adversarial disturbance (A). For the original dynamics (O), the best performance for both non-robust methods and robust methods is in bold (lower loss is better). We use "unstable" to indicate cases where the relevant method became unstable. Our robust methods (denoted by ") improve performance over Robust LQR in the average case, while remaining stable under adversarial dynamics, whereas the non-robust methods either went unstable or received much larger losses.
24

---

Published as a conference paper at ICLR 2021
Non-robust Methods
Robust Methods
PLDI
Loss
100
Loss
H
10
0
250
500
750
1000
0
250
500
750
1000
Training epochs
Training epochs
MBP
PPO
RARL
Robust MBP
Robust PPO'
Setting:
Original
Adversarial
Figure 1.1: Representative results for our experimental settings, For each training epoch (10 updates for the MBP model and 18 for PPO), we report average quadratic loss over 50 episodes, and use "X" to indicate cases where the relevant method became unstable. (Lower loss is better.) Our robust methods (denoted by ") improve performance over Robust LQR in the average case, while (un- like the non-robust methods) remaining stable under adversarial dynamics throughout the training process.
J NOTES ON LINEARIZATION VIA PLDIS AND NLDIS
While linearize the cart-pole and quadrotor dynamics via NLDIs in our experiments, note that these dynamics can also be characterized via PLDIs. More generally, in this section, we show how can use the framework of PLDIs model lincarization errors arising in the analysis of nonlinear systems.
Consider the nonlinear dynamical system
i= fx, u) with f(0,0) 0.
(J.1)
for x E R and u E Ra. Define & = (z, u). We would like to represent the above system as PLDI in the region { E { < { including the origin. The mean value theorem states that for each component of f, we can write
(((5) f((0) + V)((>*
(I.2)
for some = tG, where t E [0, Now, let p = + a. Defining the Jacobian f as
(=)
Jy(2) =
(J.3)
and recalling that f(0) = 0, we can rewrite (.2 as
(()) J)()).
(I.4)
Now, suppose we can find component-wise bounds on the matrix 32(2) over R, i.e,
M s Jy(z) < M for all 2 E R.
(I.5)
We can then write
1x(2)=
Σ E. with myy(t [may. m.yl
(J.6)
15tJjP
where Paj ezez and is the i-th unit vector in R
25

---

Published as a conference paper at ICLR 2021
We now seek bound the Jacobian using polytopic bounds. To do this, note that write
2d
Jy(2)
MxA
The
0,
Σ Nh
(J.7)
where A₂'s are the vertices of the polytope in U.6 i.e.,
A, e ==
Σ
my BiyBig miy {mey must
(J.8)
SSL.SP
Together, Equations PLDI.
J.2
and
J.8 characterize the original nonlinear dynamics as
We note that this PLDI description is potentially very large; in particular, the size of V is exponential in the square the number of non-constant entries in the Jacobian [z):), which could be as large as 20 = 2(*+a)² This problem size may therefore become intractable for larger control settings.
We note, however, that we can in fact express this PLDI more concisely an NLDI. More precisely, we would like to find matrices A, B, C' parameterizing the form of NLDI below, which is equivalent that presented in Equation 3
(see Chapter of Boyd al. 1994)
Df(z) (A+ BAC |||||2 < 1} for all 2 R.
(J.9)
It can shown that the solution to the SDP
minimize tr(V + W)
subject to >
(J.10)
V
(Ax A)" W
= VA. EV
-
yields the matrices A, B, and C with V = CTC and W = BBT, which can be used to construct NLDI (.9). While the NLDI here is more concise than the PLDI, the trade-off that the NLDI norm bounds obtained via this method may be rather loose. As such, for our settings, NLDI bounds numerically (see Appendices [ and E). as these are tighter than NLDI specifications obtained via the above method (though they potentially slightly inexact). An alternative approach would be to examine how to tighten the conversion from PLDIs to NLDIs, which has been explored in other work (e.g. Kuiava al. (2013)) ).
obtain
26


Reviews:
[{"review_id": "cMB6Uqkffx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a new robust controller design approach, in which the controller is parameterised by DNN. They show that by integrating custom convex-optimization-based projection layers into a nonlinear policy, they can construct a provably robust neural network policy class.\n\n\n1.This is paper is heavily in control theory. The proof of Theorem 1 and corollary 1 is standard in the sense of robust control and LMI. The key contribution I can see is Section 4.3 on deriving differential projections. \n\n2.In control community, adaptive dynamic programming is exactly to deal with the problem proposed in the paper. Frank Lewis and many others had done a lot of work in this field. I saw the authors cited one of his papers in 2006 in the section of RL. Unfortunately, in comparison with the classic and recent progress in this field is missing.  \n\n3.The definition of stability and exponential stability is not given, which learning community may not be aware of.\n\n4.A key reference is missing “H∞ Model-free Reinforcement Learning with Robust Stability Guarantee, arXiv:1911.02875”. In this paper, the authors propose a model-free approach to learning the Lyapunov function and policy simultaneously. I think this paper is more general than this ICLR submission. A remark and comparison are needed.\n\n5.Regarding the experiment, I didn’t see how the nonlinear systems are converted to (1). If I assume this is possible, how such approximation gap can be quantified and how will this gap affect the theoretical results.\n\n6.One important detail in the experiment is missing: how is the initial condition selected. Let’s take the cart pole as an example: if the initial condition is (very) close to the equilibrium point, the nonlinearities will be minimal. I don’t think robust LQR will be worse. While RL is outstanding in the nonlinear region, the authors should make a fair comparison/add more scenarios.   \n\n7.Comparison with PPO and is MBP is not proper and unfair. PPO and MBP is a data-based method, i.e., a concrete system model is NOT needed. While the proposed method must need a model and the theoretical result depends on the model parameters\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YPA5SBw9_dL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n  [1] The problem of RL policies with both robustness guarantee and good average performance is interesting and useful for many practical applications.\n  [2] Paper is well-written and clear.\n  [3] The proposed method/approach is inspiring and novel, an the results look promising for this proposed new method.\n\n\nMain concern:\n  The results presented in the experiments section is not very comprehensive and not convincing enough to justify the claimed benefits of the proposed approach v.s. traditional robust LQR approach. In particular, as we could see from Table 1, although in the original dynamics scenario, the proposed robust MBP and robust PPO approach has better performance than the robust LQR, in the adversarial disturbance scenarios,  the proposed robust MBP and robust PPO might perform significantly worse (e.g., about 8 times larger loss for Microgrid case) than the traditional robust LQR, which implies that, for many scenarios that are not worst case, the proposed method could perform worse than the robust LQR (i.e., from 8 times larger loss (e.g., 7.12 v.s. 0.86) in the adversarial case, to  slightly smaller loss (0.61 v.s. 0.73) in the original case, there is a large gap there). And there is not a clear and well justified criteria in the paper to clarify, for most real world applications, what disturbance is defined as normal(/average/original) case, and what scenario is for adversarial case, and also how about the disturbance between these two scenarios? Due to above concerns, and also note that the examples provided in this paper is very limited rather than comprehensive, it is not convincing enough to claim the overall performance benefits of the proposed approach over the traditional robust control techniques. More comprehensive experimental studies and evidence are needed to well justify the claimed performance benefits.\n\nEdit: upgrade the rating to 6 with the clarifications from the authors, with which the submission is clearer and more convincing now.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6R_Ve77TnNs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper combines robust control theory with NNs to obtain robustness guarantees. The paper shows stable behavior on adversarial dynamic models on many different simulated tasks. The paper's main idea revolves around projecting the output of a NN policy to be within the set of a Lyapunov function (stability condition) (e.g., exponential stability). I have some concerns which are listed below,\n\n1. What are the limitations of the approach for larger dimensional control problems, such as a 7-dof arm? It's unclear if the method can scale to large dimensional control problems. \n\n2. Fig.1 methods are hard to separate, plotting with qualitatively different colors would be helpful. Additionally, explaining the horizontal lines that start before 0 would be useful.\n\n3. While the results show stable behavior, the loss in the adversarial setting  doesn't improve during training. Is there some reasoning for this? I am suspecting the adversarial dynamics is randomizing after every epoch? Showing the method improving with a fixed adversarial dynamics over epochs would be good to show improvement in addition to stability.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4jbsKGPD0CP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a neural control method is proposed with stability guarantees. The control is assumed to be from a neural network that takes in the state. Stability is guaranteed by projecting the control to the set that satisfies the Lyapunov stability condition for the LQR problem. In particular, minimizing the cost of LQR cost subject to stability constraints can be cast as an SDP for norm-bouned linear differential inclusions. Through making use of the convex optimization layers proposed in Agrawal et al. (2019), the SDP can be added as a layer after the neural policy and efficient projections can be derived such that implicit function theorem can be utilized to differentiate through the fixed point (the optimal conditions of the SDP), such that end to end learning is possible. The proposed approach is compared with the unconstrained method on various tasks.  Both model-based and model-free RL algorithms are used as the neural policy for comparison. The stability-guaranteed approach is able to remain stable even under bounded adversarial dynamics. In comparison, the non-robust methods fail to maintain stability.\n \nIn general, I like the idea of enforcing stability by introducing the convex optimization layer. Although the dynamics model used is still relatively basic, but the nice convex formulation provides an opportunity to incorporate stability certificates to neural policy that enable end-to-end training. The stability is roughly maintained as illustrated in the experiments. However, it would be better if trajectories can be visualized in some way to show that the proposed method can stabilize the system. One potential concern of the method is that it would be more computationally expensive than the unconstrained method, therefore it is of interest to compare the running time. \n\nSome minor points regarding the experiments: the description of the methods used are not properly defined or referenced (such as PPO, RARL, MBP). Also, the experiments for PLDIs and H-infinity control settings seem to be missing (only description of the setup is found). \n\n----------------------After author's response--------------------\n\nThe response addressed most of my concerns and included experiments results of trajectory visualization and run-time comparison. I think the paper would be an interesting contribution to the conference.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers seem to like the main idea of the work, they had several concerns, particularly regarding the experiments (both their setup and description) and the overall language of the paper that they found it more suitable for the control community than the ML and representation learning community. The authors provided very long response and tried to address the issues raised by the reviewers during the rebuttals. Fortunately, the response addressed some of the issues they raised and now they all see the paper marginally above the line. However, reading the reviews and response shows that the paper can highly benefit from better writing and describing the experiments. So, I would strongly recommend that the authors include all the information they provided for the reviewers during the rebuttal phase in the paper and improve its quality. 
Decision: Accept (Poster)