Adversarially Robust Neural Lyapunov Control
Authors: Li Wei
Yuankun Jiang
Chenglin Li
Wenrui Dai
Junni Zou
Hongkai Xiong
Venue: Submitted to ICLR 2023 (2023)

Abstract: State-of-the-art learning-based stability control methods for nonlinear robotic systems suffer from the issue of reality gap, which stems from discrepancy of the system dynamics between training and target (test) environments. To mitigate this gap, we propose an adversarially robust neural Lyapunov control (ARNLC) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. Specifically, inspired by adversarial learning, we introduce an adversary to simulate the dynamics discrepancy, which is learned through deep reinforcement learning to generate the worst-case perturbations during the controller's training. By alternatively updating the controller to minimize the perturbed Lyapunov risk and the adversary to deviate the controller from its objective, the learned control policy enjoys a theoretical guarantee of stability. Empirical evaluations on five stability control tasks with the uniform and worst-case perturbations demonstrate that ARNLC not only accelerates the convergence to asymptotic stability, but can generalize better in the entire perturbation space.

Under review as a conference paper at ICLR 2023
ADVERSARIALLY ROBUST NEURAL LYAPUNOV CONTROL
Anonymous authors Paper under double-bling review
ABSTRACT
tfate-of-the-ar learning-baas based stability control methods for nonlinear robotic sys- tems suffer from the issue of reality gap. which stems from discrepancy the system dynamics between training and target (test) environments. To mitigate this gap, we propose an adversarially robust neural Lyapunov control (ARNI.C) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. Specifically, inspired by adversarial learning, we introduce an adversary simulate the dynamics discrepancy, which is learned through deep reinforcement learning to generate the worst-case perturbations dur- ing the controller's training. By alternatively updating the controller minimize the perturbed Lyapunov risk and the adversary to deviate the controller from its objective, the learned control policy enjoys theoretical guarantee of stability, Empirical evaluations on five stability control tasks with the uniform and worst- case perturbations demonstrate that ARNLC not only accelerates the convergence to asymptotic stability, but can generalize better in the entire perturbation space.
INTRODUCTION
Designing stable and robust controller to stabilize nonlinear dynamical systems has long been challenge. Lyapunov stability theory performs a fairly significant role in the controller design for stability control of robotic systems (Lddin et all 2021; SSharma & Kumar, 2020; Liu all 20206: Norouzi et 2020; Pal all (02(). However, many previous approaches are restricted to the poly- nomial approximation of system dynamics (Kwakeeaaak Sivan, [969 Pamlo 2001), and suffer from sensitivity issues when searching for the Lyapunov functions (Lötberg 2009). Recently, by leveraging deep learning-based methods, some works have successfully incorporated the Lyapunov stability theory with the powerful expressiveness of neural networks and convenience of gradient descent for network learning (Chang et ar, 2ITE Abate et all 2020, Mehrjou et anz 2021: Dawson
2022). One outstanding method among them is the neural Lyapunov control (NLC) (Chang all 2019), where both the Lyapunov function and controller policy are approximated by neural networks. In NLC. the networks trained by minimizing Lyapunov risk stemmed from the Lya- punov stability theorem. Nevertheless, most existing learning- based controller are trained without any distinction between the training and test environments (Cobhe all X0T+% Witty et all 202T). Since the training simulator cannot perfectly model the target environment for testing, a reality gap will incur inevitably by such a modelling error (i.e., discrepancy of system dynamics), which de- grades the performance of controller at the actual deployment. Hence, learning- -based controller needs to consider the uncertainty of physical parameters (or external forces) that may cause the modelling error (Liu et all 20203; Garg & Panagou. 2021; Islam et all 2015;; Zhao et all (020). Motivated by this, we focus this paper on addressing the challenging problem of learning a con- troller to stabilize the nonlinear dynamical system in face of such a modelling error.
Over the years, several approaches have already been proposed to alleviate the controller's per- formance degradation incurred by modelling errors. The majority of them is built upon another splendid learning-based control method: deep reinforcement learning (RL) (Sutton & Barto, 2018: Schulman et all, 2017b). These deep RL-based control methods treat the modeling error as extra disturbance to the system (Başar & Bernhard 2008). and have achieved a great success in control- ling (Pinto et al, 2017: Wessler all, 2019; Zhang et all 2020, 2021: Mankowitz et all 2020). For example, in robust adversarial reinforcement learning (RARL), the policy learning is formulated as zero-sum game between the controller and adversary that generates disturbances to the system,

---

Under review as a conference paper at ICLR 2023
where the learned controller is proved to have improved capability of robustness and generalization. Since RL methods train policies by maximizing the sum expected rewards that the agent obtains during the interaction with environment, its performance depends greatly on the manually designed reward function while the learned policy is sensitive to the preset control interval (Tallec et all 2012 Park et all (021)). Hence, RL is prone to fail in the control tasks with relatively small control in- terval, as will later be verified in our experiments. While our aim is to find the control policy that can enable a stable control, which is also robust to the choice of control intervals.
In this paper, we present a novel method that can automatically learn robust control policies with a provable guarantee of stability. Specifically, we formulate a perturbed Lyapunov risk for learning a controller in the dynamical system, which is imposed with the adversary's 's perturbations in a certain range. To train the controller policy to resist to the worst-case perturbations within that range, we formulate the learning of adversary as a Markov decision process (MDP), and train the adversary policy by proximal policy optimization (PPO). In the case of known system dynamics, the action space in the MDP can be the range of external forces space of physical parameters, which causes the modelling error. More practically for the unknown dynamics, the original NLC no longer works since update of the networks is infeasible without prior knowledge of the system dynamics. We therefore train an environment model by sampling from the system, while the adversary's ' action is set the offset to the output of this environment model. We further formulate adversarially robust controller learning problem, which is approximately solved by alternatively updating the controller policy with Lyapunov methods and the adversary policy by PPO. Our contributions can be summarized as follows.
We propose a perturbed Lyapunov risk for learning the control policy under perturbations.
We formulate an optimization problem for adversarially robust controller learning, to learn a policy in face of the worst-case perturbations that are imposed by the RL-trained adversary.
We propose an adversarially robust neural Lyapunov control (ARNLC) approach to approxi- mately solve this problem, and demonstrate its performance on several stability control tasks.
2 RELATED WORK
Adversarial training. The idea of viewing the gap between training and test scenarios as extra disturbance of the system was first proposed in Mormooto & Doya (2(05)), with the problem formu- lated as finding a min-ma max solution of the value function that takes the perturbations into account. Inspired by Morimoto & Doya (2(N))), Pinto et all (2017) propose the robust adversarial reinforce- ment learning (RARL), where an adversary is learned simultaneously to prevent the agent from accomplishing its goal, while the agent's policy and the adversary policy are trained alternately. Zhang et all (2021) propose robust reinforcement learning based on perturbations on state observa- adversary to apply disturbances on the state observations of the agent. Tessled al (2019) focus on a scenario where the agent attempts to perform an action, which behaves dif- ferently from expected due to disturbances. All of the above literature mainly studies training the adversary for RL settings, while our focus in this work is on introducing adversarial training to the Lyapunov stability control.
tions, introducing
Neural Lyapunov stability control. Chang et al. (201 propose the neural Lyapunov control, which uses neural networks to learn both the control and Lyapunov functions for nonlinear dynam- ical systems based on the Lyapunov stability theory. Saba (21121) learn a control law that stabilizes an unknown nonlinear dynamic system. However, it needs design a Lyapunov func- tion manually. Dawson et all (2(22) also propose an approach for learning the robust nonlinear controller based on the robust convex optimization and Lyapunov theory, achieving generalization beyond system parameters seen during the training process. However, this approach only considers the control-affine dynamical systems, not the more general nonlinear ones. this work, we focus on improving the robustness and generalization for control policies of nonlinear dynamical systems.
Robust model predictive control (Robust MPC). Robust MPC is another research branch to deal with the uncertainty in physical parameters (Sun etal 20T8; Hu & Ding, 2019; Köhler et all 2021)). looks for the optimal feedback law among all the feasible feedback laws within a given finite horizon, in terms of given control performance criterion every sampling instant (Houska & Villnureaa 2019). However, it is usually restricted to the additive disturbances (Löfberg, 2003) and is computationally expensive (Bemporad & Morari, 1998).
2

---

Under review as a conference paper at ICLR 2023
3 PRELIMINARIES AND BACKGROUND
We consider a continuous-tmm,, time-invariant nonlinear dynamical system of the form:
it [ at):
(1)
where 2₁ E X C R" is the state and a E A S I= is the control input at time t, respectively, and in denotes the first-order time derivative of Ft. The system feedback controlled by policy function: a, = (2,). We aim to stabilize this system at an equilibrium point z = 0 E X. by finding a policy to build a closed-loop controlled dynamical system á f (Ire) with f= O, such that the equilibrium point 0 achieves asymptotic stability as defined below.
Definition 1 (Asymptotic stability in the sense of Lyapunov (Lavretsky & Wise, 2013).. The equilibrium point x of fx is stable in the sense of Lyapunov if Ve > 0, Vlo O, there exists ō(E) > 0 such that if [[z((t)] || S. o(s) then ||-(t)]| S ε for all t 2 to. The equilibrium point æ = 0 of fa asymptotically stable if stable and there exists a positive constant c c(ta) such that axt) + O as + oc, for all [|x((t)]] S C.
3.1 STABILITY GUARANTEE WITH LYAPUNOV FUNCTIONS
Lyapunov stability theory provides an elegant way to guarantee the stability, as follows.
Theorem (Lyapunov stability theorem (Lavretsky & Wise, 2013). Suppose f X R is locally Lipschitz in X C R For a continuous-time controlled dynamical system S if there exists a continuous function V: IR such that
V 0; and V(x) > 0, Vo E X - (0); and (xx <0;
then the system is asymptotically stable at x = O, where V is called a Lyapunov functiinn
The time derivative of (xx) can be derived as 1 depends on both (zx) and the controlled dynamics fn. Theorem W then states that the trajectories of the system's 's state will eventually reach the equilibrium x O, if we can design a control policy such that the Lyapunov function (xx) exists and satisfies the conditions in Eq. (2).
8V = D2
@V - de.
(x), which
3.2 NEURAL LYAPUNOV CONTROL
Neural Lyapumov control (NLC) (Chang et al. 2019) leverages neural networks to approximate both the control policy and Lyapunov function Vo(x), which are parameterized by 0 and 0, respec- tively. The network parameters x and (z) are learned by minimizing the following Lyapunov risk:
Lp = = max(0, -Vo max(0, max(0, (())))+1330))
where is a random variable following a uniformly random distribution p over the state space X. In its physical meaning, this Lyapunov risk quantifies the degree of violation of the Lyapunov conditions in Eq. (2) over the state space X, given certain policy and Lyapunov function.
4 ADVERSARIALLY ROBUST NEURAL LYAPUNOV CONTROL
In this paper, we aim to narrow down the reality gap incurred by the modelling discrepancy in the system dynamics f between training and test environments, by learning a policy of controller μ that is better stabilizing the system (i.e., achieving the asymptotic stability faster) and more robust (i.e., resisting to variations of the system dynamics). Specifically, we consider modeling such a discrepancy by an adversary, with the system function given by:
d, = fIz, a²),
(4)
where at" E A and of E Ando are the controller's action and adversary's action at time following their policies Th and ny. respectively, while the rest notations follow the definition in Eq. (M). In view of the controller, aj imposes variation to and makes the dynamics f time-varying, which can be rewritten as ă; = fn (2,, af'). Hence, introducing the adversary v imposes a time- varying modelling error to Eq. (ll during training of the controller. A practical example is where
3

---

Under review as a conference paper at ICLR 2023
the controller is applied to manipulate locomotion of robot outdoor, while the adversary may be the weather that produces unpredictable wind or rain to disturb this controller. Note that the system dynamics in view of the controller reduces to Eq. (I) when oj (2+)) = 0 for any time t. In this section, we propose the adversarially robust neural Lyapunov control (ARNLC) method to train a policy Ty for the controller J4, such that the system governed by Eq. (9) is stabilized in face of the adversary v.
4.1 PERTURBED CONTROLLER LEARNING
In our adversarial control setting, both the controller yr and adversary observe the system state 2, at time t, and then take actions of ~ Ty (Zt) and aj (2z). After that, the system evolves according to Eq. (H)) Here, we utilize neural networks to learn the controller policy Fy (zt),, adversary policy Th and candidate Lyapunov function (z)z,, as parameterized by 0H, 0" and 0, respectively. Our objective is to leverage the Lyapunov stability theory to find controller policy xμ that can achieve the stability of system in the presence of certain adversary. Namely, the resulting closed-loop controlled dynamical system & = I= (Ix) is asymptotically stable at the equilibrium point x 0. Motivated by this, our proposed ARNLC seeks minimize the following perturbed Lyapunov risk w.r.t. 0 and 0r.. to update the controller policy together with the candidate Lyapunov function.
Definition (Perturbed Lyapunov risk for controller). We consider a candidate Lyapunov func- tion Vo for a continuous- dynamical system in Eq. (e)) In the presence of adversary policy To parameterized by 0.. the perturbed I. Lyapunov risk for the controller ju is defined by:
Lp (0, 8", 8¹) = Ex-p(X)) Texx(-Voloe (max)_VV(xx),0) + max(0, vo(z)) + V3 v0)
(5)
where p(X) is the state distribution, and Vo(x) = ='= du; [f
The learning of n. and Vo then be formulated as the following optimization problem:
min (0, 0", 0²), s.t. 2 f(x, at "a") 0,6
at T a"
(6)
This perturbed Lyapunov risk Lp differs from the conventional Lyapunov risk in that the time deriva- tive V=(x) now depends on f=,.=, instead of fr. which makes the closed-loop dynamical system
time-varying for the controller. In practice, we use the following empirically perturbed Lyapunov risk, which is an unbiased estimator of Eq. (B):
1 N
mmxx)-V((z;).0) + max(0, vo(a;)) + v.(
Lnap (0, 10,00000 00)
=
However, this practical estimator cannot guarantee satisfaction of the conditions in Theorem M on the entire state space X. We thus apply additionally a falsifier constantly find the counter-examples that violate these conditions during the training process, which is common strategy also used in NLC. Specifically, the falsifier finds counter-cxample states according the following criterion:
Vo(x) S 0 (x( 2 0, Vo E X (0):
which specifies the negation of conditions in Eq. (L). During the training of मम and Vo, the falsifier constantly finds counter-examples and adds them into the training dataset.
4.2 ADVERSARY LEARNING
Compared with the conventional Lyapunov risk in Eq. (L), the perturbed counterpart I, presents some new challenges to the learning of controller policy. Due to the perturbation from adversary, the dynamical system in view the controller becomes time-varying, which prevents the learning of its policy मr from reaching the stability will be shortly shown in experiments. Inspired by the idea of adversarial training, the proposed ARNLC leverages reinforcement learning method to train the adversary policy. The intuition behind is that if case perturbation (which degrades the performance of its policy to the most) a certain range, the controller then obtains conservative policy that is robust any perturbation within that same range.
train a controller under the worst-
4

---

Under review as a conference paper at ICLR 2023
We formulate training of the adversary as discrete-time Markov decision process (MDP) with a fixed control interval, defined as the tuple (X, Aade; A, P.T. ?), where Y is the discount factor. The adversary agent observes the system state E X' at each time step and takes action a" E Aades while the controller acts at E A. The system then evolves to the next state x' according to transition probability (()aæ, at", a²) between time steps, and the adversary agent receives reward r(ær, at, a,").
Adversary's action design. Depending on whether or not the system dynamics f can be accessed, ARNLC applies different action design for the adversary agent. i) For known dynamics f. the action space Ande of adversary can be range of the external disturbing force or space of the envi- romment parameters, which changes the system dynamics view of the controller. The adversary action can then be directly imposed on the system, which simulates the external force (e.g., strong wind) and change of environment parameters (e.g., friction coefficient) that the controller may en- counter in the test environment. ii) For unknown dynamics f. NLC is unable to back propagate the gradients to update n and Vg. since f required to compute D Alternatively, we use super- vised learning to train an environment model M, that is approximated by the neural network for the unknown dynamical system. Then, FH and Vo can be updated by the gradients of My. Since coefficients of M,, do not have clear physical meaning of the environment (which are weights and biases of the network), we define actions of the adversary as the additive error to the output of Mn- However, perturbations imposed by the adversary's action may lead to unstable training, or even the non-existence of an asymptotically stable equilibrium. Hence, we limit the adversary's 's action to a certain range, which can be tuned practically to balance the stable training and adversary learning.
Adversary's reward design. Adversary should be assigned a higher reward if it leads the system an unstable state at each time step, which is contrary the controller's goal. For example, in the task where we aim to design a controller to swing the pendulum into a upright position, the reward of the adversary can be set as the square of the normalized angle between the pendulum and the vertical direction. The reward functions for training the adversary in all the tasks used in the paper are shown in T in Appendix A. In general, the controller's s action that tends to stabilize the system will decrease the reward for the adversary, while the adversary policy that achieves higher reward will prevent the controller from minimizing Lp.
Transition kernel. The transition kernel is determined by the system dynamics f. Given fixed control interval At, it can be derived as (-(aare a², ,a")) = 2z + f( af ag)
Given the controller policy Ty. the goal of RL is to find the optimal adversary policy n that maxi- mizes the following state value function:
π. = arg max Th (TT) = Eatt wa N
h r ahe a
(9)
where the state value function V (TT)) denotes the expected cumulative discounted reward starting from initial state vn- Provided with an appropriate adversary's reward design, Fे can produce the worst-case perturbation sequence, which adversarially destabilizes the system to the most extent.
4.3 ADVERSARIALLY ROBUST CONTROLLER LEARNING
Given an adversary policy oे trained by RL that performs the worst-case perturbation to the con- troller, we formulate the adversarially robust controller learning problem
min Lo,0,, 0", 0"), s.t. + = f(x a", aa"), al Tpp a" TT arg max VTT (mT)) 8.8
(10)
Note that formulations of the objective functions for neural Lyapunov-controolee and RL-adversary are totally different, hence the adversarial learning problem here cannot be formulated as a two- player zero-sum game. The proposed ARNLC in Algorithm II uses an alternating procedure to solve this problem approximately, where we summarize it for the case of unknown system dynamics f. Training environment model: at each iteration e, we sample Mj transitions in the environment with a random policy and update the environment model by minimizing the error between its pre- diction Mz a) and the next state x' on the transitions. Training controller's and adversary's policies: at each outer iteration i, we perform a two-stage optimization. i) We train the controller policy and Lyapunov function while the adversary policy is fixed based on the Lyapunov theory. We initialize a state dataset S by randomly sampling from X. At each inner iteration jリz we con- struct (ax, a a%, a'k he on the state subset of size M3 from S, and update n. and Vo by performing
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adversarially robust neural Lyapunov control (ARNLC) for unknown dynamics
Input: unknown environment M and state space A'
Output: learned policies n and Tu
I: Initialize: 1 for environment model M,, H for controller μa, 0 for adversary 0 for Lyapunov function Ve. control interval At, uniformly random policy R,
2: for e = 1, 1, .. do
Ne
& train an environment model for the system 0i, Tiji=l from M using nf with At
3: Sample transitions (ii
4: Update M, by minimizing M Em (M,(22 (z) x4 w.r.t. 77
5: end for
6: for := 1. Nster do
train controller
7: Randomly sample M2 states S ( from state space
8:
for ju = ..2....N,, do
9:
of nh (ax). of (r(E)) and u'c = (ax of)) + og on { sampled from Tu. Vo min (0, 0r, )*)) on {ze: at, ag, 2 )
10:
SGD update
11:
Find counter-example set 9 of size M4 following criterion in Eq. N and S t S end for
12:
13:
for ju = 1.2..... N2 do
train adversary
14:
(.n, ohi oh, Th n=) +- generate Tyy TT))
15:
Tr policyOptimize (172, of og Th rnla=l (80)
Neras
16:
end for
17:
end for
stochastic gradient descent (SGD) w.r.t. Eq. ( ii) We train the adversary policy while the con- troller policy fixed. At each inner iteration jr. we generate transitions on learned environment model M, with the controller's and adversary's 's policies. We then perform the policy optimization method from RL to update N. on these generated transitions. For known system dynamics, learning of the environment model at Lines ZHS in Algorithm II] is not required, and the transition generation in the two stages follows the known system dynamics ʃ. Due to space limit, the proposed ARNIC for known system dynamics is provided in Appendix
EXPERIMENT
We evaluate our proposed ARNLC algorithm on several control tasks, where the system dynam- ies are variable by external forces or perturbations on environment parameters. We compare our ARNLC with NLC (Chang et all 2019), PNLC (perturbed NLC) in Section I. RARL (Pinto all 2017) and Robust MPC (Löfberg, 2003: 2012). We use proximal policy optimization (PPO) (Schul man all 2017a) as our baseline RL algorithm for the adversary training at Line 15 in Algorithm In NLC and PNLC, we build neural networks for the controller policy and Lyapunov function, which are updated by minimizing the Lyapunov risk in Eq. (L) and Perturbed Lyapunov risk in Eq. (B), respectively. A uniformly random adversary policy is applied to impose perturbations in PNLC. In ARNLC and RARL, build neural network for the adversary policy and update it with PPO. In ARNLC, the controller policy is optimized together with the empirically perturbed Lyapunov function in Eq. (5), where the controller policy learned by PPO with the negative adversary ward. Since the RL-based adversary and discrete-time predictor of environment model are learned in ARNLC, we compute the difference of I. Lyapunov function ((x(+ae) V(x,)) instead of the time derivative ((xz) (see Appendix A. for detailed explanation). In robust MPC, a bounded pertur- bation is added to the system dynamics where the controller takes actions by searching the optimal feedback among all feasible horizons generated by the perturbed function. In our experiments, we find that the computation time for Robust MPC at each control step may exceed the control interval, which would lead to the controller failure in practice. But we simply neglect this and consider its simulation performance. For detail of the experimental settings, please refer to Appendix A. Our experiments are designed to answer the following questions.
Can our proposed ARNLC still achieve asymptotic stability in face of the worst-case perturba- tions? Will ARNLC reach the stability faster than the other baseline algorithms?
How will the controller's performance of ARNLC degrade in the entire perturbation space?

---

Under review as a conference paper at ICLR 2023
Table 1: Types of perturbations for each task
Task
Perturbation Type
Pendulum
Mass of Ball, Length of Pole, Friction Coefficient, Gravity
Cart Pole
Mass of Cart, Length of Pole, Mass of Pole, Gravity Velocity of Car, Radius of Path Pole1 Length, Pole1 Position of the Center of Mass
Car Trajectory Tracking 2-link Pendulum
Will ARNLC suffer from the issues of RL methods (Tallec al 2019 Park et all 2021), i.e.,
being sensitive to control intervals?
5.1 CONTROL OF PERTURBED NONLINEAR SYSTEMS
balance a pendulum (one end attached to ground by joint) by applying force to a ball at the other end. The system has two state variables, angular position ip and angular velocity o of pendulum, and one control input a" on the ball; 2) Cart Pole: control a cart (attached a pole by a joint) by applying a force to prevent the pole from falling. State variables of the system are cart position x, cart velocity 4 pendulum angular position & and pendu- lum angular velocity cb. Control input is force a" applied the cart; 3) Car Trajectory Tracking: control a car to follow a target circular path. The system has two state variables: the distance error He and angle error yoe between current car position and the target path. The control input is force al on the car; 4) 2-link Pendulum: control a two-joint pendulum system (two pendulums are linked by a joint and one end is linked to ground by another joint) to keep both pendulums upright. The system has four state variables: the angular position (0 and angular velocity Φ of the first pendu- lum, the angular position '22 and angular velocity L2 of the second pendulum. Two control inputs forces af and at the two joints. We evaluate different comparison algorithms on these tasks with perturbed environment parameters as shown in Table II under both known and unknown sys- tem dynamics settings. For example, in the pendulum task, the friction coefficient can be changed at each time step. We set two test scenarios: í) perturbations are randomly generated w.r.t. a uniform distribution at each control step, which are larger than training ones; ii) perturbations are taken w.r.t. the trained adversary policy of ARNLC.
We consider four balancing tasks. 1) Pendulum:
We run the training process of learning-based algorithms, i.e., ARNLC, NLC, PNLC and RARL on these tasks until the convergence. Here, we slightly modify and improve the original NLC and PNLC to make them compatible with the setting of unknown system dynamics. We then deploy the trained policies and robust MPC in the two test scenarios. For known system dynamics, control curves under uniform (U) perturbations are shown in Figs. I[a Iic Tee and [[g] while control curves under worst-case (W) perturbations learned by the adversary are illustrated in Figs. [[b] Tda
I0 and [h). For unknown system dynamics, control curves under uniform (U) perturbations are shown in Figs. [T] [[K] [ and [o]. while control curves under worst-case (W) perturbations learned by the adversary are illustrated in Figs. I0
[n and [[p. The horizontal axis is the control time, while the vertical axis is the system state. Here set the fixed control interval to 0.01s, and state zero as the equilibrium point. We observe that by incorporating a RL-based adversary during training, our ARNLC can achieve asymptotic stability under both test scenarios in all the tasks, while it reaches the stability the fastest compared to the other baselines under both conditions of known and unknown system dynamics. Though NLC reaches the stability in some tasks, it fails to reach the cquilibrium point in Car Tracking W. 2-link Pendulum U and W with known system dynamics and Car Tracking and W. 2-link Pendulum W with unknown system dynamics. PNLC trained under uniform sampled perturbations outperforms NLC in some tasks (e.g.. Pendulum and Cart Pole), but worse in Cart Tracking U and 2-link Pendulum W and U with known system dynamics and 2-link Pendulum U with unknown system dynamics. RARL and robust MPC fail to reach the stability in the 2-link Pendulum task.
5.2 GENERALIZATION IN PERTURBATION SPACE
We further evaluate the generalization capability of trained policies of learning-basee algorithms in the entire perturbation space. We exclude the evaluation of robust MPC here, since it requires to know system dynamics under each perturbation setting, which is an unfair comparison. Besides, we additionally evaluate ARNLC and RARL for unknown system dynamics in the Inverted Pendulum
7

---

Under review as a conference paper at ICLR 2023
peedarr areuk Senton 1923
POX DI DorthrNe
MALE MA
MALLC
-3.00
0:00
(a) Pendulum U
(b) Pendulum W
(c) Carl Pole U
(d) Carl Pole W
ve Car
Wencee Ser
Lengh Peosdamm
teab
lengrs Penssaum
8
-2.10
8
-2.1
2
-8.00
Srate
(c) Car Tracking U
(f) Car Tracking W
(g) 2-link Pendulum U
(h) 2-link Pendulum W
Wetten Contidect
Ericaan conticert
Mars cf
Pendulum U
Pendulum W
(k) Cart Pole U
()) Cart Pole W
Vulluity Ca
vulicty of
Langhn Peeamm
Lengh Netremm
PANCE
Nac
(m) Car Tracking U
(n) Car Tracking W
(0) 2-link Pendulum U (p) 2-link Pendulum W
Figure 1: Control curves of Pendulum, Cart Pole, Car Tracking and 2-link Pendulum with different
perturbation types under uniform perturbations (U) and learned adversary's worst-case (W) pertur-
bations in testing. I[a) known system dynamics; [[0 unknown system dynamics.
task provided by MuJoCo (oodorov el all 2012), which controls cart (attached to pendulum) to balance the whole system and keep the pendulum upright. While NLC and PNLC are not compared, since they require known system dynamics during training and cannot be trained in the Inverted Pendulum task. We use the cumulative negative reward of adversary evaluate the performance of controller policy in the environment with a certain perturbation, where a higher negative reward in- dicates the better performance of controller. The performance heatmaps of these five tasks achieved by different algorithms are shown and compared in Fig. @ with both known and unknown system dynamics, and [[g] with unknown system dynamics, where the performance is averaged on ten equal-length runs. We observe that ARNLC achieves the best generalization performance under both conditions of known and unknown system dynamics except for Car Tracking with unknown system dynamics. PNLC generalizes better than NLC in Pendulum and CartPole with known sys- tem dynamics, while showing similar or even worse performance in other tasks. RARL presents the worst performance in all the tasks except for Pendulum.
5.3 IMPACT OF CONTROL INTERVALS
Eventually, we evaluate the impact of different control intervals to our ARNLC, which are set to 0.01s, 0. 0.005s and 0.001s, respectively. The resulting control curves obtained for Pendulum shown in Figs. Ia H and Bla SUT. We observe that ARNLC and other Lyapumov based baselines can achieve asymptotic stability with all different control intervals, while RARL is sensi- tive the change of control intervals also verified in (Tallec et all, 2019) and fails to reach the equilibrium state. Note that here we report the most important results in the main text, please also see Appendices A. and A. for additional results.
8

---

Under review as a conference paper at ICLR 2023
AIRAL
18.1
PALL
SANL
ARAIC
NLC
PXLC
RARL
+200
141
-50099900047
Trada of
(a) Pendulum
(b) Cart Pole
RARL
-180
414
WW
(c) Car Tracking
(d) 2-link Pendulum
ARV
NIC
PRIC
MAR
ARNLC
MLC
Mic
NARL
mate
mas of
(c) Pendulum
(f) Cart Pole
SSANNAA
(g) Car Tracking
(h) 2-link Pendulum
Figure 2: Heatmap of averaged cumulative negative adversary's 's reward with known and unkown system dynamics. In (d) and (h), RARL fails to converge. La d known system dynamies; Me 2 unknown system dynamics.
Enitton Conthrieet
metter Confidier
Contectere
Napastadd
IBC
(a) 0.1s U
(b) 0.1s W
(c) 0.005s U
(d) 0.005s W
Contieeer
PSC
(e) 0.001s U
(1) 0.001s W
(g) Inverted Pendulum
Figure 3: (a)-(f) Control curves of Pendulum with control interval set to 1s 0.005s and 0.001s,
respectively; (g) heatmap of averaged cumulative negative adversary's reward with unknown system dynamics.
CONCLUSIONS AND FUTURE WORK
We have proposed ARNLC to improve the robustness and generalization in stability control tasks for nonlinear dynamical systems. Specifically, we formulated a perturbed Lyapunov risk stemmed from Lyapunov theorem to jointly update the controller and candidate Lyapunov function under pertur- bations generated by an adversary during training, where the adversary was trained by RL method to destabilize the system. We adopted an alternative training procedure to update the controller and adversary. We have empirically evaluated ARNLC in several stability control tasks, demonstrating its robustness under different perturbations and better generalization in entire perturbation space. An exciting future research direction could be to extend our ARNLC to non-stability control tasks for the dynamical systems that are not required to achieve equilibrium point, where the reward func- tion for the adversary can alternatively be designed based on the imitation error, like in the imitation learning. We hope to revisit this in our future works.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Abate, Daniele Ahmed, Mirco Giacobbe, and Andrea Peruffo. Formal synthesis of Iyapunov neural networks. IEEE Control Systems Letters, 5(3):773-778, 2020.
Tamer Başar and Pierre Bernhard. H-infinity optimal control and related minimax design problems: dynamic game approach. Springer Science & Business Media, 2008.
Alberto Bemporad and Manfred Morari. Robust model predictive control: A survey. In Robustness in Identification and Control, volume 245 of Lecture Notes in Control and Information Sciences, pp. 207-226. Springer, 1998.
Ya- Ea-Chien Chang, Nima Roohi, and Sicun Gao. Neural Iyapunov control. In Advances in Neural Information Processing Systems, pp. 3240-3249, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion reinforcement learning. International Conference on Machine Learming, pp. 1282-1289. PMLR, 2019.
Charles Dawson, Zengyi Qin, Sieun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barier functions. In Conference Robot Learning, pp. 1724-1735. PMLR, 2022.
Kunal Garg and Dimitra Panagou. Robust control barrier and control Iyapunov functions with fixed- time convergence guarantees. 202/ American Control Conference, ACC, pp. 2292-2297. IEEE, 2021.
Boris Houska and Mario Villanueva. Robust optimization for mpe. Handbook of Model Pre- dictive Control, pp. 413-443 Springer, 2019.
Jianchen Hu and Baocang Ding. Output feedback robust MPC for linear systems with norm-bounded model uncertainty and disturbance. Automatica, 108, 2019.
Shafiqul Islam, Peter Xiaoping Liu, and Ahdulmotaleb El-Saddik. Robust control of four-rotor un- manned aerial vehicle with disturbance uncertainty. IEEE Transactions on Industrial Electronics, 1233):1563 2015.
Johannes Köhler, Raffaele Soloperto, Matthias A. Müller, and Frank Allgöwer. A computation- ally efficient robust model predictive control framework for uncertain nonlinear systems. IEEE Transactions on Automatic Control, 66(2):794-801, 2021.
Huibert Kwakernaak and Raphael Sivan. Linear optimal control systems, volume 1072. Wiley- interscience, 1969.
Eugene Lavretsky and Kevin A Wise. Robust adaptive control. In Robust and adaptive control, pp- 317-353. Springer, 2013.
Chengju Liu, Jing Yang, Kang An, Ming Liu, and Qijun Chen. Robust control of semi-passive biped dynamic locomotion based on a discrete control Iyapunov function. Robotica, 3845):1345-1355 2020a.
Lei Liu. Yan-Jun Liu, Aiqing Chen, Shaocheng Tong. and CL Chen. Integral barrier Σyapunov function-based adaptive control for switched nonlinear systems. Science China Information Sci- ences, 6313) 2020b.
Johan Löfberg. Approximations of closed-loop minimax MPC. In 42nd IEEE Conference on Deci- sion and Control, pp. 1438-1442, IEEE, 2003.
Johan Löfberg. Pre- and post-processing sum-of-squares programs in practice. IEEE transactions on automatic control, 5455):000--0111 2009.
Johan Löfberg. Automatic robust convex programming. Optimization Methods and Software, 27(1): 115-129, 2012.
10

---

Under review as a conference paper at ICLR 2023
Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy A. Mann, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misxpecification. In International Con- ference on Learning Representations, 2020.
Arash Mehrjou, Mohammad Ghavamzadeh, and Bernhard Schölkopf. Neural Iyарunov redesign. In Proceedings of the 3rd Annual Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pp. 459-470. PMLR, 2021.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural Computation, 17(2):335- 359, 2005.
Armin Norouzi, Ali Barari, and Hadi Adibi-Asl. Stability control of an autonomous vehicle in over- taking manoeuvre using wheel slip control. International Journal of Intelligent Transportation Systems Research, 18(2):320-330 2020.
Anil Kumar Pal, Shyam Kamal, Shyam Krishna Nagar, Bijnan Bandyopadhyay, and Leonid Frid- man. Design of controllers with arbitrary convergence time. Automatica, 112:108710, 10, 2020.
Seohong Park, Jaekyeom Kim, and Gunhee Kim. Time discretizationninvaaint safe action repetition for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 267- 279, 2021.
Pablo A Parrilo.
Structured semidefinite programs and semialgebraic geometry methods in robust- ness and optimization. California Institute of Technology, 2000.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Priyabrata Saha, Magnus Egerstedt, and Saibal Mukhopadhyay. Neural identification for control. IEEE Robotics and Automation Letters, 6(3):4648 4655, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017a. URL /ttp://arxiv.orr//00
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXivv1707.06:: 2017b.
Aman Sharma and Narendra Kumar. Lyapunov stability theory based non linear controller design for a standalone pv system. In 2020 IEEE International Conference for Innovation in Technology (INOCON), pp. 1-7. IEEE, 2020.
Zhongqi Sun, Li Dai, Kun Liu, Yuanqing Xia, and Karl Henrik Johansson. Robust MPC for tracking constrained unicycle robots with additive disturbances. Automatica, 90:172 1884, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pp. 6096-6104, PMLR, 2019.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In International Conference on Machine Learning, pp. 6215-6224. PMLR, 2019,
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Nur Uddin, Hendra G Harno, and Rianto Adhy Sasongko. Altitude control system design of bicopter using Iyapumov stability approach. In 2021 International Symposium on Electronics and Smart Devices (ISESD), pp. 1-6. IEFE, 2021.
11

---

Under review as a conference paper at ICLR 2023
Sam Witty, Jun K Lee, Emma Tosch, Akanksha Atrey, Kaleigh Clary, Michael L Littman, and David Jensen. Measuring and characterizing generalization in deep reinforcement learning. Applied Al Letters, 2(4))e45, 2021. 2021.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane S. Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In Advances Neural Information Processing Systems, 2020.
Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Represeniations, 2021.
Pan Zhao, Yanbing Mao, Chuyuan Tao, Naira Hovakimyan, and Xiaofeng Wang. Adaptive robust quadratic programs using control Iyарunov and barrier functions. In 59th IEEE Conference Decision and Control, CDC, pp. 3353-3358. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 CONTROLLER LEARNING FOR DISCRETE- TIME CONTROL
We consider a discrete-time timeete-timeeeee dynamical system sampled from continuous-time dynamical system with fixed time interval At:
z I++ = +At f(z,, aft of),
(A-1)
where at A and aj Aude are the controller's action and adversary's action time N following their policies xy and nu. respectively; and z is the state at time t + At (i.c., at the next sampling step).
Theorem A-1 (Lyapunov stability theorem for discrete-time dynamical systems), For discrete-time dynamical system in Eq. (A-1)) if there exists continuous function V X - IR
such that
v(0)= = 0; and V(x) 0, Ve E {0}; and V(x') - V(x) < O;
(A-2)
then the system is asymptotically stable at x 0, where V is called a Lyapunov function. functioon
For the discrete-time time dynamical system, we focus on the difference of the Lyapunov function instead of the time derivative in the continuous-time case. To satisfy the Lyapunov stability theorem, require that: i) the value (()) is zero; ii) the value of V(x) is positive; and iii) the value of the difference V (xx) (xx) is negative.
Definition A-1 (Discrete-time perturbed Lyapunov risk for controller). We consider candidate Lyapunov function Ve paramterized by 0 for discrete-time dynamical system in Eq. (A-I). In the presence of adversary policy ny parameterized by 0,, the discrete-time perturbed Lyapunov risk for the controller μ is defined by:
Lp (0,0" 1t. 0²) = Ex) maxx(-V(<)) + max(0, Vo(x') - Vo(z)) vo(z))+
(A-3)
where p(X) is the state distribution.
In practice, we use the following empirically perturbed Lyapunov risk, which is an unbiased estima- tor of Eq. (A:3)):
N
In.p (0, "0"0"")
N max( -Vo((=)), ma Vo(a!) Vo(x:)) v2 (A-4)
1
We use the negations of the Lyapunov conditions Lyapunov stability theorem define the counter-examples, which means that if the value of Vo(x) is non-positive or the value of the differ- ence V²(2') - V=(x:) non-negative, then the state x considered as a counter-example. Therefore, the criterion for discrete-time dynamical systems can be set as follows:
((x) (x( 0 Vo(x') Vo(x) 2 Vr E X (()-
(A-5)
A.2 ARNLC FOR KNOWN SYSTEM DYNAMICS
We summarize the ARNLC for known system dynamics f in Algorithm A-l of this Appendix. Its difference from ARNLC for unknown system dynamics lies in that there is no need to learn an environment model. This algorithm follows the same alternating procedure as described Section 4.3 in the main text to train the controller's and the adversary's policies.
A.3 DETAILS EXPERIMENTAL SETTINGS
The network architectures for Lyapunov function and controller policy are tuned for each task and are summarized in Table A- The detailed hyperparameters for adversary policy training are sum- marized in Table A-2 The environment specifie parameters summarized in Table A-3
13

---

Under review as a conference paper at ICLR 2023
Algorithm A-1 Adversarially robust neural Lyapunov control for known system dynamics
Input: known dynamical system f and stale space A' Output: learned policies n and Tr
I: Initialize: 0 for controller μa, 0" for adversary v, 0 for Lyapumov function Ve, control interval At
for 2, Nuter do
3: Randomly sample M1 states S {
Mr from state space A'
4: for ju 1. do
train controller
5: a = nT (2x), 0z (x(x)) and x'c = f(zn, ox at) on (zu has sampled from S Adz
6: TTy, V/ + min (0, 0", 0*) on {æx, a ag 2/ Pk 8,80
> use SGD to update
Find counter-example set O of size Ma following criterion in Eq. (A-5) and + SUN end for
8:
9:
for j. = 1.2,... do Nerii generate(f: TTy TT)
train adversary
10:
{ - 0j Th]n=l
Neros nv f- policyOOpiimize ((rz,, off 0h rala=l .80)
12:
end for end for
13:
Table A-1: Lyapunov function and controller policy specific settings
Network Architecture
Training Iterations
Learning Rate
Batch Size
Task
Lyapunov Function
Controller Policy
Pendulum
2-6-1
2-1
200
Cart Pole
4-64-64-1
4-64-64-1
20
0.01
512
Car Trajectory Tracking
2-64-1
2-1
12
2-link Pendulum
4-6-1
4-64-2
Inverted Pendulum
4-64-64-1
4-64-64-1
40
Table A-2: Adversary hyperparameters
Parameter
Value
Optimizer
Adam
Learning rate actor policy network
10 10-4
Learning rate for critic network
x 10-3
Discount (7)
0.99
Clipping ratio
0.2
Number of hidden layers (all tasks)
Number of hidden units per layer Nonlinearity
64
tanh
Table A-3: Environment specific parameters
Task
State Space
Adversary Reward
Pendulum
[ S
|+|
<
2
Cart Pole
|a||
S 1,
< 1,
|
< 0.2,
|
<
-1
|||
< 0.2
Car Trajectory Tracking 2-link Pendulum
x.
< 0.5,
| 0.5
[r.
Ipc
|
< 0.8,
( 0.8.
0.8,
| 0.8
+
[p2
Inverted Pendulum
lx]
1, [
L,
0.2,
<
kf 0.2
if
A.3.1 SYSTEM DYNAMICS
Pendulum. The system is shown in Fig. A- and the system dynamics be described
ē mgl sin(O) bō
(A-6)
ml?
where 0.15, g 99.11 0.1 and 0.5.
14

---

Under review as a conference paper at ICLR 2023
u
0
Figure A-1: Schematic diagram of the Pendulum task.
The adversary reward in this environment is set as the square of the normalized angle between the pendulum and the goal state. The actions of the adversary are set to adding an external force to the pendulum, changing the acceleration of gravity in the environment, changing the length of the pendulum, changing the coefficient friction, and changing the mass the ball, respectively.
Cart Pole. The system is shown in Fig. and the system dynamics can be described as
10 sin m. +
sin + cos
I my cos? A
(A-7)
me tMi
u + my m,1( 2 sin
16
0 cosss 0)
mc + mp
where g = 9.8, me = 1.0, m, = 0.1,1=00 0.5.
Figure A-2: Schematic diagram of the Cart Pole task.
If the angle between the pendulum and the vertical direction less than 0.2, the adversary gets reward -1. The actions of the adversary are set to adding an external force to the cart, changing the acceleration of gravity in the environment, changing the length the pendulum, changing the mass of the cart, and changing the mass of the ball, respectively.
Car Trajectory Tracking. The system is shown in Fig. A-3 and the system dynamics can be described as
$ cos(
1 don(ss)
d. sOin)
(A-8)
0. ttanuu)
vK(s) cos(
dsk(s)
where e = 6, 1
The adversary reward in this environment is set as the sum of the absolute value of the distance error and the absolute value of the normalized angle error. The actions of the adversary are set to adding extemal force to the pendulum, changing the velocity of the car, and changing the radius of the target path, respectively.
15

---

Under review as a conference paper at ICLR 2023
poth
Figure A-3: Schematic diagram of the Car Trajectory Tracking task.
2-link Pendulum. The system is shown in Fig. A- and the system dynamics can be described as
0, U22(U1 + a1203 (02 01) + bi sin 01] 12 cos(02 - 01 01) Iu2 + a210f sin(O) 02) b2 sin 02]
011022 012021 cos 02) cos ( 81)
82- 021 cos(01 - 02)[ui + 01202 sin(02 - 01) + 6, sin 0 - a /u1|122 + a2 103 sin(O1 - 02) + bo sin 02]
012021 cos(O2 01) cos 02) a11022 022
(A-9)
where
a11 11 + m11a + lme.
d12 a21 mzh1a2
422 I2 m2222
(A-10)
b (milel + (1m2)99
5 = (m2422gg
and 1 =1, my m2 1, = 1, = l2 = 0.5, g 9.8.
Figure A-4: Schematic diagram of the 2-link Pendulum task.
The adversary reward in this environment is set the sum of the normalized angles between the two pendulums and the goal state. The actions of the adversary are set adding two external forces to the pendulum, changing the length of the first pendulum, and changing the position of the center of mass the first pendulum, respectively.
Inverted Pendulum. This task is provided by MuJoCo, where the goal is to control cart (attached to pendulum) to balance the whole system and keep the pendulum upright, shown in Fig. A 5 Since the system dynamics of Inverted Pendulum are unknown, it is impossible to generate perturbations concrete physical parameters at each control step. Therefore, we only conduct the generalization test in the entire perturbation space for this task. the angle between the pendulum and the vertical direction is less than 0.2, the adversary gets reward -1. The actions of the adversary are set as the additive error to the output of the learned environment model.
16

---

Under review as a conference paper at ICLR 2023
Figure A-5: Schematic diagram of the Inverted Pendulum task.
A.4 ADDITIONAL. EXPERIMENTAL. RESULTS WITH KNOWN SYSTEM DYNAMICS SCENARIOS
This section reports experimental results on all the evaluation tasks under perturbation types that have not been presented in the main text.
A.4.1 CONTROL OF PERTURBED NONLINEAR SYSTEMS
We run the training process of learning- based algorithms on Pendulum, Cart Pole, Car Trajectory Tracking, and 2-link Pendulum under different perturbation types as shown in Table 1 in the main text until convergence. Control curves under uniform (U) perturbations are shown in Fig. A4 while control curves under worst-case (W) perturbations learned by the adversary are illustrated in Fig. A-A We observe that our ARNLC can achieve asymptotic stability under both test scenarios in almost all the tasks, while it reaches the stability the fastest compared the other baselines. The uniform external force perturbations in the Car Trajectory Tracking task is the only test scenario where our ARNLC performs the worst, as shown in Fig. A-6(K) Though NLC reaches the stability in some tasks, it fails reach the equilibrium point in Car Trajectory Tracking W, 2-link Pendulum W and U. RARL and robust MPC fail to reach the stability in 2-link Pendulum.
We additionally compute the percentile negative adversary rewards for controller policies achieved by each algorithm further evaluate their robustness. We each policy with 100 dif- ferent initial system states in each perturbed task, and then sort the cumulative negative adversary reward of each tions are shown in Fig. A while percentile plots under worst-case (W) perturbations learned by the adversary illustrated in Fig. A- In general, control policies that can gain higher rewards the same percentile perform better. While control policies in a lower percentile have better control performance they receive the same reward, i.e., they can gain higher rewards with more episodes. We observe that our ARNLC can receive the highest rewards under both test scenarios in all the tasks. RARL sometimes fail to reach the stability in Car Trajectory Tracking, and therefore, RARI. sometimes receives extremely low rewards in this task.
obtain the n-th percentile. The results obtained under uniform (U) perturba-
17

---

Under review as a conference paper at ICLR 2023
Length of Pite
SEOZUI arparr poaat on Fred
Acceleration of Craxity
Hors of Dall
Esterna Force
AANAC
Seddzum erQueer on
8:25
RAAL
CAMAC Rarmuoo
Bree
Brrwe
(a) Pendulum U
(b) Pendulum U
(c) Pendulum U
(d) Pendulum U
wrpple partic (142)
Accederstion Gravity
Externa
8:30
pootton
13.31
-3:31
MCC
-3.32
Mili
-0035
(ineen)
00
75 100 12.3 150 175 200
(c) Cart Pole U
Cart Pole U
(g) Cart Pole U
Cart Pole
0slanc enor Int
Basenc sire te1t
PALC
Ficket MNC
MLC
MCC
MCC
KANL
BARL
KARL
Uinels
(i) Car Tracking U
(i) Car Tracking
(k) Car Tracking U
(I) Car Tracking
Postion of the Ceeter of Noes Prndium))
Podtion of the Center of Maae Predicmm))
pooduum: onquie portoon (960
Externa Force
Eaterna Forre
angale po
2.2
01
20
(m) 2-link Pendulum U
(n) 2-link Pendulum U
(o) 2-link Pendulum U
(p) 2-link Pendulum U
Figure A-6: Control curves of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum
under uniform (U) perturbations in testing.
18

---

Under review as a conference paper at ICLR 2023
Length of P
Acce eradion Sreaiy
Sedzun exuue DOEET on (N4D
Hars of Dall
Esterns Farro
AAMEC
Brow
Smet0
Bmee
(a) Pendulum W
(h) Pendulum W
(c) Pendulum W
(d) Pendulum W
partic (182)
8.22
postion
Acceterstion Grauity
MBC
2aed Sam aner ponette (181)
ncel
peodicn
2.115
2.000
External
8:30
PAB
MC
18044
RAB:
000
12. 150 135 253
sinvest
(c) Cart Pole W
(f) Cart Pole W
gg) Cart Polc W
(h) Cart Pole
8:00
PALC
dhaeeco siree lrst
PULC
MCC
MCC
-0.30
KARL
Unneld
Unnes
(i) Car Tracking W
(i Car Tracking W
(k) Car Tracking W
(I) Car Tracking W
pardaam expale partos meah)
Postion of the Center Mass of Pend Jum-
Position of the Center of Hove of Pescuuu
Exeennl Force
Eewersa Faice
0.00
MBLC
-3.00
2:30
-3155
AB11C
(m) 2-link Pendulum W (n) 2-link Pendulum W
(o) 2-link Pendulum W (p) 2-link Pendulum W
Figure A-7: Control curves of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum
under learned adversary's 's worst-case (W) perturbations in testing.
19

---

Under review as a conference paper at ICLR 2023
Length of Dow
Acoeeration Breaty
New of Del
Tretoon Conficient
-403
CASEC
(ANAC
-465
CASEC
.003
PSI
Edoost
2000
Edooal
Febos
-204
(a) Pendulum U
(b) Pendulum U
(c) Pendulum U
(d) Pendulum U
External Force
Acceloration Gracity
Length
1123
2003
2000
-200
2019
2015
NCC
3019
-101
1001
CASRC
MLC
3953
2409
3509
7000
Rercentle
Percesale
Percerde
Perceede
Pendulum
Cart Pole U
(g) Cart Pole U
Cart Pole
of
AMACC PSL
2153
AMBCC
MAL
-1d
-504
Peb MMC
-160
203
J633
NLC
3933
199
:903
-262
FABL
-200
RAPS
30
40
90 100
28
00
30
106
20
40
30
100
100
(i) Cart Pole
(i) Cart Pole U
(k) Car Tracking U
(I) Car Tracking
Exteral Force
Length of P
-208
Pration of the Center of Mas of Ferduum
External force
-489
-009
PeL
-000
2009
3000
-100
KARL
1289
1020
08
1409
02
100
45
160
(m) Car Tracking U
(n) 2-link Pendulum U
(o) 2-link Pendulum U
(p) 2-link Pendulum U
Figure A-8: Percentile plots of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum
under uniform (U) perturbations in testing.
20

---

Under review as a conference paper at ICLR 2023
Length of Dow
Acceeration Breaty
New of Del
Tretoon Conficiint
-103
-600
CASEC
(ANEC
AASEC
-000
-003
PSI
-183
Edooot
Edooal
1005
Tebos
3000
(a) Pendulum W
(h) Pendulum W
(c) Pendulum W
(d) Pendulum W
External Force
Acceleratio
Length
2123
2003
2000
X019
2015
3019
100
1000
MLC
3953
-120
350
7300
2409
Rercentle
Percesile
Percende
Percedde
(c) Pendulum W
(f) Cart Pole W
gg) Cart Polc W
(h) Cart Pole
2153
AMBC
PSL
MAc Tebar
-20
fob MMC
2000
:015
303
:015
J633
MC
N.C
199
14%
-203
3933
-433
30
50
90 100
00 30
08
20
330
66
160
(i) Cart Pole W
(j) Cart Pole W
(k) Car Tracking W
(I) Car Tracking W
Externa force
Length of P
Poption of the Center of Mas of Perduum
Exterall force
-20
-200
469
2009
-000
.000
1000
KARL
-000
-200
-2200
-1000
08
109
45
100
(m) Car Tracking W
(n) 2-link Pendulum W
(o) 2-link Pendulum W
(p) 2-link Pendulum W
Figure A-9: Percentile plots of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum under learned adversary's 's worst-case (W) perturbations in testing.
21

---

Under review as a conference paper at ICLR 2023
A.4.2 GENERALIZATION IN PERTURBATION SPACE
In this subsection, we evaluate the generalization capability of learning-based algorithms in the entire perturbation space generated from other combinations of perturbation types that are not pre- sented in the main text. We observe that ARNLC achieves the best generalization performance among all the combinations of different physical parameters. PNLC generalizes better than NLC in most combinations, while showing worse performance in Figs. A-10(1) and A-10(]]
ASTUIC
'EL
PILL
RAIL
+35
SO0 FIME hall
AWMAAA Niczon confICIEE
(a) Pendulum
(b) Pendulum
AVALL
NBC
PILL
MAHL
ARNLE
BLC
PLLC
RARL
4,0% 2000
(c) Pendulum
(d) Pendulum
ARNLC
PMU
RARL
NCC
(c) Pendulum
(1) Cart Pole
RARL
(g) Cart Pole
(h) Cart Pole
NIC
JMMANNY 3A0055333
(i) Cart Pole
(j) Cart Pole
Figure A-10: Heatmap of averaged cumulative negative adversary's reward for Pendulum, Cart Pole.
22

---

Under review as a conference paper at ICLR 2023
A.4.3 IMPACT OF CONTROL INTERVALS
We evaluate the impact of different control intervals (0.01s, O.Is, 0.005s and 0.001s) to our ARNLC on Pendulum with perturbation types (Length of Pole, Mass of Ball, Friction Coefficient, Acceleration of Gravity) different from the results presented in the main text. The resulting control curves obtained for Pendulum under uniform perturbations are shown in Figs. A- and Fig. And the control curves under perturbation of learned adversary are shown Figs. A- (a A and Fig. A-12 The resulting percentile plots obtained for Pendulum under uniform
perturbations are shown Figs. A--re) and Fig.
A-1 And the percentile plots under
perturbation of learned adversary shown in Figs. A-9a) A-e and Fig. A- The results
demonstrate that our ARNLC is robust to different control intervals.
We further evaluate our ARNLC in the continuous-timg dynamical system of Pendulum, with the system dynamics given in Eq. (A-0). Here we show that our ARNLC can achieve asymptotic stability faster than NL. in Figs. I-1 and can receive higher rewards in Figs. A- A-15(h). The regions of attraction estimated by our ARNLC are comparable with those estimated by neural Lyapunov Controller. This indicates that we still have the advantage of the larger region of attraction inherited from the original method after incorporating the adversarial training.
shown in Figs. A--5(I-A-15((I) demonstrating that those
pendulum anguler postion (rad)
Length of Pole
Mass of Ball
Accoloration of Gravity
3.1
ARNUC
ARNIC
0.0
perdulum position
2.5
M.C
pendulum anguler possion
M.C
2.0
PULL
BARI
PNLC
PARL
3.5-
1.0
-1.0-
ARNLC
0.5
-1.5-
PNLC
0.0-
R.AL
-0.5-
0
1
11.0
timels]
timels]
timets]
(a) 0.1s
(b) 0.1s U
(c) 0.1s U
Lenoth of Pole
Moss Ball
pendulum angular posilion (rad)
Acceleration Gravity
3.0
pendalum angular position
0.0
anguli pustton
2.5
PULC
2.0
-1.0-
1.5-
-1.-
1.0
-2.0
ARKLI
0.5
ARNLC
MLC
0.0-
PNLC
-20-
-0.5
timels)
timels)
timets)
(d) 0.005s U
(c) 0.005s U
(f) 0.005s U
pendulum amguler (ted)
Length of Polo
pendulum enguler postion (rad)
Mass of Ball
pendulum sngulor postion (rad)
Acceleration of Gravity
0.0
0.0
2.50
-6.5-
-0.5-
3.25
-1.5-
-1.0-
ARKUC
-1.0
2,00
ARNLC
-1.5-
6.50
-25-
-2.0
-30
0.00
timels)
timels)
timets)
(g) 0.001s U
(h) 0.001s U
(i) 0.001s U
Figure A-11: Control curves of Pendulum under uniform (U) perturbations in testing with control interval set to Is 0.005s and 0.001s, respectively.
23

---

Under review as a conference paper at ICLR 2023
perdulum amguler posiiion (rae).
Length
Acceleration Gravity
ANNLIC
3.0
AMNLI M.C PNLC
pendulum enguler: position
PMLC
2.5-
PNLC
rendulum, sman coott
PARL
RARL
PARL
1.0
14
0.5
0.0-
-0.5
timelsi
timels)
timelsi
(a) 0.1s
(b) 0s W
0.1s W
pendu arnuie postton (co0)
Length Pole
Mess of Ball
pendulum anguler position [Gd)
Acceleration of Gravity
ARHLIC
ABNLC
PNLC
PNLC
ARNIC
NLC
PNLE:
PARI
timelsi
timets)
(d) 0.005s W
(c) 0.005s W
(f) 0.005s W
Length Pole
cendulum enguler position (rad)
Ball
pendiilum angular position (cad)
Acceleration ation Gravity
0.0
-2.5
1.0
2.50
-5.0
0.5
RARL
PYLC
3,00
ARULI
enguler
0.75
0.50
-0.5
-17.5
PMLC
0.25
-0.0
RARL
1.0
0.00
time(s)
timels)
timels)
(g) 0.001s W
(h) 0.001s W
(i) 0.001s W
Figure A-12: Control curves of Pendulum under learned adversary's worst-case (W) perturbations
in testing with control interval set to 0.ls, 0.005s and 0.001s, respectively.
24

---

Under review as a conference paper at ICLR 2023
Langhh of
Nees
Fretion Conticent
Agzelerator Graddy
ABsiC
ABAC
ABACC
PELC
MiL
-2
CMMCC
-13
Mar
Parcen
64
320
20
45
100
-25
20
(a) 0. U
(b) 0.1s U
(c) O
(d) 0.
Length Pate
Maas Bell
Frictir Confirinnt
Accele srien of Geaaity
APBCC
ARMLC
MPAL
PSLC
-50
PRLC
100
-200
22
60
120
10
Percentlee
Parcenti
Pe
Parcert
(e) 0.005s U
(l) 0.005s U
(g) 0.005s U
(h) 0.005s U
Length Pree
Henx Cal
Priction Coufiictett
Accelaration Gray
5000
12000
PNC
MEC
M0C
-3000
140
-2500
Percenti
Percert An
Parcentle
(i) 0.001s U
(i) 0.001 U
(k) 0.001s U
()) 0.001s U
Figure A-13: Percentile plots of Pendulum under uniform (U) perturbations in testing with control interval set to 0. 0.005s and 0.001s, respectively.
Length of
Neso
Fricton Contiicen
Accohrattor Craviyy
ABSIC
ABAIC
ABSIC
-12
FREE
PeLC
-13
-15
Parcar
300
25
166
43
20
66
(a) 0. W
(b) 0. W
(c) 0.1s W
(d) 0. W
Length of Pate
Maas Dell
Fricton Conffrient
Accateration of Cratity
AFMCC
ARMCC
PNLC
-300
PMLC
M&C
AMG.
200
120
12
Pocertic
Porcortic
Pacerbic
Poroort
(e) 0.005s W
(l) 0.005s W
(g) 0,005s W
(h) 0.005s W
Length Pare
Mass tel
Priction Conffitent
Acceteration Sraxity
-5000
-5000
20000
-0000
141
-2090
0050
100
PAR
Percerbiee
Parcert Nr
Percentie
(i) 0.001s W
(i) 0.001s W
(k) 0.001s W
(I) 0.001s W
Figure A-14: Percentile plots of Pendulum under learned adversary's worst-case (W) perturbations in testing with control interval set to 0.1s, 0.005s and 0.001s, respectively.
25

---

Under review as a conference paper at ICLR 2023
anpute posttn (nn))
Length of Pale
Mass of Ball
AETOAN angiler postWer (PNC)
Friction Conffcilet
Accelerstion of Grauity
W.F.
postoo
H.C
00 25 50
15 555 125 15.0 17a 253
03 25
50
35
125
150 1T.5
20.2
09 25 50
75 385
15.
115
00
25 50
75 18E 125 150 11.5
25.3
(a) Pendulum W
(b) Pendulum W
(c) Pendulum W
(d) Pendulum W
Longth Pa
Ma 3xt
hiclinn Cediloion
Acodhaclle of Charity
MALI
MILLE
-1000
1308
-3500
-1000
-3000
Facani
Pacart
(e)
Pendulum W
(I) Pendulum W
(g) Pendulum W
(h) Pendulum W
Length of
of
Fricton Coefficiet
Acceleration Gravity
ATUL
Anruc
Anguaa velocty/naais
wald
C o L
hald tegan
Aequar valecip/faaaa
vald legin
Vold legin
14
Anguia
1.
0.50
0.55
O Angletrad)
0 Angle(rad)
0.00
Angletrad)
Anglecrad)
(i) Pendulum W
(i Pendulum W
(k) Pendulum W
(1) Pendulum W
Figure A-15: Control curves, percentile plots and regions of attraction for pendulum balancing in continuous-time control under learned adversary's worst-case (W) perturbations in testing.
A.5 ADDITIONAL EXPERIMENTAL RESULTS WITH UNKNOWN SYSTEM DYNAMICS SCENARIOS
This section reports experimental results for unknown system dynamics. We carry out the same experiments on all the tasks, while assuming that we have no access the system dynamics. There- fore, we utilize Algorithm 1 in the main text in Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum tasks. The evaluation of robust MPC is excluded here, since requires to know exactly the system dynamics, which makes it no longer feasible.
A.5.1 CONTROL OF PERTURBED NONLINEAR SYSTEMS
We run the training process of learning-basse based algorithms on Pendulum, Cart Pole, Car Trajectory Tracking, and 2-link Pendulum utilizing Algorithm in the main text until convergence. Control curves under uniform (U) perturbations are shown in Fig. A- while control curves under worst- case (W) perturbations learned by the adversary are illustrated in Fig. A-T. We use the learned adversary in RARL to add perturbations in worst-case testing. This is because for unknown system dynamics, the adversary's actions are the additive error to the output of the learned environment model, and in testing, we require the perturbations in the true dynamics (e.g., physical parameters or external force). We observe that our ARNLC can achieve asymptotic stability under both test scenarios in almost all the tasks, while it reaches the stability the fastest compared to the other baselines. The learned adversary's external force perturbations in the Car Trajectory Tracking task is the only test scenario where our ARNLC fails to reach the equilibrium point, as shown in Fig. A-170) Though NLC reaches the stability in some tasks, it fails to reach the equilibrium point in Pendulum W, Car Tracking W and 2-link Pendulum U and W, when the perturbation type in testing is the external force. PNLC trained under uniform sampled perturbations outperforms NLC in some tasks (e.g., Pendulum and Cart Pole), but is worse in Car Tracking W and 2-link Pendulum U.
We additionally compute the percentile of rewards for controller policies achieved by each algorithm to further evaluate their robustness. We run each policy with 100 different initial system states in
26

---

Under review as a conference paper at ICLR 2023
each perturbed task, and then sort the cumulative negative adversary reward of each run to obtain the n-th percentile. The results obtained under uniform (U) perturbations are shown in Fig. A- while percentile plots under worst-case (W) perturbations learned by the adversary are illustrated in Fig. A-19 In general, control policies that can gain higher rewards at the same percentile perform better. While control policies in lower percentile have better control performance if they receive the same reward, i.c., they can gain higher rewards with more episodes, We observe that our ARNLC can receive the highest rewards under both test scenarios in all the tasks. RARL sometimes fails reach the stability in Car Trajectory Tracking, and therefore, RARL sometimes receives extremely low rewards in this task.
External
Accedention IEGrNNt
Mussi
siC
NAX
tmmbl
mmebi
tmarsi
Pendulum U
(b) Pendulum U
Pendulum
(d) Pendulum U
pandikm wrquarrrannnn poretton (ndd)
Eatomu Parss
Accenedion Cuanty
Lengh
Mox Cert
lo LEA aic
TNE
540
25
(e) Cart Pole U
(1) Cart Pole U
(g) Cart Pole U
(h) Cart Pole
Ecterne Fores
Estems Force
Bodus hum
Redua of Pech
8.8-
800-
899
-225-
MMMC
PNLC
Mil
NAL
(i) Car Tracking U
(i) Car Tracking U
(k) Car Tracking U
(I) Car Tracking U
post angale (vall
bon the Center Masset fentzume
Pos Canter of Panddaa
8.00-
R15-
-.00-
8.8-
AAACE
pooda
Teald
(m) 2-link Pendulum U (n) 2-link Pendulum U
(o) 2-link Pendulum U
(p) 2-link Pendulum U
Figure A-16: Control curves of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum
under uniform (U) perturbations in testing.
27

---

Under review as a conference paper at ICLR 2023
Eeternal Firce
Acrilett on Geidity
Length of Pale
Vaace hal
pestaum arg an COSEENNTTO
SonJann arg 2000 on
FSLL BAAA
Lmadal
(a) Pendulum W
(h) Pendulum W
(c) Pendulum W
(d) Pendulum W
paitin (931
(191)
Frle
830-
14C
03003
800-
1999
1800-
8.00
Omeoo
aão
01900
56 03
75
166 25
Umeet
(c) Cart Pole W
(f) Cart Pole W
(g) Cart Pole W
(h) Cart Pole
8.00
channe aren tnt
A15-
NAL
8.00
Umitall
Limdal
(i) Car Tracking W
(j) Car Tracking W
(k) Car Tracking W
(1) Car Tracking W
Pesition the Center of Maas Mm-5
Poation the Centor Masa of feredam
Drsumnll Farce
Iecernl Fores
poudton
0.5%
8mm-
2100
01
035
man
10
-0.0
(m) 2-link Pendulum W (n) 2-link Pendulum W
(o) 2-link Pendulum W
(p) 2-link Pendulum W
Figure A-17: Control curves of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum under learned adversary's 's worst-case (W) perturbations in testing.
28

---

Under review as a conference paper at ICLR 2023
Edwnal Tarms
Eretion Couftieat
Fges of Grawily
Lenghh of Poln
ABMCC
AMAC
-2
Brasard
PAPE
-27
100
22
10
22
PercerMe
(a) Pendulum U
(b) Pendulum U
(c) Pendulum U
(d) Pendulum U
acceentioon Graatt
2100
2009
2100
MCC.
2015
140
140
RABL
2003
205
2309
2
-121
-280
Focertie
Percom kg
Apcor110
Pendulum
Cart Pole
(g) Cart Pole U
Cart Pole
M05
1099
1525
1088
330
Parcani
Parcnc
(i) Cart Pole
(i Cart Pole U
(k) Car Tracking U
(I) Car Tracking
of km
Externll Force
1od2 of the Canter tass Postuam
Lmooth Aermmmmm)
-123
1200
-003
-X.0
A/BC
-532
PMC
-542
-1AC4
(m) Car Tracking U
(n) 2-link Pendulum U
(o) 2-link Pendulum U
(p) 2-link Pendulum U
Figure A-18: Percentile plots of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum
under uniform (U) perturbations in testing.
29

---

Under review as a conference paper at ICLR 2023
Edwnal Tarns
Fretion Couffiiatt
Fges of Gravity
Lenigh of Poln
ABMC
AMAC
-16
-22
-20
-20
100
22
10
Parcerue
(a) Pendulum W
(h) Pendulum W
(c) Pendulum W
(d) Pendulum W
Bo
Acceention Graatt
2100
2100
2100
MCC.
2015
140
140
RABL
2003
2050
2309
1003
Focortee
Percor kg
Apcor110
(c) Pendulum W
(f) Cart Pole W
gg) Cart Polc W
(h) Cart Pole
-200
-200
1099
-X0e
1525
1088
-003
330
19
Parcanc
Parcac
(i) Cart Pole
(j) Cart Pole W
(k) Car Tracking W
(I) Car Tracking W
of km
Podi the Conner Mass Protuam
Learth hormmmmi
Exteraal Force
400
-403
-423
-000
-100s
A/RCC
-340
RLC
-1545
(m) Car Tracking W
(n) 2-link Pendulum W
(o) 2-link Pendulum W
(p) 2-link Pendulum W
Figure A-19: Percentile plots of Pendulum, Cart Pole, Car Trajectory Tracking and 2-link Pendulum under learned adversary's 's worst-case (W) perturbations in testing.
30

---

Under review as a conference paper at ICLR 2023
A.5.2 GENERALIZATION IN PERTURBATION SPACE
In this subsection, evaluate the generalization capability of tearning-basel algorithms in the en- tire perturbation space. We observe that ARNLC achieves the best generalization performance under most unknown system dynamics and among all the combinations of different physical parameters. RARL presents the worst performance in all the tasks except for Pendulum.
SRNIC
M6C
Ertton COSTEANE
(a) Pendulum
(b) Pendulum
ARNIC
RARI
ARNIC
MC
PILC
R/AL
02000000000 30COL02 gaaty
liggh 20l
(c) Pendulum
(d) Cart Pole
ARNIC
NLC
PNLC
ABbUC
PNLC
RAXL
060060000 ml bel
CANAARPP mos tel
(e) Pendulum
(f) Pendulum
ASNLC
PILC
MAL
ANNLC
MLC
PILC
MAL
554 scceediinof
gadty
(g) Cart Pole
(h) Cart Pole
ASNLC
BARL
ATNLC
ALC
PILC
RALL
lighh
(i) Cart Pole
Cart Pole
10000
62
13000
8.3-
035-
025
8.5
E3
ar
ta
ad4
334
245
065
9
(k) Inverted Pendulum
(I) Inverted Pendulum
Figure A-20: Heatmap of averaged controller's reward for Pendulum, Cart Pole and Inverted Pen- dulum.
31

---

Under review as a conference paper at ICLR 2023
A.5.3 IMPACT OF CONTROL INTERVALS
We evaluate the impact of different control intervals (0.01s, 0.1s, 0.005s and 0.001s) our ARNLC on Pendulum with perturbation types (Length of Pole. Mass of Ball, Friction Coefficient, Acceler- ation of Gravity) different from the results presented in the main text. The resulting control curves obtained for Pendulum under uniform perturbations are shown in Figs. [A-1 A- and Fig.
A-21 And the control curves under perturbation of learned adversary are shown Figs. (-17 A
17d and Fig. A-22 The resulting percentile plots obtained for Pendulum under uniform perturba- tions are shown in Figs.
A- and Fig. A-23 And the percentile plots under perturbation
of learned adversary are shown in Figs. A- 19(a A- 19te) and Fig. A The results demonstrate
that our ARNLC is robust to different control intervals.
1a woweea Sone SO Lan (140
AIBCC PNC
AMEL
Lindst
Uneda
Unndal
brata
(a) 0.1s U
(b) 0.1s
(c) 0.1s U
(d) 0.1s
Geadty
of
(AMC PNL
0.50
A/RC HAL
PMC
(c) 0.005s U
(f) 0.005s U
g 0.005s U
(h) 0.005s U
Trkten Coeffeent
1O0e PAAA oodinn 201 2011
Acceterttion Cracity
Piaco Bell
MOKE
APRC PNL
(i) 0.001s U
(j) 0.001s U
(k) 0.001s
()) 0.001s U
Figure A-21: Control curves of Pendulum under uniform (U) perturbations in testing with control interval set 0. 0.005s and 0,001s, respectively.
32

---

Under review as a conference paper at ICLR 2023
Proder Cosfiss art.
Acodie eiter of Cructly
1800 Lepeon aa 1E
Lempth
HBC
PARC
(a) O. W
(b) 0.1s W
O.1S. W
(d) 0.1s W
Fretion Contidintt
Acceloracin Graviyy
Masa Bell
D0004mm w00 20r 200 000
P. 2 3
pantdam anpoara
KMAL Mac
pastsdmm
800-
FMEL
(e) 0.005s W
(l) 0.005s W
(g) 0.005s W
(h) 0.005s W
Priction Confriinnt
post Am UPOP or pr
Acowlerattnn Gractty
Vant ofll
Length Pf
-45
APRCC PAEC
PNC
RJC.
C
Laniaal
Landaal
(i) 0.001s W
(i 0.001s W
(k) 0.001s W
(I) 0.001s W
Figure A-22: Control curves of Pendulum under learned adversary's 's worst-case (W) perturbations in testing with control interval set to 0.1s, 0.005s and 0.001s, respectively.
Fictann SteTike
Accetration Cripety
ABSIC
MILE
MEC
Parcart
300
F1
390
Percendl
0 30
10
20
Perzantile
300
(a) 0.1s U
(b) 0.1s
(c) 0.
(d) 0.1s
Fradion Coefiiient
Accentation Seadty
Papa Bl
wengh Pala
ABMCC
FML.
FALL
140
100
100
23
40
(e) 0.005s U
(f) 0,005s U
(g) 0.005s U
(h) 0.005s U
Frlctton Conficeet
Acco Gracity
Paas cfail
Length
-50.0
-200%
-10000
ABMIC
FNLC
M.C
N.C
M.C
KA01
-R0K
RALL
Parcortte
Percert
Percend
Percerr
(i) 0.001s U
(i) 0.0011s U
(k) 0.001s U
()) 0.001s U
Figure A-23: Percentile plots of Pendulum under uniform (U) perturbations in testing with control interval set to 0. 0.005s and 0.001s, respectively.
33

---

Under review as a conference paper at ICLR 2023
Frizznn
Acadiration Graviyy
Lengh
CBSIC
MLL
MuL
-56
-236
-22
PRL
PALC
351
Percent
Percentie
(a) 0.Is W
(b) 0.1s
(c) 0.Is W
(d) 0.1s
Trietion Coufffient
Accs srntiintion Sravey
Maee of
Lenghh of Pole
AMMC
AMMC
FRLE
FMC
Rewand
20
30
10
10
25
(e) 0.005s W
(1) 0.005s W
0.005s W
(h) 0.005s W
Fridion Conficent
Accoliaaion Credty
Pf
Lengh
40MC PMC
ARMRC
FMC
RMC
X1X
M.C
(i) 0.001s W
(j) 0.001s W
(k) 0.001s W
(I) 0.001s W
Figure A-24: Percentile plots of Pendulum under learned adversary's 's worst-case (W) perturbations testing with control interval set 0.1s, 0.005s and 0.001s, respectively.
34


Reviews:
[{"review_id": "oTFGINAP3H", "reviewer": "Reviewer_6NKw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose an adversarially robust neural Lyapunov control (ARNLC) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. They claimed that the main contributions are:\n\n[1] propose a perturbed Lyapunov risk for learning the control policy under perturbations.\n\n[2] formulate an optimization problem for adversarially robust controller learning, to learn a policy in face of the worst-case perturbations that are imposed by the RL-trained adversary.\n\n[3] propose an adversarially robust neural Lyapunov control (ARNLC) approach to approximately solve this problem, and demonstrate its performance on several stability control tasks.", "summary_of_the_review": "This paper studied an interesting problem and proposed a novel ARNLC method for robust stabilization control problem of general nonlinear dynamic systems, and it contains some novel valuable results. The advantages of the proposed method are demonstrated via multiple empirical evaluation case studies. The main weakness is lacking of theoretical analysis of the proposed ARNLC method, and I'm not convinced by the author's claim about \"the learned control policy enjoys a theoretical guarantee of stability\".\n\nI think the contribution of this paper from theoretical perspective is weak, but the proposed method and empirical evaluations are still valuable. The paper contribution would be much more significant if the authors are able to add rigorous theoretical analysis to prove that \"the learned control policy indeed achieves theoretical guarantee of stability for the closed-loop system using their proposed ARNLC method\". Without adding that theoretical part, I think this paper's contribution is board-line, and slightly below the acceptance threshold.\n\n", "strength_and_weaknesses": "Strength:\n\n[1] The LNC robustness stabilization problem studied here is interesting and important, from both theory and application perspective.\n[2] Overall speaking, the paper presentation is clear and I found it straightforward to follow.\n[3] The empirical evaluation results based on several case studies are helpful to demonstrate the advantages of the proposed method, in comparison with other available baseline methods in the literature.\n\nWeakness:\n[1] In the paper abstract, it is claimed by authors that, \"the learned control policy enjoys a theoretical guarantee of stability\". However, based on my reading of the paper, I didn't see any rigorous theorem & proofs about such \"theoretical guarantee of stability\", for the proposed ARNLC algorithm. What's presented in the paper is some \"empirical evaluation based on cases studies\" plus some non-rigorous comment/explanations of the proposed algorithms. So the \"theoretical guarantee of stability\" seems to be a over-claim here and it is inappropriate. Note that, the Theorem 1 about using stability guarantee with Lyapunov function was only for using the exact control policy and Lyapunov function without approximation, and for the proposed ARNLC method with multiple approximations (and could potentially violate Eq2 conditions many times), there is not any rigorous \"theoretical guarantee of stability\" any more. In my opinion, lacking of any rigorous theoretical performance analysis of the proposed ARNLC method (which has multiple approximations) is a main weakness of this paper, and the paper would be significantly improved if the authors are able to add/derive any rigorous stabilization performance analysis of the proposed ARNLC method. And at the very least, the authors should remove their claim about  \"the learned control policy enjoys a theoretical guarantee of stability\" for their proposed ARNLC method.\n\n[2] The system considered in this paper is general continuous-time nonlinear dynamic systems, without other assumptions/conditions (e.g., lipschitz condition on f) on the system dynamics. Is it really the case that the proposed ARNLC method could work for such \"general nonlinear systems\" with \"theoretical guarantees about the stability\", without any other assumptions/conditions on system dynamics? It is well known in nonlinear dynamic system control theory that, for nonlinear dynamic systems, it could even possibly has a nonlinear system finite escape time. For the proposed ARNLC method with multiple approximations,  I'm not convinced that the closed-loop system using the proposed ARNLC method could achieve  \"theoretical guarantee of asymptotical stability\" for such general nonlinear unknown dynamic systems (without other assumptions/conditions). Rigorous mathematical analysis about the closed loop system performance for the proposed ARNLC method would be required if the authors want to justify their claim about theoretical stability guarantee of their proposed method. Without that rigorous mathematical analysis, the results in this paper is still valuable/helpful, but its main contributions is from empirical application side, not theoretical side.\n\n[3] In page 4, it was mentioned that Equation (7) is an empirical unbiased estimator of Equation (6). But I think Equation (7) is an empirical unbiased estimator of Equation (5) rather than Equation (6)? Also, for the notation N in equation (7), it might be good to add some clarification for it, as well as what value of N was used for the approximation and also justify it.\n\n[4] In the paper, it was mentioned that \"if we can train a controller under the worst-case perturbation (which degrades the performance of its policy to the most) in a certain range, the controller then obtains a conservative policy that is robust to any perturbation within that same range\". In my opinion, such a statement is inaccurate and also not sufficiently convincing, and it might possibly cause misunderstanding about the proposed method's performance against generic perturbations within that same range. In particular, considering all the perturbations within that same range, the controller was trained/optimized against the so-called \"worst-case\" perturbation and thus the final obtained controller might possibly perform better against that specific perturbation (since it has be somewhat trained/optimized to adapt to that type of perturbations),  and there is not any rigorous mathematical analysis to support the statement that the obtained controller would perform well for all other perturbations (for which the obtained controlled were not trained/optimized against) within that same range. It might be helpful for the authors to explain/elaborate this in a more accurate/convincing way to avoid potential misunderstandings about the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ccx9AQ5uNl", "reviewer": "Reviewer_tGgb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers jointly learning a stable controller and neural network represented by the Lyapunov function, which is \"empirically\" robust with respect to the dynamics difference between the training and test environment. Experiments are conducted on Pendulum, Cart Pole, car tracking, and 2-link pendulum. ", "summary_of_the_review": "This paper considers jointly learning a stable controller and neural network represented by the Lyapunov function, which is \"empirically\" robust with respect to the dynamics difference between the training and test environment. Experiments are conducted on Pendulum, Cart Pole, car tracking, and 2-link pendulum. Main concern lies in the significance of the proposed approach both intellectually and empirically. ", "strength_and_weaknesses": "Strength:\n* The writing is clear and well-organized. \n* Detailed Description of the experiment setup and results\n\nWeaknesses:\n* The intellectual contribution of this proposed work in comparison to existing neural Lyapunov control seems incremental. The basic framework is based on the Neural Lyapunov Control paper; with the same Lyapunov risk function, the same falsifier, and the same training process, with the addition of state transition dynamics uncertainty.\n\n* In addition, the performance gain of the algorithm compared to Neural Lyapunov Control seems not obvious. For instance, in Fig A-15, the region of attraction of the proposed approach (ARNLC) looks similar to the Neural Lyapunov control (NLC) method. As shown in the Neural Lyapunov control paper, a better Lyapunov function usually leads to a large ROA - so is the learned Lyapunov better than the NLC without considering the disturbances in dynamics?\n\n* On the same vine, not enough visualization about the learned Lyapunov function is provided (like Fig 2 in the Neural Lyapunov Control paper). I am interested in how the perturbation influences the Lyapunov functions. With the perturbation, some states x which satisfy the Lyapunov condition can have negative V(x) or positive V(x)’ when the perturbation is large, it might demonstrate some interesting properties. In particular, a comparison of the learned Lyapunov function shape for ARNLC in comparison to the learned Lyapunov function without adversarial training (NLC) would be interesting, at least for the pendulum example - and comment on the difference.\n\n* Adversarial training usually requires large samples and works for small perturbations. However, in the proposed method, the authors consider the disturbances in the system model. For example, in the Pendulum example, a change of the ball mass, pole length, friction coefficient, and gravity, etc. But then the model disturbances are converted to a state perturbation to be considered in the training, i.e., xk'=M(x_k, a_k)+a_k^v. How do you convert the model discrepancy to a value of a_k^v? What range for the disturbance $a_k^v$ is reasonable to be considered? Also, in many nonlinear control problems, a small change in the system parameters can lead to a significant difference in the state transition, which makes a_k^v sufficiently large. \n\n* Can the authors better describe the difference between PNLC and ARNLC? There seems no proper description of PNLC.\n\n* For figure 1, for example, in (e) and (f), in (e), the curve is increasing, and in (f) the curve is decreasing, it is hard to compare. And some plots do not share the same y limits, which makes the results hard to follow.\n\n* For the training process in the pseudo-code, xk'=M(x_k, a_k)+a_k^v, where M is the learned model, despite that this learned model will introduce error. In the training process, at least for the forward pass, why not use the real simulator to generate data (e.g., like what the authors did in \"Sablas: Learning Safe Control for Black-Box Dynamical Systems\"? Using the learned model may bring unnecessary errors.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HSC0wjK9cQl", "reviewer": "Reviewer_g1he", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies training a neural network based controller under potential\nadversaries that maliciously alter the states of the dynamic system. The\ntechnical approaches the paper uses consist of three parts: (1) train an\napproximate environment model for the system; (2) use typical reinforcement\nlearning algorithm like PPO to train an adversarial on the approximated system\nmodel to find the worst case perturbation; and (3) train a neural network\ncontroller under this adversary using Lyapunov loss. The paper claims to learn\ncontrol policy with theoretical guarantee of stability using this approach.\nExperimentally, the paper evaluates this approaches on a few classic control\nenvironments such as pendulum and 2-link pendulum.", "summary_of_the_review": "I feel this paper definitely studies an intriguing problem and I got excited\nwhen seeing its title and abstract. Unfortunately I feel some key claims in the\npaper are not well supported and the paper is not able to deliver its promise.\nSo I tend to reject the current version of this paper, but I am happy to\ndiscuss with the authors further for clarifications. I am glad to re-evaluate\nthe paper based on responses from the authors.\n", "strength_and_weaknesses": "Strengths:\n\n1. The paper studies an important problem of learning stable and robust neural\nnetwork controllers. The approach of combining a RL agent as the adversarial\nwith neural Lyapunov learning is interesting. The use of neural Lyapunov under\nthe context of adversarial robustness is also novel.\n\n2. Empirical results are comprehensive and the visualization of results look\nnice. The proposed approach achieves empirical stability in a few simulation\nruns and it converges better under the attack of the learned adversary.\n\nQuestions and weaknesses:\n\nOne major benefit of using Lyapunov theory in neural controllers is that it can\nformally guarantee stability. However, the paper does not convincingly show\nsuch a formal guarantee.  The paper emphasized a few times that the learned\npolicy enjoys a theoretical guarantee of stability (e.g., in the abstract\n\"learned control policy enjoys a theoretical guarantee of stability\"), however\nmost evaluations were done only empirically, and there is no real stability\nguarantee. The benefits of neural Lyapunov control is that it can provide\nstability guarantee, however training with an empirical Lyapunov loss function\nwith an RL learned non-optimal adversaray based on approximated system dynamics\nseem far away from giving formal guarantees. The algorithm presented in\nAlgorithm 1 has not convergence guarantees, and the paper is also unclear about\nthe details of the falsifier (which might be able to give some formal\nguarantees) used during training.\n\nThe biggest question is, given a fixed system dynamic function and a fixed\ncontroller function, can you *formally* show that the controller is stable\nunder *any bounded adversary* inside a certain region of convergence using Lyapunov theory?  The\nresults presented in this paper do not support this - results show a few\nempirical simulation of the systems only.\n\nIf the controllers do have guaranteed stability, then these critical results\nmust be presented:\n\n1. How large is the region of the attraction?  \n2. How large is the allowed perturbation for the adversary for the guarantee to hold?\nTechnically, the adversary needs to be the optimal adversary to give a meaningful\ntheoretical guarantee. If the guarantee is only valid for a trained adversary it is much\nweaker since we don't know how good the adversary is.\n3. What is the cost of computing such a guarantee (e.g., time to run the falsifier)?\n\nIf these guarantees cannot be obtained, then this paper needs a major revision to\nremove claims on stability guarantees, such as those in abstract. And it will\nbecome less exciting since the use of Lyapunov theory does not provide any\nformal guarantees. In that case, it is unclear if the use of Lyapunov theory is\nnecessary, and a strong baseline would be using normal RL training to replace\nthe Lyapunov loss to guarantee empirical stability. The benefits of using\nLyapunov is unclear here. In addition, since the proposed approach is evaluated\nunder the learned adversary, there is a chance that the controller overfits to this\nadversary so the evaluation is not fair - this must be addressed in experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1qPFjaCrEf", "reviewer": "Reviewer_1FTL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an extension to Neural Lyapunov Control (NLC) to minimize model mismatch: Adversarially Robust Neural Lyapunov Control(ARNLC). This approach treats model mismatch as an additional control input to the system that an adversary may use to reduce the performance of the system. ARNLC then takes a multi-step training procedure. First the authors train on the perturbed Lyapunov Risk function: an extension to the NLC's Lyapunov risk with the new adversary as an input. Then, they train the adversary using PPO in an RL formulation with reward designed to go against the goal of the nominal controller. Finally the authors demonstrate the performance of their approach in synthetic control tasks namely the Pendulum, Cartpole, Car Trajectory Tracking and 2-link pendulum. ", "summary_of_the_review": "Overall this paper tackles the interesting and topical subject of model mismatch and sim-to-real in a fresh way. The use of a trained adversary to in an actor-critic fashion to fit a lyapunov function is certainly novel and has future potential. Unfortunately, this particular instantiation of the idea is lacking a few critical and minor points. Although the experimental section is written clearly, it does not state explicitly that the adversarial policy used for evaluation is new see point [8] above which weakens significantly the empirical claims of the paper. Similarly as stated in point[1] the theoretical guarantees promised do not have a theorem or similar statement in the paper. Finally, the paper does not mention the extensive literature in Robust control on achieving robust stability with Lyapunov-like conditions see [2,5]. Because of these problems I have to recommend the paper for rejection.", "strength_and_weaknesses": "# Strengths \n1. Thorough description of experimental setup and methodology.\n2. Experiments detailed sufficiently to be reproducible even without code.\n\n# Weaknesses\n1. ***Lack of Stability Guarantees***:\nIn the abstract the authors claim: \"the learned control policy enjoys a theoretical guarantee of stability.\"\nAlthough this may be true for NLCs that achieve zero Lyapunov risk loss, ARNLC significantly changes the setting by the inclusion of adversarial disturbances where it is not clear how the guarantees go through. To make this claim the authors should include a theorem with the specific assumptions on the adversary or modeling error.\n4. ***Lack of Reference to robust control theory***:\nThe topic of robustness with Lyapunov theory has been extensively studied not just in controls but also in the area of controls applied to learning. At least some comparison should be done with standard robust control approaches for this to be accepted. Using black-box environments should not be an obstacle given that the authors are learning an approximation of the dynamics in $\\mathcal{M}_\\eta$. It may be that control methods achieve similar performance without having to learn an adversary or policy. Possible papers to cite include but are not limited to (I highlight these references because they discuss robustness with provable guarantees in the learning context because stability guarantees are mentioned in the abstract):\n    1. Robustness in the context of Lyapunov theory precisely when trying to use learning to overcome model mismatch: Taylor, A. J., Dorobantu, V. D., Krishnamoorthy, M., Le, H. M., Yue, Y., & Ames, A. D. (2019, December). A control lyapunov perspective on episodic learning via projection to state stability.\n   2. Robust  MPC with provable guarantees: Aswani, A., Gonzalez, H., Sastry, S. S., & Tomlin, C. (2013). Provably safe and robust learning-based model predictive control. Automatica, 49(5), 1216-1226.\n1. ***Unclear theoretical setting***:\nThe authors should clearly specify the assumptions on the dynamics function $f$ in particular they should clearly state the assumptions required for uniqueness and existence of solutions on which the Lyapunov stability theorem relies. \n1. ***On the generality of control-affine systems*** In the second paragraph the author states that : \" However, this approach only considers  the control-affine dynamical systems, not the more general nonlinear one\"\nA general nonlinear system can always be placed into control affine form by choice of an integral controller. Furthermore the test systems are mechanical systems which can be placed into control-affine form. It would be interesting to see a case where the generality of ARNLC is required.\n1. ***On Exponential Stability and Robustness***: It can be shown that exponentially stable systems converge to a small area around the equilibrium point rather than the equilibrium point itself under perturbation. This property is commonly referred to as Input-to-State Stability(ISS). Depending on the magnitude of the perturbation from the nominal dynamics, an exponentially stable system may converge close enough to the equilibrium to achieve the results shown. Some experiments showing that the resulting stability cannot be achieved by a sufficiently exponentially stable controller would strengthen the claims of the paper. I would consider citing the following: Sontag, Eduardo D. \"Input to state stability: Basic concepts and results.\" Nonlinear and optimal control theory. Springer, Berlin, Heidelberg, 2008. 163-220 . or Liberzon, D., Sontag, E. D., & Wang, Y. (2002). Universal construction of feedback laws achieving ISS and integral-ISS disturbance attenuation. Systems & Control Letters, 46(2), 111-127.\n1. ***On the generality of additive noise*** Suppose you have a nominal dynamics $f(x,\\hat{w})$ and a true dynamics $f(x,w)$. I can write the true dynamics as the nominal dynamics with additive noise as follows: $\\dot{x} = f(x,\\hat{w}) + \\epsilon$ where the additive noise $\\epsilon = f(x,w) - f(x,\\hat{w})$. The authors mention that some prior work is only restricted to additive disturbances as follows: \"However, it is usually restricted to the additive disturbances (Löfberg, 2003) \". The paper would be significantly strengthened by showing a case where an additive disturbance model is not sufficient either theoretically or empirically.\n1. ***Unclear performance measurement***:\nThe paper shows curves of controllers successfully stabilizing the desired input. Although this clearly shows that the controllers can stabilize the system, they may still be wildly inefficient at achieving this goal. You may be able to achieve very high robustness while using very high inputs. To evaluate the performance of the controller, the authors should consider showing the norm of the control input as the pendulum converges. Similar robustness results could be achieved by scaling the control input to increase the scale of the system relative to the noise. \n1. ***Unclear Statements about worst-case perturbation***:\n> The intuition behind is that if we can train a controller under the worst-case perturbation (which degrades the performance of its policy to the most) in a certain range, the controller then obtains a conservative policy that is robust to any perturbation within that same range.\n\nIn general, its not safe to assume that robustness to the worst case perturbation results in robustness to any perturbation. Even in a linear case, making the system robust to a single worst-case perturbation does not stop other perturbations from causing instability. There may be a version of this statement that is true but, again, the authors would have to state the assumptions clearly in a theorem and provide proof. \n\nFor the empirical evaluation I am not sure if the ARNLC results are evaluated on a new adversary or the one that was used to train the policy and a lyapunov function. Not retraining the adversary while keeping the weights of the policy constant would seem akin to reporting training accuracies since the policy has been able to adapt to that particular instance of the adversary and may not generalize. An ablation study of a newly trained adversary to which the policy cannot adapt and a uniform random adversary would show empirical evidence of the worst-case perturbation assumption stated previously.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject