Model-free Learning Control of Nonlinear Stochastic Systems with Stability Guarantee
Authors: Minghao Han
Yuan Tian
Lixian Zhang
Jun Wang
Wei Pan
Venue:  (2020)
Keywords: Reinforcement learning
nonlinear stochastic system
Lyapunov

Abstract: Reinforcement learning (RL) offers a principled way to achieve the optimal cumulative performance index in discrete-time nonlinear stochastic systems, which are modeled as Markov decision processes. Its integration with deep learning techniques has promoted the field of deep RL with an impressive performance in complicated continuous control tasks. However, from a control-theoretic perspective, the first and most important property of a system to be guaranteed is stability. Unfortunately, stability is rarely assured in RL and remains an open question. In this paper, we propose a stability guaranteed RL framework which simultaneously learns a Lyapunov function along with the controller or policy, both of which are parameterized by deep neural networks, by borrowing the concept of Lyapunov function from control theory. Our framework can not only offer comparable or superior control performance over state-of-the-art RL algorithms, but also construct a Lyapunov function to validate the closed-loop stability. In the simulated experiments, our approach is evaluated on several well-known examples including classic CartPole balancing, 3-dimensional robot control and control of synthetic biology gene regulatory networks. Compared with RL algorithms without stability guarantee, our approach can enable the system to recover to the operating point when interfered by uncertainties such as unseen disturbances and system parametric variations to a certain extent. 

Under review as a conference paper at ICLR 2020
MODEL-FREE CONTROL OF NONLINEAR STOCHASTIC SYSTEMS WITH STABILITY GUARANTEE
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) offers principled way to achieve the optimal cu- mulative performance index discrete-ting time nonlinear stochastic systems, which are modeled as Markov decision processes. Its integration with deep learning techniques has promoted the field of deep RI. with an impressive performance complicated continuous control tasks. However, from control-theoretic perspec- tive, the first and most important property Unfortunately, stability is rarely assured in RL and remains open question. In this paper, we propose a stability guaranteed RL framework which simultaneously learns Lyapunov function along with the controller policy, both of which are parameterized by deep neural networks, by borrowing the concept of Lyapunov function from control theory. Our framework can not only offer comparable superior control performance over the state-of-the ofrthc-arttttttt RL algorithms, but also construct a Lyapunov function to validate the closed-loop stability. In the simulated experiments, our approach is evaluated on several well-known examples including classic CartPole balancing, 3-dimensional robot control and control of synthetic biology gene regulatory networks. Compared with RL algorithms without stability guarantee, our approach can enable the system to recover to the operating point when interfered by uncertainties such as unseen disturbances and system paramet- ric variations to a certain extent. (Anonymous code is available to reproduce the experimental results"]
system to be guaranteed stability.
INTRODUCTION
Control of discrete-time nonlinear stochastic systems is an important topic in both control theory and reinforcement learning. In the past decades, the advancement of nonlinear control theory in the control community has been successfully applied in aircraft, automobiles, advanced robots and space systems (Slotine et al. 1991 [sidoii 1995). Concurrently, reinforcement learning was developed in the machine learning community to address similar nonlinear control problems (Sutton al. 1992] Tesauro 1995 [eertsekas & Tsitsiklis 1996). Until recently, significant progress has been made by combining advances in deep learning (LeCun et al. 2015) with reinforcement learning. Impressive results are obtained in a series of high-dimensional continuous nonlinear control problems (Duan approach
al., 2016 Zhang et al. 2016 Zhu et al. 2017 Gu et al. 2017) in which
typically difficult to apply.
Given a control system, regardless of which controller design method is used, control theory or reinforcement learning, the first and most important property of system needs to be guaranteed is stability, because an unstable control system is typically useless and potentially dangerous (Slotine et al. 199T). Qualitatively, a system is described as stable if starting the system in the neighborhood of its desired operating point implics that it will stay around the point ever after. For aircraft control systems, typical stability problem is intuitively related to the following question: will trajectory perturbation caused by gust result in a significant deviation in the later flight trajectory? Here, the desired operating point of the system is the flight trajectory in the absence of disturbance. Every control system, whether linear or nonlinear, involves a stability problem which should be carefully studied.
https:: http:///www..ooooooooo/////////a

---

Under review as a conference paper at ICLR 2020
The most useful and general approach for studying the stability of control systems is Lyapunov method (Lyapunov] 1892), which is dominant in control engineering (Aström & Wittenmark, 1989; Mayne et al. 20000. In Lyapunov method, a scalar "energy-like" function called Lyapunov function is constructed to analyze the stability of the system. For linear dynamical system, a quadratic function is typically chosen as Lyapunov function in some classic controller design method, such as linear quadratic regulator (LQR) and model predictive control (MPC). Unfortunately, there is no universal method for constructing Lyapunov functions.
In this paper, we propose a stability guaranteed reinforcement learning framework to jointly learn the controller or policyBandd and a Lyapunov function both of which are parameterized by deep neural networks, with focus on stabilization and tracking problems in discrete-tine nonlinear stochastic systems modeled by Markov decision process. The contribution of our paper can be summarized as follows: 1) a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing Lyapunov function parameterized by deep neural network; 2) a practical learning algorithm is designed to search the stability guaranteed controller; 3) the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent. In our experiment, controller is more capable of handling uncertainties compared to those without such guarantees in nonlinear control problems including classic CartPole stabilization tasks, control of 3D legged robots and manipulator and reference tracking tasks for synthetic biology gene regulatory networks.
show that the stability guaranteed
1.1 RELATED WORKS
In model-free reinforcement learning (RL), stability is rarely addressed due the formidable chal- lenge of analyzing and designing the closed-loop system dynamics in a model-free manner (Buşoniu et
2018). and the associated stability theory in model-free RL. remains an open problem (Buşoniu
ct al.
2018
Gorges 2017).
Recently, Lyapunov analysis is used in model-free RL solve control problems with safety constraints (Chow et al. 2018; 2019). In Chow et al. (2018). Lyapunov-based approach for solving constrained Markov decision process proposed with a novel way of constructing the Lyapunov function through linear programming. In Chow et al. (2019), the above results were further generalized to continuous control tasks. Even though Lyypunov- methods were adopted in these results, neither of them addressed the stability of the system.
As basic tool in control theory, the construction/leaming of Lyapunov function is not a trivial issue and many works are devoted to this problem. In Perkins & Barto (2002). the RL agent controls the switch between controllers designed using Lyapunov domain knowledge, so that any policy is safe and reliable. Petridis & Petridis (2066) proposes a straightforward approach for constructing Lyapunov function for nonlincar systems using neural networks. Richards et al. (2018) proposes lcarning-based approach for constructing Lyapunov neural networks with maximized region of attraction. Results on learning and construction of Lyapunov functions are referred to Noroozi et al. (2008) Prokhorov 1994] Serpen
2005): Prokhorov & Feldkamp (1999).
Other interesting results on the stability of learning-basse control systems are reported in recent years. In Postoyan al. (2017), initial result is proposed for the stability analysis of deterministic nonlin- ear systems with optimal controller for infinite-horizon discounted cost, based on the assumption that discount is sufficiently close 1. In [Berkenkamp et al. (0017). learning model-based safe RL approach with safety guarantee during exploration is introduced but limited to Lipschitz continuous nonlinear systems such as Gaussian process model. In addition, the verification of stability condition requires the discretization of state space, which limits its application to tasks with low-dimensional finite state space.
2 PROBLEM STATEMENT
We consider discrete-time nonlinear stochastic systems modeled by the Markov decision process (MDP). A MDP is defined by the tuple (S, AS,A,c, c, P.P), where S C T2" is the set of states, A C R is the set of actions, c(s, a): S x A + R4 is the cost function, P(s' (1881,,, a) is the transition probability function, and p(s) is the starting state distribution.
Controller and policy will be used interchangeably throughout the paper.
2

---

Under review as a conference paper at ICLR 2020
In this paper, we focus the stabilization and tracking problems for discrete-tingg -imee nonlinear stochastic systems modeled by MDP. For both problems, the goal is to find a policy я which can bring the cost to zero. In stabilization problems, the cost function is defined as the norm of states ||s|| where || denotes the Fuclidean norm. In tracking problems, we divide the state 8 into two vectors, st and 5², where s is composed of elements of that aimed at tracking the reference signal r while s² contains the rest. For tracking, I|s r is chosen to be the cost function.
From control theoretic perspective, the task of stabilization and tracking could be addressed as ensuring the closed-loop system or error system to be asymprotically stable, i.e., starting from an initial point, the trajectories of state always converge single point or the reference trajectory. Let Ca (s) (ss, a) denote the cost function under policy x. the definition of stability studied in this paper is given as follows.
The stochastic system is said to be stable in mean cost if lime-40o Es, Cn (s;) = 0 holds for any initial condition E (s0) S b). If b is arbitrarily large then the stochastic system is globally stable in mean cost.
Definition
Remark Form of the cost is strictly ruled as the Euclidean norm of state or partial state, while other forms are not considered in this paper. The stability studied in this paper is type of local asymptotic stochastic stability, which is different to the definition of mean square stability (MSS) that
extensively studied on stochastic systems in control theory (Shaikhei 7997: Huang 2012)
Before proceeding, some notations are to be defined. The closed-loop transition probability is denoted as P= (s' - IA ((s)s) a)da. We also introduce the closed-loop state distribution at certain instant P(s n, t). which could be defined in an iterative way: P(s'le n. + 1) Is (( t)ds, Vt E 2, and P(s|p, n, 0) = p(s).
3
MAIN RESULTS
In this section, we propose the main assumptions and new theorem.
Assumption The stationary distribution of state (s) A limy-soo P(slp, tt,) exists.
Assumption There exists a positive constant such that p(s) O, Vs E (s|c=(s) S... b).
Our approach is to construct/find Lyapunov function which can be used to amalyze the stability of the closed-loop system. The Lyapunov method has long been used for stability analysis and controller design in control theory (Boukas & Liu 2000), but mostly exploited along with known model, whether deterministic or probabilistic (Corless & Leitmann 1981 Thowsen 1983. Huang et al.
2011).
The Lyapumov function a class of semi-positive definite functions L S R4. The general idea of exploiting Lyapunov function is ensure that the difference (or derivative, if the system is in continuous time) of Lyapunov function along the state trajectory is semi-negative definite, so that the state goes in the direction of decreasing the value of Lyapunov function and eventually converges to the origin or sub-level set of Lyapunov function. Next, we give sufficient conditions for a system to be stable in mean cost in the following.
Theorem
The stochastic system is stable in mean cost if there exists a.function L: S > R4 and positive constants 01, Oz and O:3, such that
C1 (s) < L(s) S ax2cx (8) E Exnjuy (E L(s') L(s))((+=))))))))) -agBraue Cr (s)
1 where (s((s) limm +20 N EN P(s: = s|p, ग. f) is the sampling distribution.
Due to space limitations, we will include the detailed proof in Appendix A Eq. 2 is called the energy decreasing condition, i.e.. requiring the expectation of Lyapunov function to be decreasing between two consecutive instants. Eq. (I) is the constraint for Lyapunov function, though a rather broad range of parameterization is covered. The of quadratic polynomials, e.g., I(s) s Qss where Q is
3

---

Under review as a conference paper at ICLR 2020
positive definite matrix, are extensively used in the control theory. Such Lyapunov functions can be efficiently discovered by the semi-definite programming solvers and bring in limited conservatism for the control tasks where the cost also of quadratic form. In (Richards et al. 2018), neural
network (.) () is designed to construct the I Lyapunov function, [(s) (o(()* de (s). As explored
in (Chow et al., 2018) and (Berkenkamp al. 2017), the value function could be exploited
Lyapunov function as well. Additionally, the sum of cost over a limited time horizon could also be
++ (e,((s)) which is valid Lyapunov candidate in model predictive control literature (Mayne & Michalska 1990; Mayne et al. 2000).
employed as Lyapunov function, i.e., L(s) > et
The choice of Lyapunov function candidate plays an important role in learning policy. Value function evaluates the infinite time horizon and thus offers a better performance in general, but is rather difficult to approximate because of significant variance and bias (Schulman et al. 2015). On the other hand, the finite horizon sum of cost provides an explicit target for learning a Lyapunov function, thus inherently reduces the bias and enhances the learning process. However, the model is unknown, predicting the future costs based on the current state and action inevitably introduces variance, which grows the prediction horizon extends. In principle, for tasks with simple dynamics, the sum-of-cost choice enhances the convergence of learning and robustness of the trained policies, while for complicated systems the choice of value function generally produces better performance. In this paper, we use both value function and sum-of-cost over various horizons as Lyapunov function candidates in different tasks and compare their strength and weakness respectively. Now we would like to give the following two remarks.
This remark is on Assumption 7 and sampling distribution Pn. If an MDP is ergodic then the existence of ga is naturally assured, but all states have to be positive recurrent and aperiodic (Pa- poulis & Pillai 2002). The existence of sampling distribution (s) is guaranteed by the existence of 4a (8). Since the sequence (P(s)p, n. R,(),, L converges to qa (8) as I approaches oo, then by the Abelian theorem, the sequence { + Ei=o P(slp, n. .) N E ]+} also converges and (s( = qu (s). Thus we use Ux to approximate the On since the evaluation of 9= requires data be sampled after infinite instants the episode begins. Even if the sampling period N << 0C, one can still assure that On converges to a neighborhood of zero, which is related the initial state distribution and length of
Remark 2
N, i.e., Cx S
_22 --22 (s0)(s0)
-o14o Ei= Es, Cx (St).
03
Remark 3 This remark is on the connection to previous results concerning the stability of stochastic systems. should be noted that the stability conditions of Markov chains have been reported in (Shaikher 1997: Meyn & Tweedie (012). however, of which the validation requires the full knowledge of the model, i.e., the transition probability P(s'|s, a). On the contrary, our approach solely depends on data to analyze the stability of the closed-loop system, which further enables the model-free learning algorithms with stability guarantee. However, the validation of stability through sample-bane based approach theoretically requires tremendous, if not infinite, amouut of samples to thoroughly estimate the distributions, which is the drawback of our approach. We would demonstrate empirically that the algorithm built upon this theorem is reliable though only limited sample is collected.
4 ALGORITHM
In this section, we propose an off-policy RI. algorithm to learn stability guaranteed policies for discrete-time nonlinear stochastic system modeled by MDP. First, based on the maximum entropy actor-critic framework, we use the Lyapunov function as the critic in the policy gradient formulation. In this algorithm, a Lyapunov critic function Le is needed, which satisfies L(s) = Le a). The objective function (xx ))) is given as follows
J(T) = E, - [ ((log(To([ s)]{)) + H.)) + A(Ia(ss ((le(', - Lo(ss + a3c)] (3)
where the policy To is parameterized by a deep neural network fe, is an input vector consisted of Gaussian noise. It should be noted that the Lyapunov critic Lo a) will be parameterized by the square of a neural network to ensure the semi- positiis definiteness of Lyapunov function required in Eq. inspired by the structure explored in Richards et al. (2018). More specifically, (.( = o² (s, (),a) a), where ø(s, a) is a multi-layer fully connected neural network. D is the distribution of previously sampled states and actions, replay buffer. In the above objective,
4

---

Under review as a conference paper at ICLR 2020
and À are Lagrange multipliers which control the relative importance of policy entropy versus energy decreasing constraint derived from Eq. (2). Similar to Haarnoja et al. (2018) the entropy of policy is expected to remain above the target entropy H,. The parameters of policy network are updated through gradient descent, where the gradient of Eq. is approximated by
VoJ(n) 1e
VaB log( s) AV Vooooo,, (4)
We ((La) in the following equation the objective function to update the Lyapunov critic,
1
J(Lc) ED
(L)(s, Laarge (s.
(5)
where Lunget is the approximation target for Le. the sum of cost is chosen as Lyapunov function candidate, we have
Laager ss a) = 2x > Ct.
(6)
Here, the time horizon N is hyperparameter be tuned, of which the influence will be demonstrated in the experiment in Section 5.5 If the value function is chosen as Lyapunov function candidate,
T (8, a) c ( (()
where L' is the target network parameterized by 0 as typically used in the actor-critic meth- ods (Haarnoja al., 2018; Lillierap et al., 2015b). I. has the same structure with Lc. but the parameter is updated through exponentially moving average of weights of Le controlled by a hyper- parameter т. fact, the value function the discounted sum of cost over infinite time horizon. Later in Section 5 we will show the influence of choosing different Lyapunov function candidates.
The value of Lagrange multipliers À and B are adjusted by the gradient method maximizing the following two objectives respectively,
((3) = ED (og(ooo((()s)) + Hi]
(8)
J(A) = EDA(LT(s' folc, Le(s, a) + azsc]
(9)
should be noted that the value of and 3 are clipped to be positive. addition, to prevent from growing unlimitedly causing the algorithm to diverge, we set an upper bound for X. In our experiments, we found that 1 is suitable value without much further tuning. Pscudo code of the proposed algorithm is shown in Algorithm in Appendix B
Remark 4 The convergence of the algorithm is composed of the convergence of Lyapunov critic Le and policy To respectively. Empirically, the convergence can be judged by the error of Lyapunov function approximation and value of the Lagrange multiplier (close to zero at convergence). In practice, we found that the algorithm converges well in different experiments without much tuning.
EXPERIMENT
In this section, we illustrate four simulated examples to demonstrate the general applicability of the proposed method. First of all, the classic control problem of CartPole balancing from control and RL literature (Barto et al.) 1983) is illustrated. Then, we consider more complicated high-dimensional continuous control problem of 3D robots, c.g., HalfCheetah and FetchReach, using MuJoCo physics engine Todorov et al. 2012). Last, we extend our approach to control robots in nanoscale, i.e., molecular robots. Specifically, we consider the problem of reference tracking for a synthetic biology gene regulatory network known the Repressilator (Elowitz & Leibler, 2000).
The proposed method evaluated for the following aspects:
Convergence: does the proposed training algorithm converge with random parameter initial- ization and does the stability condition (2) hold for the learned policies;
Performance: can the goal of the task be achieved or the cumulative cost be minimized;
Robustness: how do the trained policies perform when faced with uncertainties unseen during training, such as parametric variation and external disturbances;
5

---

Under review as a conference paper at ICLR 2020
Generalization: the trained policies generalize follow reference signals that different from the one seen during training.
We compare our approach with soft actor-critic (SAC) HHarnoja et al. 2018). one of the state-
of-the-art off-policy actor-critic algorithms that outperform series off-policy and on-policy methods such as DDPG (Lilicrap et al. 2015a), PPO (Schulman et al. (0077) on the continuous control benchmarks. The variant of safe proximal policy optimization (SPPO) CCoo et al. 2019), Lyapunov-based method, is also included in the comparison. The original SPPO is developed to deal with constrained MDP, where safety constraints exist. our experiments, we modify Lyapunov constraints on the MDP tasks and see whether it can achieve the same stability guarantee as LAC. CartPole example, we also compare with linear quadratic regulator (LQR), a classical model-based optimal control method for stabilization.
apply the
The outline of this section is as follows. In Section 5.1 a brief introduction will be given on the background and problem description of each example. Then in Section 5.2 the convergence, and performance of the proposed method demonstrated and compared with SAC. In Section the ability of generalization and robustness of the trained policies are evaluated and analyzed. Finally, in Section 5.5, we show the influence of choosing different Lyapunov function candidates upon the performance and robustness of trained policies.
5.1 BACKGROUND AND PROBLEM DESCRIPTION
In this section, will give a brief introduction the examples considered in this paper. Detailed setup information of the first three examples can be found in Appendix
5.1.1 CARTPOLE
This is classical control problem. The controller is to stabilize the pole vertically at given position. The cost determined by the norm of the angular position of the pole and the horizontal position of the cart. The control input is the horizontal force F [-20, 20] applied in the cart. The agent is dead the angle between pole and vertical position exceeds a threshold, and the episode ends.
5.1.2 HALFCHEETAH
The goal is to control a 17-dimensional 2-legged robot simulated in the MuJoCo simulator. The control task belongs to the reference tracking problem, i.e., to enable the robot run at the speed of Im/s in the X-axis direction. The cost is determined by the Euclidean difference between current speed and target speed. The control input the torque implemented at each joint.
5.1.3 FETCHREACH
The agent is to control a simulated manipulator to track a randomly generated goal position with its end effector. The cost is determined by the Euclidean distance between end effector and goal. The control input is the torque implemented at each joint. The manipulator is also simulated in the MuloCo simulator.
5.1.4 REPRESSILATOR
The repressilator is a synthetic biology gene regulatory network with a ring structure pioneered in Elowitz & Leibler (2000). in which each gene represses the other gene cyclically. The dynamics of temporal gene expression exhibit periodic oscillatory behavior. The dynamics of repressilator can be quantitatively described by set of discrete- nonlinear difference equations consisting of six states, three mRNAs for transcription and three proteins for translation, based on biochemical kinetic laws. We also include complicated repressilator example with 4 genes to be controlled, which exhibits an unstable oscillation and even harder to control.
The objective is to force one protein concentrations to follow a priori defined reference trajectories using partially observed states. Detailed setup information of these examples are in Appendix D
5.1.5 MARKOVIAN JUMP SYSTEMS
In addition to the systems described above, we introduce two Markovian jump systems (MJS), named MJSI and MJS2, which contain both discrete switchings (or jumps) and continuous dynamics (Shi &

---

Under review as a conference paper at ICLR 2020
Li 2015), test beds for the proposed and baseline methods. The objective is to force the full state to zero. Both MISs contain unstable subsystems and the dynamics change abruptly and randomly according to the switching signal, and thus difficult to tackle for the model-free algorithms. Moreover, MJS2 contains an unstable subsystem that is not controllable, which makes it even harder stabilize. More details the examples could be found in Appendix F
5.2 PERFORMANCE
We parameterize the policy and Lyapunov critic using deep neural networks. For each example,the hyperparameters including time horizon N and DNN architectures selected construct Lyapunov functions and DNN training parameters can be found in Appendix [ For both algorithms, the hyperparameters are tuned to reach their best performance. In each task, both LAC and SAC are trained for 10 times with random initialization, average total cost and its variance during training are demonstrated in Figure I
In the first three examples (sce Figure a) (c)( SAC and LAC perform comparably in terms of the total cost at convergence and speed of convergence, while SPPO could converge in Cartpole and ReacheReach. In the Repressilator and MJS examples (see (see/Figure eed,e.)) SAC is not always able to find a policy that capable of completing control objective, resulting in the bad average performance. On the contrary, LAC performs stably regardless of the random initialization.
302
400
380
322
1
1.0
(a) CartPole
(b) HalfCheetah
(c) FetchReach
0.00
SAC
SAC
LAC
350000
LAC
4/4
100
MWW
2000
(d) Repressilator
(0) Markov Jump Sytem-1
(f) Markov Jump Sytem-2
Figure 1: Cumulative control performance comparison. The Y-axis indicates the total cost during one episode and the X-axis indicates the total time steps in thousand. The shadowed region shows the 1-SD confidence interval over 10 random seeds. Across all trials of training. LAC converges to stabilizing solution with comparable or superior performance compared with SAC and SPPO. The is deferred to Appendix
experiment on
A distinguishing feature of stability assured policy is that it can force and sustain the state tracking error to zero. This could be intuitively demonstrated by the state trajectories of closed-loop system. We evaluated this property of trained policies in the Repressilator, Complicated-Represtt two MJS examples. In our experiments, we found that the LAAC agents stabilize the systems well in all tasks. All the state trajectories converge to the reference signal or equilibrium eventually (see Figure 1 (a,c) and Figure 12 (a.c)). On the contrary, without stability guarantee, the state trajectories either diverge (see Figure 1Tb and Figure 12 d), or continuously oscillate around the reference trajectory or equilibrium (see Figure 1id and Figure b). Empirical results are deferred to Appendix due to space limit.
and
5.3 CONVERGENCE
As shown in Figure LAC converges stably in all experiments. Moreover, the convergence and validation of stability guarantee could also be checked by observing the value of Lagrange multipliers. When 2 is satisfied, À will continuously decrease until it becomes Thus by checking the value
7

---

Under review as a conference paper at ICLR 2020
(a) Cartpole
(b) HalfChectah
(c) FetchReach (d) Repressilator
(c) MJS1
(f) MJS2
Figure 2: Value of Lagrange multiplier during the training of LAC policies. The Y-axis indicates the value of Ả and the X-axis indicates the total time steps in thousand. The shadowed region shows the I-SD confidence interval over random seeds. The value of À gradually drops and becomes zero at convergence, which implies the satisfaction of stability condition.
and variation of X, the satisfaction of stability condition during training and at convergence could be validated. In Figure 2, the value of À during training is demonstrated. Across all training trials in the experiments, À converges to zero eventually, which implics that the stability guarantee is valid. A detailed discussion this is referred to Appendix G
5.4 EVALUATION ON ROBUSTNESS AND GENERALIZATION
is well-known that over-varamann peraparrmeteri policies are prone to become overfitted to specific training environment. The ability of generalization is the key to the successful implementation of the algorithm an uncertain real-workl environment. In this part, first evaluate the robustness of policies in the presence of system parametric uncertainties and process noise. Then, we test the robustness of controllers against external disturbances. Finally, we evaluate whether the policy generalizable by setting different reference signals. To make a fair comparison, converge in SAC and only evaluate the ones that perform well during training. During testing, we found that SPPO appears to be prone to variation in the environment, thus the evaluation results referred to Appendix
removed the policies that did not
5.4.1 ROBUSTNESS TO DYNAMIC UNCERTAINTY
In this part, during the inference, we vary the system parameters in the model/simulator evaluate the algorithm's robustness against dynamic uncertainty. the example CartPole, we vary the length of pole (. In the example of repressilator,
vary the promoter strength a, and dissociation rate K;, Due to stochastic nature in gene expression, we also introduce uniformly distributed noise ranging from Ō- 6] (we indicate the noise level by 8) to the dynamic of repressilator. The stabilization performance of CartPole and tracking performance of Repressilator by LAC and SAC in the varied environment demonstrated in Figure 3
K-S
ADCK
M
Aut
Kuck
oigginl
(a) LAC-CartPole
(b) SAC-CartPole
(c) LAC- AAC-Reeressilator
(d) SAC- AAC-Repressilator
Figure 3: State trajectories time under policies trained by LAC and SAC and tested in the
presence of parametric uncertainties and process noise, for CartPole and Repressilator. Solid line indicates the average trajectory and shadowed region for the 1-SD confidence interval. In (a) and (b), the pole length varied during the inference. In (c) and (d), three parameters are selected to reflect the uncertainties in gene expression. The X-axis indicates the time and Y-axis shows the angle of pole in (a.b) and concentration of protein to be controlled in (c.d), respectively. Dashed line indicates the reference signal. The line in orange indicates the dynamic in original environment. For each curve, only the noted parameter is different with the original setting. We also show the curves in separate zoom-in view Appendix [.]
As shown in Figure (a) and (c), the policies trained by LAC are very robust to parametric uncertainties of different values and achieve high tracking precision in each case. On the other hand, though SAC performs well in the original environment (Figure 3(bb and (d)). it fails to track the reference signal all of the varied environment.
8

---

Under review as a conference paper at ICLR 2020
5.4.2 ROBUSTNESS TO DISTURBANCES
An inherent property of a controller for stabilization is to enable the system to recover to the normal status from perturbations such as external forces and wind. To show this, we introduce persistent external disturbances with different magnitudes in each environment and observe the performance difference between policies trained by LAC and SAC. We also include LQR as the model-based baseline. In CartPole, the agent may fall over when interfered by an external force, ending the episode in advance. Thus in this task, we measure the robustness of controller through the death-rate, i.e., the probability of falling over after being disturbed. For other tasks where the episodes of the same length, we measure the robustness of controller by the variation in total cost. Under each disturbance magnitude, the policies are tested for 100 trials and the performance are shown in Figure 4
always
SAC
SAC
SP
DE
(a) CartPole
(b) HalfCheetah
(c) FetchReach
(d) Repressilator
Figure 4: Performance of policies trained by LAC, SAC and SPPO, along with controllers designed
by LQR in the presence of persistent disturbances with different magnitudes. X-axis indicates the magnitude of the applied disturbance. For CartPole (a) the Y-axis indicates the probability of falling over and in other three examples (b)-(d) indicates the total cost. Both policies are evaluated for 100 trials in each setting.
As shown in the Figure 4 the controller trained by LAC outperforms SAC and LQR by great extent when faced with external disturbances in CartPole and repressilator (lower death rate and total cost). the repressilator example, the policies trained by SAC are extremely vulnerable to disturbances, this is potentially due to the existence of an unstable limit cycle in the uncontrolled dynamic (Strelkowa & Barahona 2010). In HalfCheetah, SAC and LAC are both robust to small external disturbances while LAC is more reliable larger ones. In FetchReach, SAC and LAC comparable with maintaining a low cost in a great range of external disturbances. This is perhaps due to the manipulator's inherent robust mechanical design.
5.4.3 GENERALIZATION OVER DIFFERENT TRACKING REFERENCES
In this part, we introduce four different reference signals that are unseen during training in the repressilator example: sinusoids with periods of 150 (brown) and 400 (blue), and the constant reference of 8 (red) and 16 (green). We also show the original reference signal used for training (skyblue) a benchmark. Reference signals are indicated in Figure by the dashed line in respective colors. Both trained policies are evaluated to track each reference signal for 10 times, and the average dynamics of the target protein concentration are shown in Figure 5 with the solid line, while the variance of dynamic is indicated by the shadowed area.
As shown in Figure 5 the policies trained by LAC could generalize well follow previously unseen reference signals with low deviation (dynamics are very close to the dashed lines), regardless of whether they are in the same mathematical form with the one used for training or not. On the other hand, though SAC tracks the original reference signal well after the unconverged training trials being removed (see the skyblue lines), it is still unable to follow some of the reference signals (see the brown line) and possesses larger variance than LAC when following others.
5.5 INFLUENCE OF DIFFERENT LYAPUNOV FUNCTION CANDIDATES AND STRUCTURES
In this part, we evaluate the influence of choosing different Lyapunov function candidates and network structures. First, we adopt candidates of different time horizon N E (5, 10, 15, 20, oo to train policies in the CartPole example, and compare their performance in terms of total cost and robustness. Both of the Lyapunov critics are parameterized by L(s) = o(s) d(sss where ((s) is a neural network with m dimensional output. Here, N oo implies using value function Lyapumov candidate. For evaluation of robustness, we apply an impulsive force F' at 100,, instant and observe the death-rate of trained policies. The results demonstrated in Figure 6 (a.b). Then we fix the

---

Under review as a conference paper at ICLR 2020
030
324
1000
(a) LAC
SAC
Figure 5: State trajectories under policies trained by LAC and SAC when tracking different reference signals. Solid line indicates the average trajectory and shadowed region for the 1-SD confidence interval. The X-axis indicates the time and Y-axis shows the concentration of protein to be controlled. Dashed lines in different colors are the different reference signals: sinusoid with period of 150 (brown); sinusoid with period of 200 (skyblue):sinusoid with period of 400 (blue); constant reference of 8 (red); constant reference of 16 (green). We also show the curves in separate zoom-in view in Appendix I.2
horizon of candidates to be N 5 but vary the structures of Lyapunov critic, and compare their performance using the same metric as described above. More specifically, these different structures are: L(s) - (s) (LAC-biquad); L(s) Σラ [o(s) (LAC-abs); L(s) o(s) (Ls)
quad).
JAGent
LACAtI
Shc.
5-15
(a) Candidates Training (b)
(c) Structure- Training
Stractue-Training
(d) Structure-Robuse Robustness
Figure 6: Influence of different Lyapunov function candidates and network structures. In (a) and (c), the Y-axis indicates total cost of policies during training by LAC with Lyapunov function candidates of different length of horizon N and structures, and the X-axis indicates the total time steps in thousand. (b) and (d) shows the death-rate of policies the presence of instant impulsive force F ranging from 80 150 Newton.
As shown in Figure 6, in the CartPole environment, both choices of Lyapunov candidates converge fast and achieve comparable total cost convergence. However, in terms of robustness, the different choices of N play an important role. As observed in |Figure 6 (b), the robustness of controller decreases the time horizon N increases. On the other hand, LAC with different structures converge well and possesses similar robustness to impulsive forces. This further proves that our framework allows for general class of Lyapunov functions, long as the function is semi- comitiositive definite. Besides, it is interesting to observe that LQR is more robust than SAC when faced with instant impulsive disturbance.
CONCLUSIONS
In this paper, we proposed a model-free approach for analyzing the stability of discrete-time nonlinear stochastic systems modeled by Markov decision process, by employing the Lyapunov function from control theory. Based on the theoretical result, practical algorithm for designing stability assured controllers for the stabilization and tracking problems. We evaluated the proposed method in various examples and show that method achieves not only comparable superior performance compared with the state-of-the- RL algorithm but also outperforms impressively in terms of robustness to uncertainties and disturbances.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Karl Aström and Björn Wittenmark. Adaptive control. Courier Corporation, 1989.
Andrew Barto, Richard Sutton, and Charles Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5); 834-846, 1983.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with reinforce- ment learning. Proceedings of the 34th International Conference on Machine Learning- V-ooume 70, pp. 459 468..66..66 JMLR. org, 2017.
Yoshua Bengio, Nicolas
and Razvan Pascanu. Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8624-8628. IEEE, 2013.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model- reinforcement learning with stability guarantees. In Advances in neural information processing systems, pp. 908-919 118-918,22 2017.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programmingg volume 5. Athena Scientific Belmont, MA, 1996.
EK Boukas and ZK Liu. Robust stability and h/sub/spl infin//control of discrete-time jump linear systems with time-delay: an Imi approach. In Decision and Control, 2000. Proceedings of the 39th IEEE Conference on, volume 2. pp. 1527-1532. IEEE, 2000.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: (oo:::166005440 2016
Lucian Buşoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control, 2018.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman and Mohammad Ghavamzadeh. A lyapunov- based approach to safe reinforcement learning. arXiv preprint arXiv: /805:07705 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar Duenez- Guzman. Lyapunov-based safe policy optimization for continuous control. arXiv preprint 19X1::9901.10331, 2019.
Martin Corless and George Leitmann. Continuous state feedback guaranteeing uniform ultimate boundedness for uncertain dynamic systems. IEEE Transactions on Automatic Control, 26(5): 1144, 1981.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Michael Elowitz and Stanislas Leibler. A synthetic oscillatory network of transcriptional regulators. Nature, 403(6767):335, 2000.
Daniel Gorges. Relations between model predictive control and reinforcement learning. IFAC- PapersOnLine, 50(1):4920-4928 2017.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Tuomas Haarojj,, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.059990 2018.

---

Under review as a conference paper at ICLR 2020
Chengming Huang. Exponential mean square stability of numerical methods for systems of stochastic differential equations. Journal of Computational and Applied Mathematics, 236(16):4016 4026, 2012.
Jun Huang, Zhengzhi Han, Xiushan Cai, and Leipo Liu. Uniformly ultimately bounded tracking control of linear differential inclusions with stochastic disturbance. Mathematics and Computers in Simulation, 81(12):2662-261 2672, 2011.
Alberto Isidori. Nonlinear control systems. Springer Science & Business Media, 1995.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning, nature, 521(7553):436, 2015.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint PXXv::509.00977 2015a.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (XX.::15000029700 2015b.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion (in Russian). PhD Dissertation, Univ. Kharkov, 1892.
David Mayne and Hannah Michalska. Receding horizon control of nonlinear systems. IEEE Transactions on automatic control, 35(7):814-824, 1990.
David Q Mayne, James B Rawlings, Christopher V Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789-81) 2000.
Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability. Springer Science & Business Media, 2012.
Navid Noroozi, Paknoosh Karimaghaee, Fatemeh Safaei, and Hamed Javadi. Generation of lyapunov functions by neural networks. Proceedings of the World Congress on Engineering, volume 2008, 2008.
Athanasios Papoulis and Unnikrishna Pillai. Probability, random variables, and stochastic processes. Tata McGraw-Hill Education, 2002.
Theodore J Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research, 3(Dec):803-833 2002.
Vassilios Petridis and Stavros Petridis. Construction of neural network based Iyapunov functions. In The 2006 IEEE International Joint Conference on Neural Network Proceedings, pp. 5059-5065. IEEE, 2006.
Romain Postoyan, Lucian Buşoniu, Dragan Nešić, and Jamal Daafouz. Stability analysis of discrete- time infinite-horizon optimal control with discounted cost. IEEE Transactions Automatic Control, 62(6):2736-2749, 2017.
Danil V Prokhorov. A Iyapunov machine for stability analysis of nonlinear systems. In Proceedings of 1994 IEFE International Conference on Neural Networks (ICNN'94), volume 2, pp. 1038-1031. IEEE, 1994.
Danil V Prokhorov and Lee A Feldkamp. Application of svm to 1yapunov function approxima- tion. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339), volume 1, pp. 383-387. IEEE, 1999.
Spencer M Richards, Felix Berkenkamp. and Andreas Krause. The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, pp. 466-476, 2018.
Halsey Lawrence Royden. Real analysis. Krishna Prakashan Media, 1968.
12

---

Under review as a conference paper at ICLR 2020
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint (.KY::5500002333 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint urXiv:1777.06:3 2017.
Gursel Serpen. Empirical approximation for Iyapunov functions with artificial neural nets. In Proceedings. 2005 IEEE International Joint Conference on Neural Newworks, 2005., volume 2, pp. 735-740. IEEE, 2005.
L Shaikhet. Necessary and sufficient conditions of asymptotic mean square stability for stochastic linear difference equations. Applied Mathematics Letters, 10(3):11--15,, 1997.
Peng Shi and Fanbiao Li. A survey markovian jump systems: modeling and design. International Journal of Control, Automation and Systems, 13(1):1-16, 2015.
Jean-Jacques Slotine, Weiping Li, al. Applied nonlineur control, volume 199, Prentice hall Englewood Cliffs, NJ, 1991.
Aivar Sootla, Natalja Strelkowa, Damien Ernst, Mauricio Barahona, and Guy-Bart Stan. On periodic reference tracking using batch-mode reinforcement learning with application to gene regulatory network control. In 52nd IEEE conference on decision and control, pp. 4086-4091. IEEE, 2013.
Natalja Strelkowa and Mauricio Barahona. Switchable genetic oscillator operating in quasi-stable mode. Journal of The Royal Society Interface, 7(48):1071-1082, 2010.
Richard S Sutton, Andrew G Barto, and Ronald J Williams. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems Magazine, 12(2):19-22, 1992.
Gerald Tesauro. Temporal difference learning and td td-gammon. Communications of the ACM, 38(3): 58-68, 1995.
Arild Thowsen. Uniform ultimate boundedness of the solutions of uncertain dynamic delay systems with -tateddependent and memoryless feedback control. International Journal of control, 37(5): 1135-1143, 1983.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 1rX1v:151100688/ 2015.
Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous aerial vehicles with ipp-guided policy search. 2016 IEEE international conference on robotics and automation (ICRA), pp. 528 535. IEEE, 2016.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
13

---

Under review as a conference paper at ICLR 2020
Appendix
A PROOF OF THEOREM
The existence of sampling distribution Me, (s) is guaranteed by the existence of qr((s) (Assumption 1p
Since the sequence {P(s)p, n. t),, E ]+} converges to (x(s) as t approaches co, then by the Abelian theorem, the sequence { En t= p(s|p, x, x,t), N Z+} also converges and (r( = q- (s). Combined with the form of Ur. Eq. infers that
lim N-
Plel L(s))ds & -agBengr Ca (s)
(A.1)
First, on the left hand-side, according to Eq. 1. L(s) & (2)n(s) for all E and consider that t)) 1,
dtlp,.,t)LL S a2 (5), vs E e
On the other hand, the sequence + E p(s x, (s)), N Z4} converges pointwise the function qu L(s). According to the Lebesgue's 's Dominated convergence theorem(Royden, 1968). sequence (s) converges pointwise function f and is dominated by some integrable function g in the sense that,
[s) S g(s). Ys vn
Then
fn(s)ds= = / lim (s)ds
lim S
Thus the left hand side of Eq.
1.
N
lim
P(s|p 1|l P=(()ss)((s))d'' - L(s))ds
N N
N
1=0
1 N+1
lim
Ep(sle0,00 L(s) Σ Ep(s).n,) ((s))
N
t=
lim N
(BP())...411)((() L(s)
N
Thus taking the relations above into consideration, A.I infers
(EPtelen.NtD))(9)) ((s) < -03 lim E Pf (s) No0
lim
(A.2)
1-000
Since Loco)((8) is a finite value and L is semi-positive definite, it follows that
1
lim Epcelp.m,jCcr (8) S lim t0000
( L(s)
(A.3)
N N
N Q3
Suppose that there exists a state 80 E fsolen (50) S b) such lim Ep(elea.r.t (8) = C, <>> or lim,- Ep(sian.7.1) CT (s) xo. Consider that p(5o)
for all starting states in { (sp) < 6)
(Assumption) 2). then lim E. ((),) Gr (st) 0. which is contradictory with Eq. AA.3. Thus
Vsg E {sule c, (8)) 6}. lim Ep(s (s) 0. Thus the system is stable in mean cost by Definition i
14

---

Under review as a conference paper at ICLR 2020
B PSEUDO CODE OF ALGORITHM
Algorithm Lyapunov-based Actor-Critic (LAC)
Initialize replay buffer 'D and Lagrange multiplier X, B;
Randomly initialize Lyapunov critic network [s(s,, a), actor ({]s)) with parameters oz. Initialize the parameters of target network with + 0: for each iteration do Sample Su according to p: for each time step do
Sample a, from (s) and step forward;
Observe 8t+1, Ct and store (St, at Ct 8641) in D; end for
for each update step do
Sample minibatches of transitions from D and update Lo, x, Lagrange multipliers with
gradients:
Update the target networks:
0 +
end for end for
15

---

Under review as a conference paper at ICLR 2020
C FURTHER EXPERIMENT SETUP
We setup the experiment using OpenAi Gym (Brockman et al. 2016). A snapshot of environments can be found in Figure]
(a) CartPole
(b) HalfCheetah
(c) FetchReach
Figure 7: Snapshot of environments using OpenAl Gym.
C.I CARTPOLE
In this experiment, the controller is to sustain the pole vertically at a target position x = 0. This is modified version of CartPole Brockman al. (2016) with continuous action space. The action is the horizontal force applied on the cart (a E 20, 20)). Jihreshold and Ohreshold represents the maximum of position and angle, respectively, Thhrahold = 10 and Pthrehood = 20 The controller 0 Otreesoodd and the episodes end in advance. Cost function
dies if [z] Tihreshold
Tamass )a + 20 * (
9 )², The episodes are of length 250. For robustness evaluation in Section 5.4
9
we apply an impulsive disturbance force F on the cart every 20 seconds, of which the magnitude
ranges from 80 to 150 and the direction is opposite to the direction of control input. In Section 5.5
the impulsive disturbance has the same magnitude range and direction with that in Section 5,4 but only applied once at instant = 100.
C.2 HALFCHEETAH
HalfCheetah is modified version that Gym's robotics environment (Brockman al. 2016). The task is to control a HalfCheetah (a 2-legged simulated robot) to run at the speed of 1 m The reward r (v 1)2 where is the forward speed of the HalfCheetah. The control input the torque applied on each joint, ranging from -1 to 1. The episodes are of length 200.
For robustness evaluation in Section 5.4] we apply an impulsive disturbance torque on each joint every 20 seconds, of which the magnitude ranges from 0.2 to 2.0 and the direction is opposite to the direction of control input.
C.3 FETCHREACH-V1
We modify the FetchReach in Gym's robotics environment (Brockman al. (016) where the controller is expected to control manipulator's 's end effector to reach a random goal position. The cost designed as input is the torque applied on each joint, ranging from -1 to 1. The episodes are of length 200.
version,
where is the distance between goal and end-effector. The control
For robustness evaluation in Scction 5.4 we apply impulsive disturbance torque on each joint every 20 seconds, of which the magnitude ranges from 0.2 to 2.0 and the direction is opposite to the direction of control input.
16

---

Under review as a conference paper at ICLR 2020
D SYNTHETIC BIOLOGY GENE REGULATORY NETWORKS
Since this system considered here is in nano-scale whose physical property different from the ones considered in Section [ and the system exhibit interesting oscillatory behavior, we illustrate this example separately in this section.
D.1 MATHEMATICAL. MODEL OF REPRESSILATOR
In this example, we consider a classical dynamical system in systems/synthetic biology, the repres- silator, which we use to illustrate the reference tracking problem at hand. The repressilator is synthetic three-gene regulatory network where the dynamics of mRNAs and proteins follow an oscillatory behavior (Elowitz & Leibler 2000). A discrete-time mathematical description of the repressilator, which includes both transcription and translation dynamics, is given by the following set of discrete-time equations:
1)) + dt
(t) +
K1 =t) + un (t)
an
(t)),
02
2t 1) = (f) + dt
(2)2(t) +
+ (t)
Ett)
K2 x2(()
za(1+ 1) = =1 + di
(1 +
K3 + =f((t) + us
03
Es(1),
(D.1)
(1 1) = (1) + di ms(1 1) (1) di
(1) + (1)] + (,(),
C22r (k) (2)2 (4) (s(),
ot 1) (1) + dt-
C30e (1) + (1) + Eo(()
Here, 21, 22, 23 (resp. 34, I', F6) denote the concentrations of the mRNA transcripts (resp. proteins) of genes 1, 2, and 3, respectively. E,, Vi are i.idd. uniform noise ranging from [-8, 6). i.c., E, ~ 0 and for evaluation 5 is set to 0.5 and respectively in Section 5.4 a1, 02 ag denote the maximum promoter strength for their corresponding gene, 71 Y2: " denote the mRNA degradation rates, c, C2, C3 denote the protein degradation rates, 31, 82, Ba denote the protein production rates, and Ki, K2, K3 are the dissociation constants. The set of equations in (D. corresponds topology where gene 1 repressed by gene 2, gene is repressed by gene and gene 3 is repressed by gene 1. dt is the discretization time step.
U(-6,, 6). During training,
In practice, only the protein concentrations fluorescent markers (e.g., green fluorescent protein, GFP or red fluorescent protein, mCherry). The control scheme u; will be implemented by light control signals which can induce the expression of genes through the activation of their photo-sensitive promoters. To simplify the system dynamics and as it is usually done for the repressilator model (Elowitz & Leibler 20000, we consider corresponding parameters of the mRNA and protein dynamics for different genes to be equal. More background on mathematical modeling and control synthetic biology gene regulatory networks can be referred to Strelkowa & Barahona (010); (Sotll et al. (013). In this example, the parameters are as follows:
observed and given as readouts,for instance via
Vi: K₁ 1, = 1.6, = 0.16, 3 0.16, = 0.06, dt
In Fig8, single snapshot of the state temporal evolution without & is depicted. We uniformly initialized between 0 to 5, i.e., (0) ((. (), which is the range we train the policy in Section persistent oscillatory behavior are also exhibiting similar the snapshot in Fig
17

---

Under review as a conference paper at ICLR 2020
17.5
mRNA
mRNA
mRNA
12.5
Protein
Protein
Protein
7.5
2.5-
Figure 8: A snapshot of natural oscillatory behaviour of repressilator system consisting of 3 genes. The oscillations have period of approximately 150 arbitrary time units. The X-axis denotes time and Y-axis denotes value/concentration of each state.
D.2 COMPLICATED REPRESSILATOR
To further evaluate the performance of different algorithms, we additionally include a more compli- cated gene regulatory network, which composed of 4 genes instead 3. Such a network with genes would posses an unstable oscillatory behaviour, as shown in Figurr 9 making it even harder to stabilize. The discrete-time mathematical description of the complicated repressilator is given by the following set of discrete-time equations:
(1(++)) = (1) + di
(1))(1) +
a1
+ u (1)
+ Et)
Ki + x1)
02
22(t 1) = x2(() + dt
(t) +
+ (t
+ Ee(t)
K₂ + xt)
za(t + 1) = =t( + dt
(1) +
03
+ u1(
Ca())
Kg + (1)
(D.2)
2t(t+ 1) = (t) + dt
(t) +
Kg +t) + (t(())
04
+ Ett).
ms([+1) 1) =5( + di
c120 (1) + (1)] {s((1,
11 = 26(() + dt
22 (1) + (f))
1)(++1) (7(() dt
0327 (1) + +
11(1+1) (1) + dl
C42 (1) + (())
Here, T1, 22, T3, F4 (resp. Fs, To, I7, .ss) denote the concentrations of the mRNA transcripts (resp. proteins) of genes 1, 2, and 4, respectively.
The parameters are as follows,
Vi E (1,2,3,4) K2= 1.0; -16, = 0.16, = -0.16, = 0.066, dl
18

---

Under review as a conference paper at ICLR 2020
mRNA
mRNA 2
mRNA 3
mRNA 4
Protein 1
Protein 2
Protein 3
Protein 4
0-
Figure 9: A snapshot of natural behaviour of repressilator system consisting of genes. The X-axis denotes time and Y-axis denotes value/concentration of each state.
19

---

Under review as a conference paper at ICLR 2020
E MARKOVIAN JUMP SYSTEMS
In addition to the systems described in Section C and Section D we introduce two Markovian jump systems (MJS), which contain both discrete switchings (or jumps) and continuous dynamics (Shi & Li 2015), as test beds for the proposed and baseline methods. Specifically, we borrow two simple examples from the linear case of MJS, i.e. Markovian jump linear systems (MJLS). The dynamic of the MJLS could be described by the following state space model.
Fk+1 = Ac, Ik + Beg "k
(E.1)
where Ta and up are the state and control inputs respectively; Ar. and Brg are the parameter matrices. Th is the switching signal governing the switching of subsystems, which takes value in finite set f1....,Ay N] where N the number of subsystems. In MJS, the value
governed by a Markov process, 0k+1 P(ax+1 (xk). The task of the agent is to stabilize the system and cost function is c(x)
|
For the first MJS named MJSI, the parameter matrices given follow,
0.3672
0.7038
A p
B
1.8462
2.0094
0.3468
0.6324
A =
0.7774
1.1872
B₂
(E.2)
0.3468
0.6324
A3 P
B₃
0.7774
1.1872
where p 1.3 and the transition probability of switching signal each instant is uniformly distributed across all three modes. Among the three subsystems, subsystem and 2 are unstable without control input.
To further make the task more difficult, we include a second MJS system, MJS2, of which the parameter is given as
00.227
0.7710
A P
B
1.1600
-0.9912
-0.5084
0.4536
0
A2 P
B2
(E.3)
1.0901
-0.7266
0.4772
0.7313
Ag = P
B₃ =
1.3938
-0.7266
[2]
where p 0.859 and the switching signal is also uniformly distributed. Note that the subsystem and 3 are unstable without control input. Moreover, subsystem 2 in uncontrollable, since the control input cannot effect the system dynamic under this mode.
20

---

Under review as a conference paper at ICLR 2020
F FURTHER VALIDATION OF STABILITY GUARANTEE
In this part, further comparison between the stability assured method (LAC) and that without such guarantee (SAC) is made, by demonstrating the closed-loop system dynamic with the trained policies (in the two Repressilator and two MJS examples).
20000
LIV
SAC
LAC
15000
12500
10000
5000
2500
80
(a) Complicated-Repeesilaae Repressilator
Figure 10: Cumulative control performance comparison. The Y-axis indicates the total cost during one episode and the X-axis indicates the total time steps in thousand. The shadowed region shows the 1-SD confidence interval over 10 random seeds. Across all trials of training, LAC converges to stable solution with comparable or superior performance compared with SAC.
Protein
Protein
Reference
Reference
4C
(a) Repressilator-LAS
(b) Repressilator-Se
Protein
Protein Reference
Reference
(c) Complicated ComplicaatedRepressilaaoo-- Repressilator-LACC
(d) Complicated
Repressilator-SAA Complicaaed-Rppreesslaar---s--
Figure 11: State trajectories over time under policies trained by LAC and SAC in the Repressilator and Complicated Repressilator, In each experiment, the policies are tested over 20 random initial states and all the resulting trajectories are displayed above. The X-axis indicates the time and Y-axis shows the concentration of Protein 1.
As shown in the figures, without stability guarantee, the state trajectories either diverge (see Figure II b and Figure 12 d), continuously oscillate around the reference trajectory equilibrium (see
21

---

Under review as a conference paper at ICLR 2020
Figure 11
and Figure 12 b). In the MJS examples, the trajectories diverge oscillate in unacceptable magnitude (1c7 and le10). Contrarily, the stability assured method stabilizes the system well in all tasks (i.e. the state trajectories converge to the reference signal or equilibrium). In the MIS examples, though temporal oscillation occurs in some of the trials due to the existence of unstable uncontrollable subsystems, eventually all of the trajectories stabilized.
1
2
22
-206
250
(a) MISI LLAC
(b) MJSI-SAC
2080
5
5
5
5
1055
-1080
(c) MJS2-LAC
(d) MIS2-SAC
Figure 12: State trajectories over time under policies trained by LAC and SAC in the two Markovian jump systems. In each experiment, the policies are tested 20 random initial states and all the resulting trajectories are displayed above. The X-axis indicates the time and Y-axis shows the value of states.
22

---

Under review as a conference paper at ICLR 2020
G FURTHER VALIDATION OF STABILITY GUARANTEE
In addition to the evaluation stability in terms of system dynamic in previous sections, this part presents more direct approach for validating the satisfaction of stability condition (2). The
Algorithm 1 aims to solve the dual problem of the original policy optimization problem, i.c. solving
the following min-max max problem,
max (TT) X,9
(G.1)
where À and 3 are positive Lagrange multipliers and updated by gradient ascent. When (2) is satisfied, will continuously decrease until it becomes zero. Thus by checking the value and variation of X, the satisfaction of stability condition during training and at convergence could be validated. More specifically, the decline of À implies the satisfaction of stability condition at that update; À only converges to zero if the stability condition is assured.
Clipping the maximum value of À is necessary, in case that grows too much due to the violation of stability condition during the early training stage, resulting in the inappropriate step length for the policy update. Clipping useful technique prevent instability of optimization, especially in gradient-basee based methods, see (Schulman et al. (2017): y: [engio et al. (2013); Bello et al. (2017); Wang et al.] (2015). Conversely, when the stability condition satisfied, À quickly drops and helps convergence of the algorithm, which inherently prevents overfitting and enhances robustness and generalization.
23

---

Under review as a conference paper at ICLR 2020
H ROBUSTNESS AND GENERALIZATION EVALUATION OF SPPO
In this part, we evaluate the robustness and generalization ability of policies trained by SPPO in the same. First, the robustness of the policies is tested by perturbing the parameters and adding noise in the Cartpole and Repressilator environment, as described in Section 5.4.1 Generalization of the policies is evaluated by setting reference signals that are unseen during training. State trajectories of the above experiments are demonstrated in Figure 13 and Figure 14] respectively. As demonstrated the figures, the SPPO policies could hardly deal with previously unseen uncertainty or reference signals, and failed in all of the Repressilator experiments.
The SPPO algorithm originally developed for the control tasks with safety constraints, i.e. keeping the expectation of discounted cumulative safety cost below a certain threshold. Though Lyapunov method is exploited, the approach is not aimed providing stability guarantee.
naiie levelim..
length=2.0
length-1.0
Kim10
length-1.5
nolze level-1
original
(a) Cartpole
(b) Repressilator
Figure 13: State trajectories over time under policies trained by SPPO and tested in the presence of parametric uncertainties and process noise, for CartPole and Repressilator. The setting of the uncertainty is the same as Section 5.4.1
(a) Repressilator
Figure 14: State trajectories under policies trained by SPPO when tracking different reference signals. The setting of the uncertainty is the same as in Section 5.4.3
24

---

Under review as a conference paper at ICLR 2020
1 ZOOM-IN VIEWS
1.1 ZOOM-IN VIEW OF FIGURE 3
6420
original
length=12
0.15
3.3
4.120
-3303
4.30
.83.
0168
length-1.5
019
length-2.0
0.81
0.39
302
-009
-305
166
359
Figure 15: Zoom-in view of Figure 3 (a)
-006
59
109
290
26
2
05
46
1.0
03
00
153
204
16
Figure 16: Zoom-in view of Figure 3 (b)
25

---

Under review as a conference paper at ICLR 2020
original
K
224
30
200
1280
K=11
8:=3.2
200
1380
4,8
level=1
12.
noise level=0.5
2.6
1906
1290
Figure 17: Zoom-in view of Figure (c)
26

---

Under review as a conference paper at ICLR 2020
original
K-8
126
25
80
3008
K-10
noise level-0.5
noise level=1
Figure 18: Zoom-in view of Figure 3 (d)
27

---

Under review as a conference paper at ICLR 2020
1.2 IOOM-IN VIEW OF FIGURE 5
830
3300
1289
200
1069
3200
15.0
30.0
5.0
1289
Figure 19: Zoom-in view of Figure
28

---

Under review as a conference paper at ICLR 2020
1200
3-0-
2.5-
12.0
66
54-
LO08
1200
Figure 20: Zoom-in view of Figure 5
29

---

Under review as a conference paper at ICLR 2020
J HYPERPARAMETERS
Table Hyperparameters of LAC
Hyperparameters
Repressilator
CartPole
FetchReach
HalfCheetah
Time horizon N
5
5
5
00
Minibatch size
256
256
256
256
Actor learning rate
le-4
le-4
le-4
le-4
Critic learning rate
3e-4
3e-4
3e-4
3e-4
Lyapunov learning rate
3e-4
3e-4
3e-4
3e-4
Target entropy Soft replacement(T) Discount(y)
-3
-1
-5
-6
0.005
0.005
0.005
0.005
0.75
1.0
1.0
0.995
a3
1.0
1.0
1.0
1.0
Lyapunov critic network structure
(256,256,16)
(64,64,16)
(64,64,16)
(256,256,16)
For LAC, there policy network, we use a fully-connected MI.P with two hidden layers of 256 units, outputting the mean and standard deviations of Gaussian distribution. As mentioned in secciinn it should be noted that the output of the Lyapunov critic network is a square term, which is always non-negative. More specifically, we use fully-connected MLP with two hidden layers and one output layer with different units as in Table outputting the feature vector ((s, a). The Lyapunov value is obtained by Le(8, a) = o² (s, a (s..))((.....)))..) All the hidden layers use Relu activation function and we adopt same invertible squashing function technique Haarnoja et al. (2018) to the output layer of the policy network.
networks: the policy network and the Lyapunov critic network. For the
30


Reviews:
[{"review_id": "SyeJTy7ccr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce an algorithm to learn a stable controller using deep NN actor-critic method. They define the stability in the mean cost criteria,  which is used to constrain the critic network as a Lyapunov function. In addition, the semi-positive definiteness of the Lyapunov function is enforced by constructing the critic.\nThe problem is important to control with deep RL. The paper is written clearly. The reviewer has the following questions regarding the stability of the learned policy.\n\n- How is the stability in the mean cost related to the stability of stochastic systems? See, for example, the Lyapunov stability of stochastic systems (survey in [1])?\n- The authors enforce semi-positive definiteness using the construction of the value function approximation as the quadratic function bases. Then the semi-negative definiteness is enforced using penalty on the Lyapunov stability of the critic. Then the target network is trained to minimize the difference between the target and the critic. The question is, how is the stability of the target ensured by minimizing the difference with a Lyapunov critic? Is it possible to have an unstable target function that happens to minimize the distance? \n\n[1] H. J. Kushner, “A partial history of the early development of continuous-time nonlinear stochastic systems theory,” Automatica, vol. 50, no. 2, pp. 303–334, 2014.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eyRil5tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n######## Rebuttal Response:\n\nThanks for the thorough response.\n\nQ2: The title still hasn’t changed on the current draft\nQ4: To be more precise: \n‘a novel data-based approach for analyzing the stability of the closed-loop system is proposed by constructing a Lyapunov function parameterized by deep neural network’ - this alone is not novel, you would need to specify how your method of doing this is new \n‘a practical learning algorithm is designed to search the stability guaranteed controller’ - this is a natural consequence of contribution 1, some further justification is needed as to why this could be viewed as an interesting contribution (i.e. the Lagrangian approach, if this is novel)\n‘ the learned controller is able to stabilize the system when interfered by uncertainties such as unseen disturbance and system parameters variations of certain extent’ - this is not a contribution, but an experimental result.\n\nQ5: The review believes that model-free control and stability-guarantees are fundamentally orthogonal ideas, rather than just under-studied work as the authors have been suggesting in the script and rebuttal. Given that discrete-time Lyapunov stability is defined through expressions along the lines of L(f(x)) - L(x) < 0, for Lyapunov function L and closed-loop dynamics f, claiming that stability is being ‘analyzed’ without f is disingenuous. Instead, by making the value function a Lyapunov function, the goal is that the *converged* value function should produce a stable policy, and still, this is surely only assured within the space of samples. Moreover, the use of the discount factor \\gamma, popular in MFRL, essentially acts as a time horizon, so I’m not convinced a Lyapunov function learned with a \\gamma < 1 can be called stable in the pure infinite-horizon sense.\nWith this in mind, I think the work would benefit from a revised central claim: that the use of Lyapunov value functions (as an inductive bias) provides more *robust* model-free controllers. I believe this message highlights the value of this work for MFRL, without making false assertions. This, in particular, would highlight the fact that many MFRL algorithms are benchmarked on deterministic environments, and therefore incredible brittle as the experimental results suggest.\n\nQ9: This remark was aimed at earlier in the paper, either the introduction or main section, rather than the experimental section. The fact that a value function can be viewed as a Lyapunov function makes sense but I’m not sure it is a well-known fact in the wider community. Basically, an introduction to the intersection of Lyapunov stability and optimal control would improve the paper.\n\nQ:10 The fact that the clipping of the multiplier corresponds to unstable policies during learning demonstrates that this pitfall needs to be expressed explicitly. Whether the stability guarantees apply to the converged policy or also intermediate policies is not clear on the initial reading of the paper.\nFor me, this highlights another weakness in the paper. This initial theorems talk of L(s), which relates to the critic L_c by L(s) = E_{u\\sim\\pi(s)} [L_c(s,u)], however in the subsequent objectives (eg Eq 2), this marginalization never occurs, therefore I don’t feel like you can say Theorem 2 applies to your resultant algorithm.  Moreover, with the \\alpha_3 c term in Equation 3, c should be c(s, a) with a marginalized, which it doesn’t appear to be, and the hyperparameter alpha_3 is never discussed nor tuning explained. Assuming this not done in the code, the experiments need to be re-evaluated with Theorem 2 properly enforced through a sample approximation of the marginalization.\n\nQ11/Q12: Thank you for the Markov jump experiments. I’m not sure I understand why LAC is able to learn the task while SAC cannot. To me, this suggests perhaps a lack of hyperparameter tuning for SAC or further investigation. Moreover, there are typos in captions Fig 1, e and f.\nA note of figures: Please ensure all axes should be labeled and should be of sufficient size. Many are too small and unreadable. Figure 2 looks like it could be 1 plot (though perhaps requires normalization).\nGiven that the strength of this method is the added robustness upon convergence, I think it would be valuable to focus less on time-domain results (Figure 3) (these can be added to the appendix for clarity), but instead show how each parameter/noise variation affects the mean and variance of the episodic return. I would expect that, while LAC provides significant robustness, it is still limited. The results don’t demonstrate this. It would also be interesting to know which hyperparameter controls this limit. I imagine there is a robustness/performance tradeoff.  \n\nIn conclusion, while I appreciate the efforts the authors put into the rebuttal, the extended discussions made me rethink my rating and I have decreased my rating to reject. I believe fixing the issues highlighted above and redrafting the central message must be done before this paper is ready for publication. \n\n\n######## Review:\nThis paper investigates the use of Lyanpunov theory as an inductive bias for improving the stability / robustness of policies in a model-free actor-critic reinforcement learning setting. Through viewing the Critic as a Lyapunov function, optimizing the policy with a Lyapunov-based constraint is meant to ensure the stability of the policy through a ‘cost stability’ metric.. Experimental results show that Lyapunov-based Soft-Actor Critic (LAC) is more robust than SAC on some linear and nonlinear environments.\n\nThe reviewer believes that the study of intersections between Control Theory and RL to be immensely valuable and the authors outline a principled formulation. However,  the implementation, experiments and general manuscript suggest that paper requires further work before it is conference-ready. \n\nAs the author understands it, the current state of the literature of Lyapunov methods for Deep Reinforcement Learning can be summarized as:\nRichards et al, 2018, Classify stable region and learn neural Lyapunov function for a safe exploration strategy\nBerkenkamp et al, 2018: Classify the stable region via GP, move there for exploration\nChow et al 2018 Constrained MDPs for discrete gridworld environments\nChow et al 2019 Constrained MDPs for continuous environments through a projection on the policy\nThis work: Actor-Critic constrained policy optimization with a lyapunov-based value function critic\n\nIn the introduction and the related work, too much emphasis is put on explaining stability and discussing methods like Model Predictive Control (MPC) which do not benefit the rest of the paper. Additionally, the three contributions listed do not seem particularly novel given the past literature. \n\nThe premise of the formulation also presents several unquestioned assumptions and design decisions:\nWhy model-free RL, as the authors also state that many samples are required to validate stability?\nHow does the requirement of stability inform the search strategy in this work? Especially as SAC uses a maximum entropy stochastic policy to aid exploration. \nDo you really get ‘guarantees’ with sample-based methods? I would expect bounds based on the number of samples\nThe cost-based measure of stability seems open to abuse - i.e. for the half-cheetah environment only the centre-of-mass horizontal velocity in covered in the cost function, the stability of the embodiment (joint angles and velocities) are ignored. For the Fetch Reacher, a cost function in cartesian space ignores instabilities from kinematic singularities in joint space. I would image the cost function needs to be a measure on the entire dynamic state. \n\nThe notion of a Value function as a Lyapunov function is very interesting, and since it was the basis of the work, would have benefitted from more discussion, i.e. for which cost/reward function families the equivalence is valid for, and how it compares to other Lyapunov candidate functions.\n\nWith the RL formulation, the requirement of clipping with the lagrangians is suspicious, as it suggests the objective and/or its numerics are not well posed.\n\nWith the choice of experiments, they do not seem to question the central problem outlined by the paper. Rather than show environments SAC returns unstable trajectories during learning, the experiments aim to demonstrate instead a general robustness. The reviewer appreciates that stability is difficult to assess; however, while stability is heavily linked to robustness, a paper title promising stability guarantees should demonstrate some strong empirical evidence stability.\nAdditionally, the choice of environments do not seem to be ideal test beds for stability - i.e, the half-cheetah is stabilized via interactions with the ground. The reviewer would prefer to see simpler nonlinear environments, such as Markov Jump Processes / Switching Linear Dynamics, where SAC clearly demonstrates instability during learning which LAC is sufficiently regularized against. Additionally, while the `repressilator’ is an interesting application to the domain of bioengineering, its addition does not seem to be especially motivated by the central goal of the paper, so just adds to confuse the reader with unnecessary theoretical content.\n\nMoreover, a brief literature review uncovered some relevant earlier work which was not cited: \nConstruction of neural network based Lyapunov functions, Petridis et al, 2006\nGeneration of Lyapunov functions by neural networks, Noroozi et al, 2008\nLyapunov Design for Safe Reinforcement Learning, Perkins et al, 2002\nSome of the references also appear incorrectly formatted or incorrect, i.e. the reference for Spencer et al, 2018 should be the CoRL 2018 version rather than arxiv. \n\nAlso, the general use of grammar in the manuscript would benefit from another draft. In particular, the title could be improved, i.e.\n    Model-free Control of Nonlinear Stochastic Systems with Stability Guarantees\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e1O5yJ9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors studied the model-free RL approach for learning a policy with stability guarantees. Leveraging the Lyapunov stochastic stability criterion, instead if minimizing the cumulative cost (plus a soft entropy), they propose optimizing an objective function with a specific Lyapunov critic, which is a specific critic function that satisfies the Lyapunov criterion to guarantee stability. They also show in several Cartpole, Mujoco, and Repressilator experiments that this approach is more robust to perturbations (such as sinusoids), where the agent are more robust to dynamic uncertainties and disturbances. \n\nIn general, the topic of guaranteeing stability is a topic in safe RL, and I find this work of enforcing stability in model-free RL interesting. Through the specific parameterization of quadratic Lyapunov function (in the latent space), the authors proposed learning a new critic function that is a value function but at the same time (almost) satisfies the Lyapunov constraints. While this is an interesting idea, and the experimental results look promising, I do have several questions.  \n\nFirst, regarding the learning problem of Lyapunov function, how does the proposed way of learning L differ from the one in Richard'18: The lyapunov neural network: Adaptive stability certification for safe learning of dynamic systems, where the problem is formulated as a classification (while in here it is a regression problem)? \nSecond, while this approach is intuitive, since the approach is penalty-based (Lagrangian based), I do not see how the Lyapunov criteria in Theorem 1 is guaranteed, in this case is stability guaranteed by the policy learning  algorithm? If not, what do the authors do to enforce that? \nThird, if one formulates the immediate constraint cost of the CMDP to be the distance of the state to the equilibrium point,  then the (undiscounted, shortest-path type)  CMDP total cost constraint should guarantee stability (because the total distance cumulative cost is bounded, meaning that the distance cost converges to zero). Then, one can use the Lyapunov approach by Chow'19 (in modulo to their setting in discounted MDPs) to enforce stability (which is a specific notion of safety in this case).  How does the proposed method compare with this approach? Can the authors provide numerical comparisons with the method proposed by Chow'19 as well?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method to guarantee the stability of a learnt continuous controller by optimizing the objective through a Lyapunov critic. The method is demonstrated on low dimensional continuous control problems such as cart pole. 

The reviewers were mixed in their opinion of the paper, especially after the authors' rebuttal. The concerns center around some of the authors' claims regarding theoretical results, in particular that stability guarantees can be asserted for a model-free controller. This claim seems to be incorrect especially on novel data where stability cannot be guaranteed, thus indicating that 'robust controller' might be a better description. There are also concerns about the novelty and the contributions of the paper. Overall, the method is promising but the claims need to be carefully written. The recommendation is to reject the paper at this time.
Decision: Reject