Reinforcement Learning for Control with Probabilistic Stability Guarantee
Authors: Minghao Han
Zhipeng Zhou
Lixian Zhang
Jun Wang
Wei Pan
Venue:  (2021)
Keywords: control
Lyapunov stability
REINFORCE
finite-sample bounds

Abstract: Reinforcement learning is promising to control dynamical systems for which the traditional control methods are hardly applicable. However, in control theory, the stability of a closed-loop system can be hardly guaranteed using the policy/controller learned solely from samples. In this paper, we will combine Lyapunov's method in control theory and stochastic analysis to analyze the mean square stability of MDP in a model-free manner. Furthermore, the finite sample bounds on the probability of stability are derived as a function of the number M and length T of the sampled trajectories. And we show that there is a lower bound on T and the probability is much more demanding for M than T. Based on the theoretical results, a REINFORCE like algorithm is proposed to learn the controller and the Lyapunov function simultaneously. 

Under review as a conference paper at ICLR 2021
REINFORCEMENT LEARNING FOR CONTROL WITH PROBABILISTIC STABILITY GUARANTEE
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning is promising to control dynamical systems for which the traditional control methods are hardly applicable. However, control the- ory, the stability of a closed-loop system can be hardly guaranteed using the policy/controller learned solely from samples. In this paper, will combine Lyapunov's 's method in control theory and stochastic analysis to analyze the mean square stability of MDP in model-free manner. Furthermore. the finite sample bounds on the probability of stability are derived as a function of the number M and length T of the sampled trajectories. And we show that there is a lower bound on T and the probability much more demanding for M than Based on the theoretical results, a REINFÖRCE-liK algorithm is proposed to learn the controller and the Lyapunov function simultaneously.
INTRODUCTION
Reinforcement learning (RL) has achieved superior performance on some complicated control
tasks (Kumar et al., 2016 Xie al. 2019 Hwangbo al. 2019) for which the traditional control
engineering methods
be hardly applicable (Aström and Wittenmark 1973 Morari and Zafiriou
1989; [llotine et al. 1991.. The dynamical system to be controlled is often highly stochastic and nonlinear which is typically modeled by Markov decision process (MDP), i.e..
81+1 P(st+1|8;, a;), VI E
(1)
where s E S C I denotes the state, A C R" denotes the action and P(s 0z) is the transition probability function. An optimal controller can be learned from samples through "trial and error" by memorizing what has been experienced (Kaelbling et al., 1996: [ertsekaa 2019). However, there major caveat that prevents the real-world application of learning methods for control engineering applications. Without using a mathematical model, the current sample- based RL methods cannot guarantee the stability of the closed- loop system, which is the most important property any control system as control theory.
The most useful and general approach for studying the stability of dynamical system is Lyapunov's 's method m Lyapunov (1892), which is dominant in control engincering [iang and Jiang (2012); Lewis et al.] (2012) Boukas and Liu (2000). In Lyapunov's 's method, a suitable "energy-like function L(s) selected and its derivative along the system trajectories is ensured to be negative semi-definite, i.c., L(s++1) L(ss) < for all time instants and states, so that the state goes in the direction of decreasing the value of Lyapunov function and eventually converges to the origin sub-level set of the Lyapunov function.
Lyapunov
In the traditional control engineering methods, a mathematical model must be given, i.e., the transition probability function in ① known. Thus the stability can be analyzed without the need assess all possible trajectories. However, in learning methods, as the dynamic model is unknown, the "energy decreasing" condition has to be verified by trying out all possible consecutive data pairs in the state space, i.e., to verify infinite inequalities L($++1) - L(s)) < 0. Obviously, the "infinity" requirement makes it impractical to directly exploit Lyapunov's method in model-free framework.
In this paper, we show that the mean square stability of the system can be analyzed based on finite number of samples without knowing the model of the system. The contributions of this paper are summarized as follows:
1. Instead of verifying an infinite number of inequalities over the state space, it is possible to analyze the stability through sampling-hasse method where only inequality is needed.

---

Under review as a conference paper at ICLR 2021
2. Instead of using infinite sample pairs {st+1, st). finite-sample stability theorem is proposed to provide a probabilistic stability guarantee for the system, and the probability is an increasing function of the number M and length T of sampled trajectories and converging 1 as .M and T grow.
3. As independent interest, also derive the policy gradient theorem for learning stabilizing policy with sample pairs and the corresponding algorithm. We further reveal that the classic REINFORCE algorithm (Williams) 1992) is special case of the proposed algorithm for the stabilization problem.
We also conclude two takeaways for the paper:
Samples of a finite number M and length T of trajectories can be used for stability analysis with a certain probability. The probability is monotonically converging to 1 when M and T grow.
There a lower bound on T and the probability is much more demanding for M than T. The REINFORCE like algorithm can learn the controller and Lyapunov function simultane- ously.
The paper is organized follows: In Section 2 related works are introduced. In Section 3 the definition of mean-square stability (MSS) and the problem statement is given. In Section[]] the sample- based MSS theorem is proposed. In Section [ we propose the probabilistic stability guarantee when only finite number of samples are accessible and the probabilistic bound in a relation to the number and length of sampled trajectories is derived. In Section 6 based on the stability theorems, the policy gradient is derived and model-free RL algorithm (L-REINFORCE) is given. Finally, a simulated Cartpole stabilization task is considered to demonstrate the effectiveness of the proposed method In Section 7 the vanilla version of L-REINFORCE is tested on a simulated Cartpole stabilization task to demonstrate the effectiveness; it is further incorporated with the maximum entropy framework to control the more high- gigh-dimensional and stochastic systems, including legged robot, HalfCheetah, and the molecular synthetic biological gene regulatory networks (GRN) corrupted by the additive and multiplicative uniform noises.
2 RELATED WORKS
Lyapunov's Method As a basic tool in control theory, the construction/leaming of the Lyapunov function is not trivial and many works are devoted to this problem (Noroozi et al. 2008 Prokhorov 1994, Serpenn 2005| Prokhorov and Feldkamp 1999). In PPerkins and Barto (2002). the RL agent controls the switch between designed controllers using Lyapunov domain knowlede ledge so that any policy is safe and reliable. Petridis and Petridis (2006) proposes straightforward approach to construct the Lyapunov functions for nonlinear systems using neural networks. Richards et al. (2018) proposes learning-basse approach for constructing Lyapunov neural networks with the maximized region of attraction. However, these approaches require the model of the system dynamics explicitly. Stability analysis in model-free manner has not been addressed. In BBerkenkmp et al. (017) local stability is analyzed by validating the "energy decreasing" condition on discretized points in the subset of state space with the help of a learned model, meaning that only a finite number of inequalities need to be checked. This approach is further extended by using a Noise Contrastive Prior Bayesian RNN in Gallieri et al. (2019). Nevertheless, the discretization technique may become infeasible as the dimension and space of interest increases, limiting its application rather simple and low-dimensional systems.
Reinforcement Learning In model-free reinforcement learning (RL), stability is rarely addressed due to the formidable challenge of analyzing and designing the closed-loop system dynamics by solely using samples Buçoniu et al.] (2018). and the associated stability theory in model-free RL remains as an open problem Buşoniu et al. 22118; Gorges (017). Recently, Lyapunov analysis is used in model- -free RL to solve control problems with safety constraints Chow et al. (2018) 2019). In Chow et al.) (2018). Lyapunov- based approach for solving constrained Markov decision processes is proposed with a novel way of constructing the Lyapunov function through linear programming. In Chow et al. (2019). the above results were further generalized to continuous control tasks. It should be noted that even though Lyapunov-based methods were adopted these results, neither of them addressed the stability of the system. In Postoyan et al. (2017), an initial result is proposed for the stability analysis of deterministic nonlinear systems with optimal controller for infinite-horizon discounted cost, based on the assumption that discount is sufficiently close to 1. However, in practice, it is rather difficult to guarantee the optimality of the learned policy unless certain assumptions on
2

---

Under review as a conference paper at ICLR 2021
the system dynamics are made Murray et al. (2003 thal-f and Lewis (2005) Jiang and Jiang (2015). Furthermore, the exploitation multi-layer neural networks as function approximations Mnih et al. (2015): Lillicrap et al. (2015) only adds to the impracticality of this requirement.
Given certain information on the model, Adaptive dynamic programming (ADP) can guarantee convergence to the optimal solution, and thus stability is naturally ensured (Balkkrihhaan et al. (2008). For nonlinear systems with input-affine structure, model-free ADP algorithms can guarantee the stability of the closed-loop system Murray ct al. (2003); [Abu- (Abu-Khalai and Lewis (2005); Shih et al. (2007); Jiang and Jiang (2015); Deptula et al. 20180. This paper steps beyond the scope of control- athne systems and are devoted to learning controller with a stability guarantee for the general stochastic nonlinear system. To the best of the author's 's knowledge, the finite sample-based approach for the stability analysis of stochastic nonlinear systems considered in this paper still missing.
For the model- approaches, promising results on stability analysis are reported but generally based on certain model assumptions. Model predictive control (MPC) has long been studying the issue of optimal control of various dynamical systems without violating state and action constraints, and Lyapunov stability is naturally guaranteed (Mayne and Michalska 1990, Michalska and Mayne 1993] Mayne et al. 2000). Favorable as it may seem, the nice properties above are built upon the accurate and concise modeling of the dynamics, which narrows its scope to certain fields. In Ostafew et al. (2014), a learning- baannd nonlinear MPC algorithm is proposed learn the disturbance model online and improve the tracking performance of field robots, but first, a priori model is required. swsaii ett al. 2013) proposed new learning-hasse -aased MPC scheme that provide deterministic guarantees on robustness while performance is improved by identifying a richer model. However, it is limited to the case that linear model with known uncertainty bound is available. Other results concerning learning-based MPC are referred to Aswani et al. (201T): Bouffard al. (2012): Di Cairano et al.
(2013).
In Bobiti 2017): Bobiti and Lazar 2018 a sampling-basee approach for stability analysis and
domain of attraction estimation is proposed for deterministic nonlinear systems. The reliability of the estimation is addressed with a probabilistic bound on the number of samples, however, based on the assumption that all the samples are independently distributed. This infers that given multiple state trajectories, only the first-step data are applicable for the stability analysis, which is inefficient in a model-free framework and will be improved in this paper. Nevertheless, the aforementioned approach can be favorable in model-based setup (Gallieri et al. 2019). given that 1-step predictions can be performed in parallel. It should also be noted that this paper is to address the stability analysis and control of stochastic systems, while the results above are focused on the deterministic nonlinear systems.
3 PROBLEM STATEMENT
Before establishing any stability theorem, the definition of stability needs to be properly given. In this paper, we will focus on the mean square stability (MSS) which is commonly known in control theory. The definition of MSS is given as follows.
Definition 1 (Shaikher 1997) The stochastic system is said to be mean square stable (MSS) if there exists a positive constant b such that lim, Es, || 1|3 = 0 holds for any initial condition so E {so ||| 2/2 < b). I is arbitrarily large then the stochastic system is globally mean square
stable (GMSS).
MSS basically says that, on average, the state of a system, starting from an initial position in the state space, tends towards the equilibrium as time goes to infinity, It should be noted that the stability conditions of Markov chains have been reported in (Shaikhet) 1997 Meyn and Tweedie, 2012). however, of which the validation requires verifying infinite inequalities on the state space if S is continuous. Unfortunately, the finite sample-based approach for stability analysis where only one inequality needs to be checked is still missing.
For the sample- cample-based approach, the key challenge is the theoretical gap in "finity" guarantees, i.c., (I) from infinite L($(+1) L(s)) to only a single inequality related to sample expectation: (2) from infinite samples expectation to finite samples expectation. Thus in this paper, two sets of theoretical questions need to be answered.
3

---

Under review as a conference paper at ICLR 2021
Q1. What does sample-based Lyapunov theorem look like and what are the assumptions and conditions needed to use a single Einfinite samples (L(8++1) - L(+)) instead of the infinite
[(s+11== L(s,?
Q2. What will be the number of samples needed to guarantee stability for a given probability, Emtinite samples is changed to Ejiitt samples? ? What is the analytical form of the probability as function of the number M and length T of the sampled trajectories?
Before proceeding, some notations are to be defined. We introduce c(s) è min mmin([=||2..)) (), to
denote the clipped norm of state. The closed-loop transition probability is denoted as P=(s'(s) & LA ((s)s) d)d.. We also introduce the closed-loop state distribution at a certain instant & as P(s|p, n. t), which could be defined iteratively: P(s'lp, म. t + 1) [s (s(ss IT, t)ds, Vt [1,,)) and P(s|p, TI = p(s), where p(s) is the starting state distribution.
4 SAMPLE-BASED LYAPUNOV STABILITY GUARANTEE
In this section, we will answer Q1 in Section and present the key results sample-based stability analysis. We will show that only a single inequality Eintinite samples ((+++1) ) [(xr)) < 0 is enough for the verification of stability. First, make the following assumption which is commonly exploited by many RL literature (Sutton et al. 2009 Korda and La 2015 2019).
Bhandari et al. 2018 Zou et al.
Assumption The Markov chain induced by policy π is ergodic.
follows that there exists a unique stationary distribution (s) lim P(s|p, n, t). The verification of ergodicity is in general an open question in practice. There are many systems proved to be ergodic physics, statistic mechanics, economics, e.g. gambling games Peters (2019), the Anosov flow [Anosov (2010), and dynamical billiards Park (2014). etc. The study of ergodicity of various systems and its verification composed major branch of mathematics. If the transition probability is known for all states, the validation is possible but requires a large source of computation power to enumerate through the state space. As matter of fact, the existence of the stationary state distribution is generally assumedd to hold in the RL literature Mclo et al. (2008); ;; Levin and Peres (2017): Bhandari et al. (2018); Zou et al. (2019). In this paper, focus on analyzing the stability of such systems with probabilistic bound, as well as developing an algorithm to find stabilizing controllers.
In Definition I stability is defined in relation to the set of starting states, which is also called the region of attraction (ROA). If the MSS system starts within the ROA, its trajectory will be surely attracted to the equilibrium. To build sample-based stability guarantee, we need to ensure that the states in ROA are accessible for the stability analysis. Thus the following assumption is made to ensure that every state in ROA has a chance to be sampled.
Assumption 2 There exists a positive constant b such that p(s) O, Vs E {s|c(s) S b).
Based the above assumptions, we exploit Lyapunov's 's method to prove the sample- based stabil- ity theorem. In Lyapunov's 's method, a positive definite function called Lyapunov function is needed. The selection of the Lyapunov function is not trivial and largely determines the result of stability analysis. In this paper, we construct the Lyapumov function using the following parameterization,
[s() = (fo(s) - (0)) + ac(ss
(2)
where fa(s) is a fully-connected neural network (NN) with ReLU activation function. á denotes the parameters of the network and is a small positive constant.
Theorem The stochastic system Q is mean square stable if there exists a function L: S+RR. RR+ and positive constants on, 0: and 033, such that
(1)(s) < L(x) S 0020 (x)
(3)
E Egnpy (E L(8') T(x) + age (s)) S
(4)
where
1 T
(s()
lim T0000
((s)p, 11,
T
1=1
is the infinite sampling distribution (ISD).
4

---

Under review as a conference paper at ICLR 2021
The proof can be found in Section in the Appendix. The general idea of the proof will be summarized the following. First, we prove that ISD fa exists if yn exists. Then we exploit the Abelian theorem and Egorov theorem to prove that L(sx) converges to zero at the infinite instant. Finally, 3 establishes the relation between ((s)) and c(s;) and concludes the proof.
Proof:
Remark For the Lyapunov function in (2) the value of 0r2 can be approximately estimated. In practice, we are typically concerned with the stability in a finite space S where the ||>|] I e and c(s) = =s11.. Thus max, (s) (0)02///ls /||=|1 a is valid choice for a2- Considering that the neural network f with Relu activation is Lipschitz continuous, follows that as = L1 + a where the Lipschitz constant I.g can be efficiently estimated by using approaches in the literature (Scaman
and Virmaux 2018 Fazlyab et al. 2019- Zou el al. 2020
It can be found that in Theorem | the infinite number of energy decreasing conditions are replaced by only single sample-based inequality (4). However, the validation of stability through sample-based approach comes with a cost: it theoretically requires a tremendous, if not infinite, number of samples to thoroughly estimate the state distributions at instants from 0 to infinity, which is impractical.
Theorem iss valid for both model-free and model-based approach since the sample- -ased energy decreasing condition is aimed at canceling the requirement of point-wise energy decreasing condition. Nevertheless, in the model-free setting, the estimation of transition probability in (4) only adds to the complexity of sampling. In the next section, we will show that a finite number of samples should be informative enough to guarantee stability with certain probability. More specifically, the probabilistic stability bound will be given by closing the gap between infinite and finite- sample guarantees.
5 FINITE SAMPLE PROBABILISTIC STABILITY BOUND
In this section, we will answer Q2 in Section 3 and present the finite sample-based stability theorem.
To estimate the μ in Theorem I an infinite number of trajectories of infinite time steps are needed, whereas in practice only M trajectories of T time steps are accessible. Thus in this section, we will first introduce the finite-time sampling distribution (FSD) n² & + D.= P(slp, m, t), as an intermediate to study the effect of the finite sumple-based estimation. Apparently, limy M = Mr
The general idea of exploiting / is: we first derive the deviation of AL()) from E,, AL(s) with respect to T,, where
( A Egso,, L(8'= - L(8) + a3c(8)
then we study the effect of estimating E,y (sss) with sample average and derive the probabilistic bound. Finally, the above effects are unified to propose the finite sample-based stability guarantee.
Now, we first close the first gap in terms of deviation between E, (L(s) and E,,, AL(s.. To quantitatively analyze this effect, we introduce the following assumption.
Assumption There exist constant 7 E (0,1) 1) such for any π
T
|P(s|T. 4 (s)ll1 < 2T
(5)
where ||
qr (8)]l denotes the Li-distance between probability measures P and Q.
Remark 2 It should be noted that the assumption above is not strict and should be generally easy to satisfy for ergodic MDPs. Because y= denotes the stationary state distribution, it naturally follows that Σ || p,t) - (T)(T) 2T, where (T) E 1] without any further
assumption. The assumption proposed here merely replaces this time-varying ((T) with constant. As a matter of fact, uniform ergodic for irreducible and aperiodic Markov chains (Levin and Peres 2017 Bhandari et al. 2018 Zou et al.) (019) is a special case of the above assumption, where the state distribution is required to converge to qu exponentially the rate of 7². Nevertheless, Assumption 3 allows us to give the quantitative bound for the deviation between E,,, A AL(s) and
Eyr (s)s) with respect to T.
5

---

Under review as a conference paper at ICLR 2021
Based Assumption introduce the following Lemma.
Lemma Let T denotes the length of trajectories (also known as episodes or sequences). If there exist positive constants an, 02 such that (3 hold, then
- AL(s s 20(as + +2)
(6)
The proof can be found in Section B in the Appendix. As shown in (6). the deviation of finite-time estimation of AL(s) from the infinite time estimation decreases as T grows and converges to zero if I' is infinity
Proof:
In the following, of T steps. It is worth mentioning that since M trajectories are independent from each other, each trajectory as whole is applicable for the estimation (s) under μ This will be demonstrated in the following lemma, where increasing M is desirable for the reduction of estimation deviation, while T doesn't It effect the probabilistic bound.
will derive the probabilistic bound estimating B.rAA( with M trajectories
Lemma Let M denote the number of trajectories and denote the length of trajectories. If there exist positive constants on, 02 such that 3 hold, then v/3 2 O,
Σ (L(sc41,m) m.). + a))(st. - AL(s) S -B) s exp Lexp(- (-(2a2 (xxp(1120))) 2M MT
T M
)=
2= mL
where 8t,m denotes the sampled state the m-th trajectory at time
Proof: The proof can be found in Section C in the Appendix. A noteworthy fact is that L(.) in (7) bounded since (3) hold and a positive semi-definite variable clipped by Thus it straightforward to apply Hoeffding's inequality to derive the probabilistic bound.
Now, the finite sample estimation of (Lss) and E,,, (s(s) are connected with E, (L(s) respec- tively by Lemma D 2 we will unify them derive the desired probabilistic stability guarantee.
Theorem 2 If there exists function L S + R. and positive constants on and az such that 3 hold, and for a number of M trajectories with T time steps there exists a positive constant & such that
T T
(8)
T
MT ΣΣ (I(s+1,m) Is + azc(s1,))
m=11
then the stochastic system can be guaranteed be mean square stable with probability least
P (LE,. 0) 2 1- (-2M( -TTT--- -T"-bb y)
(10)
b2
where bj 2(as + az and b2 (2a² 0z)a T. If desired confidence of stability guarantee is
least 5, the associated overall sample complexity is at least Ologg 1 To achieve confidence
M and have to satisfy M(c - M 6,) 22 + ag)² log
Proof: The proof can be found in Section D in the Appendix. The idea is that estimate (
with finite samples in (9) and strengthen this finite sample-based condition with a constant c, such that small deviation in estimation will not cause misjudgment of stability. In practice, is hyperparameter to be tuned according to the number of samples available.
Remark In (Kearns and Singh 2002 Strehl el al. 2006 Jin el al. 2018), the finite-sampl analysis and asymptotic convergence of various classical RL algorithms have been extensively studied. However, to the best of our knowledge. Theorem 2 the first finite-sample analysis for sample-based stability analysis, providing a probabilistic stability guarantee that is related to the number of samples. The probabilistic bound 10 monotone increasing function T and M.

---

Essentially, despite the scale of T. function and sets 0g This implies that given system (I). REINFORCE actually updates the policy towards a solution that can stabilize the system, although it is now aware of under what view REINFORCE, as Under review as a conference paper at ICLR 2021
and approaches to I as T and M tend infinity. Intuitively, the trajectories with inadequate length cannol reflect the evolution of the system dynamics and thus are not applicable the stability analysis. Thus the requirement that the length of the trajectories T be greater than minimum value 88) is reasonable. Nevertheless, it is possible to derive tighter bounds in the future by applying other inequalities, such as Bernstein inequality. The sharpness of the derived bound will be illustrated combined with a Caripole example Section
SAMPLE-BASED CONTROL WITH STABILITY GUARANTEE
Based on the theoretical results in the previous sections, one can judge whether the system is stable given several finite-length trajectories by estimating ② The theoretical results the stability theorems using Lyapunov's method do not, however, give a prescription for determining the Lyapunov function and controller. To translate the theorem into practical algorithms, the high-level plan is to parameterize [(s) with (2) and the controller π [a]s)) with an arbitrary NN To (a) s). Then e and 0 will be updated separately and iteratively using stochastic gradient descent algorithms until system (T) stabilized such that (9) is satisfied. We use T to denote a trajectory (r = {sy a1 ST }..
and т is the shorthand for indicating that the distribution over trajectories depends on P(r) p(s1) II'., P(s++1 (1,,4)))) (()()()))))
6.1 POLICY GRADIENT
In this subsection, we will focus on how to learn the controller in an iterative manner, repeatedly estimating the policy gradient of the target function with samples and updating 9 through stochastic gradient descent. (s) is temporarily assumed to be given, i.e., & are fixed. In Section 6.2 we will
show how the Lyapunov function is selected and learned after 0 is updated.
Since the left-hand side of (9 is the unbiased estimate of AL on pI the problem can be formulated by
find 0, s.t. P.T (L(s) S-c
(11)
A straightforward way of solving the constrained optimization problem above would be the first-order method [Bertsekas] (2014) (Chapter 4), also known gradient descent. each update step, the gradient of (11) with respect to 6 is estimated with samples, and 0 updates a small step in the opposite direction of the estimated gradient vector. The gradient of (I1) with respect is derived in the following theorem.
Theorem The gradient of Lyapunov condition 77 is given by the following,
VOLNT AL(s)= = =E PTo
ve
log (e( (aaC + MOTHIll
(12)
T~
T
1=1
where Ceista Die,, ((8)) denotes the sum of cost over a time interval and C++
The proof of Theorem can be found in Section in the Appendix.
Surprisingly, we found that the policy gradient derived in Theorem is very similar to that used in the vanilla policy gradient method, ie.. REINFORCE Sutton and Barto (2018). in the classic RL
paradigm. In RL, the objective is to minimize a certain objective function Jo and the policy gradient is given as follows:
E T~ag
ΣT c(st)
V
V log (a C:)
(13)
Toumt
12 and 13 are equivalent if one chooses c(s) (s) to be the Lyapunov
conditions the solution is guaranteed be stabilizing. In particular,
7

---

Under review as a conference paper at ICLR 2021
special case of our result, since we prove that many other choices of as and Lyapunov functions are admissible to find a stabilizing solution. The default setting c(s) as L(s) and 03 - 1 in REINFORCE may not satisfy 9 while we reveled that many other feasible combinations of L and 03 potentially exist.
In light of this connection with REINFORCE, is natural to propose a similar learning procedure based Theorem B] which name "rapunoy-REINFORCEE (termed as 'L-RRENFORCE), L-REINFORCE updates the policy with the policy gradient proposed in (12). Instead minimizing the objective function (..1). L-REINFORCE aims to learn stochastic policy (alss) such that the conditions in Theorem are satisfied.
Furthermore, to reduce the variance in the estimation of (12) and speed up learning, it is desirable to introduce a baseline function b(s) in (12) and the estimation is still unbiased Sutton and Barto (2018):
1 T
VOR AL(s) Pig
E T~ME
Vo ( (0gC441r
+ 1(841)) [o) b(s))]
6.2 LYAPUNOV FUNCTION
The Lyapunov function is parameterized using DNN fg in 2). In fact, any real function f is admissible in ( construct Lyapunov function, thus many ways of updating o are applicable in our framework, e.g. Prokhorov (1994); Petridis and Petridis (2006): Richards et al. (2018))))) In this paper, intuitively choose the value function to be the update target for to examine the effectiveness of proposed results as exploited |Berkenkamp et al. (2017): Chow et al. (2019) and leave other possible choices for future work.
To wrap up, the L-REINFORCE algorithm is summarized in Algorithm !
Algorithm L-REINFORCE
repeat
for 1, 2, M do
Collect transition pairs following Te for T steps end for 0 < 0 Vo AL(s)
Update e of I Lyapunov function/value network to approximate the designed target until There exists 03 such that E,r, AL(s) < -6
7 EXPERIMENT
In this section, two sets of experiments are conducted. First, the vanilla L-REINFORCE algorithm is evaluated Cartpole example with comparison to REINFORCE and soft actor-critic (SAC). Then, to further demonstrate the effectiveness of the proposed framework on more complicated systems, L-REINFORCE is incorporated with the maximum entropy method and tested in the more high-dimensional and stochastic systems.
7.1 CARTPOLE EAXAMPLE
To demonstrate the effectiveness of the proposed method, we consider the stabilization task of a simulated Cartpole Brockman et al. (016). The goal to stabilize the pole vertically at the position I = 0. We adopi REINFORCE as the baseline method for comparison. In addition, soft actor-critic (SAC) Haarnoja et al. 2018), state-of-the-art off-policy RL algorithm, is also included. L-REINFORCE and REINFORCE select the action among { f-10, 0, 10). SAC selects the control force in the continuous space with the same minimal and maximal value, thus better performance can be potentially achieved. The detailed experiment setup and hyperparameters are presented in Appendix F
is important to note that the stability of a system can not be judged from the cumulative cost (or return) because stability is an inherent property of the system dynamics and a stable system may not be optimal in terms of the return. Thus in Figure T show the transient system behavior under the
8

---

Under review as a conference paper at ICLR 2021
$100
4005
22-
11.
5200
200
-3
-200
-20
L-REINFORCE
(b) REINFORCE
(c) SAC
Figure 1: Phase trajectories of the agents trained by L-REINFORCE, REINFORCE, and SAC. The X-axis denotes the position x and the Y-axis denotes the angle 0 in rads. The trajectories are 500 timesteps and the states at different instants are indicated by respective colors, corresponding to the color-bar to the right.
learned policies of L-REINFORCE, REINFORCE, and SAC. The agents initialized different positions in the space and their subsequent behaviors are observed. As shown in Figure starting from different initial states, L-REINFORCE can efficiently stabilize the system to the origin. On the contrary, the cartpole diverges under the control of REINFORCE (Figure [ b). SAC can also keep the pole vertically, but the cart can not be stabilized to the position where x = O. In some cases, the SAC agent even slowly drifts away (see the left side of Figure T c). To have more clear view, these trajectories are further illustrated in the time domain in Appendix
To let the readers have intuitive sense of the probabilistic stability bound, the bound for Cartpole under the control of the L-REINFORCE agent shown in Figure 2 As shown in Figure 2 the probability of stability increases sharply the minimum T requirement (8) is satisfied. Increasing M and T are both helpful for raising the confidence of stability guarantee. The probabilistic stability guarantee measures the reliability of learned policies. By tuning the hyperparameters such M, T, and ag, one can achieve the confidence of stability guarantee according the real needs.
1.0
0.75
0.8
0.50
0.2
0.25
0.0
8
6000
100000
3000g
0.00
8000g
20000
100000000
300
Figure Visualization of the probabilistic stability bound. The X-axis indicates the length of trajectories and Y-axis indicates the number of episodes M. The Z-axis indicates the probability of stability and the values colored differently according to the bar.
7.2 HIGH DIMENSIONAL EXAMPLES
In this part, will illustrate the effectiveness of the proposed framework some high dimensional control problems, where the system dynamics are highly nonlinear and even corrupted by various noises, thus making them more stochastic and challenging. Three examples are included: high- dimensional continuous control problem of 3D robots, HalfCheetah, the molecular synthetic biological gene regulatory networks (GRN) corrupted by the additive and multiplicative uniform noises. In the living cells of biological systems, gene expression very noisy and there are strong evidence on the genetic basis on these noises in literature of genetic regulatory networks (Swain et al. (2002): Bar-Hven al. (206)). Details of the experimental setup are referred to the Appendix.

---

Under review as a conference paper at ICLR 2021
To achieve high performance these continuous control tasks, further incorporate the maximum entropy method (Shi et al. 2019 Haarnoja al. 2018 Zhao et al. 2019) in the proposed framework. By introducing the entropy arizaton,ation, the policy is encouraged explore more and less easy to early convergence to suboptimal solutions. To have fair comparison, only SAC is included as the baseline in these examples, given to its superior performance on continuous control tusk:Haarnojj al. 2018), while REINFORCE is excluded due its poor performance. Implementation details of the a algorithm referred to the Appendix.
In Figure 3 and the state trajectories of the systems shown in time-domain. It is observed that even though the systems are highly nonlinear and stochastic due to the noises, L-REINFORCE still able to stabilize the tracking error to zero in the mean. In comparison, although SAC succeeded in stabilization in some of the trials, Figure and but its success appears be very random and can be hardly guaranteed.
8 CONCLUSION
In this paper, we proposed campling-basee approach for stability analysis of nonlinear stochastic systems modeled by the Markov decision process model-free manner. Instead of verifying energy decreasing point-wisely on the state space, we proposed stability theorem where only one sampling-based inequality is to be checked. Furthermore, we showed that with finite number of trajectories of finite length, is possible to guarantee stability with a certain probability and the proposed model-free learning algorithm to learn the
probabilistic bound is derived. Finally, controller with a stability guarantee and revealed its connection to REINFORCE. REINFORCE is not the tate-of-the-are art RI. algorithm for complicated continuous tasks. In the future, an important direction is to extend the theoretical analysis to more efficient algorithms than REINFORCE.
Spood
faroand Spooed
(a) HalfCheetah
(b) L-REINFORCE
(c) SAC
Figure 3: State trajectories of the agents trained by L-REINFORCE (b) and SAC (c). The X-axis denotes the time / and the Y-axis denotes the forward speed of the robot. The task is to control the robot to run forward at the reference speed 1m/s.
Froshn
(a) GRN with additive noise
(b) L-REINFORCE
(c) SAC
Figure 4: State trajectories of the agents trained by L-REINFORCE (b) and SAC (c). (a) shows the uncontrolled dynamic of the GRN with additive uniform noises. The X-axis denotes the time and the Y-axis denotes the concentration of each component. The task is to control the concentration of Protein to track reference signal, which is sine signal.
REFERENCES
Vikash Kumar, Abhishek Gupta, Emanuel Todorov, and Sergey Levine. Learning dexterous manipu- lation policies from experience and imitation. arXiv preprint arXiv: /6/ 1.05095, 2016.
10

---

Under review as a conference paper at ICLR 2021
Prpoak seforente
Prosudt Reforence
(a) GRN with multiplicative noise
(b) L-REINFORCE
(c) SAC
Figure 5: State trajectories of the agents trained by L-REINFORCE (b) and SAC (c). (a) shows the uncontrolled dynamic of the GRN with multiplicative uniform noises, The X-axis denotes the time t and the Y-axis denotes the concentration of each component. The task is control the concentration of Protein to track a reference signal, which is a sine signal.
Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonathan Hurst, and Michiel de Panne. Iterative reinforcement learning based design of dynamic locomotion skills for cassie. arXiv preprint wrXiv:/903.00553 2019.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, x(26))eaau5872, 2019.
Karl Johan Aström and Bjöm Wittenmark. On self tuning regulators. Automatica, 9(2):185-19, 1973.
Manfred Morari and Evanghelos Zafiriou. Robust process control. Morari, 1989.
Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199, Prentice hall Englewood Cliffs, NJ, 1991.
Leslie Pack Kaelbling, Michael I. Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.
Dimitri Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion (in Russian). PhD Dissertation, Univ. Kharkov, 1892.
Yu Jiang and Zhong- Shoon-PPngg Jiang. Computational adaptive optimal control for continuous time linear systems with completely unknown dynamics. Automatica, 48(10):2699-22 2200499997774,, 2012.
Frank Lewis, Draguna Vrabie, and Vassilis Syrmos. Optimal control. John Wiley & Sons, 2012.
EK Boukas and ZK Liu. Robust stability and Ho control of discrete-time jump linear systems with time-delay: LMI approach. In Decision and Control, 2000. Proceedings of the 39th IEEE Conference on, volume 2, pages 1527-1532. IEEE, 2000.
Ronald J Williams. Simple statistical gradient-followin algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256 1992.
Navid Noroozi, Paknoosh Karimaghace. Fatemeh Safaci, and Hamed Javadi. Generation of Lyapunov functions by neural networks. In Proceedings of the World Congress on Engineering, volume 2008, 2008.
Danil V Prokhorov. A Lyapunov machine for stability analysis of nonlinear systems. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) volume 2, pages 1028-1031. IEEE, 1994.
Gursel Serpen. Empirical approximation for Lyapunov functions with artificial neural nets. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 735-740. IEFF, 2005.
11

---

Under review as a conference paper at ICLR 2021
Danil V Prokhorov and Lee A Feldkamp. Application of svm Lyapunov function approxima- tion. In IJCNN'99, International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339), volume 1, pages 383-387. IEEE, 1999.
Theodore J Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning, Journal of Machine Learning Research, 3(Dec):803 832, 2002.
Vassilios Petridis and Stavros Petridis. Construction of neural network based Цyарunov functions. In The 2006 IEEE International Joint Conference on Neural Network Proceedings, pages 5059-5065. IEEE, 2006.
Spencer M Richards, Felix Berkenkamp, and Andreas Krause. The Lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, pages 466-476, 2018.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems, pages 908-918, 2017.
Marco Gallieri, Seyed Sina Mirrazavi Salchian, Nihat Engin Toklu, Alessio Quaglino, Jonathan Masci, Jan Koutník, and Faustino Gomez. Safe interactive model-based learning. arXiv preprint arXiv:1911.06556, 2019.
Lucian Buşoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control, 2018.
Daniel Gorges.
Relations between model predictive control and reinforcement learning. IFAC- PapersOnLine, 50(1): (0(1))))ie,.50099999999 4928, 2017.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman and Mohammad Ghavamzadeh. A Lyapunov- based approach to safe reinforcement learning. arXiv preprint arXiv:/8855007703, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar Duenez- Guzman. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.1003/ 2019.
Romain Postoyan, Lucian Buşoniu, Dragan Nešić, and Jamal Daafouz. Stability analysis of discrete- time
optimal control with discounted cost. IEEE Transactions on Automatic Control, 62(6): 2017.
John J Murray, Chadwick J Cox, and Richard E Saeks. The adaptive dynamic programming theorem. In Stability and control of dynamical systems with applications, pages 379-394. Springer, 2003.
Murad Ahulaf and Frank L Lewis. Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network hjb approach. Automatica, 41(5):779-791 2005.
Yu Jiang and Zhong- Ping Jiang. Global adaptive dynamic programming for continuous-time nonlinear systems. IEEE Transactions on Automatic Control, 60(1 1):2917-2929, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-577 533, 2015.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0297 2015.
SN Balakrishnan, Jie Ding, and Frank I. Lewis. Issues on stability of adp feedback controllers for dynamical systems. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4):91-991, 2008
Peter Shih, B Kaul, Sarangapani Jagannathan, and Drallmeier. Near optimal output- feedubck control of nonlinear discrete-time systems in nonstrict feedback form with application to engines. In 2007 International Joint Conference on Neural Networks, pages 396-401. IEEE, 2007.
12

---

Under review as a conference paper at ICLR 2021
Patryk Deptula, Joel A Rosenfeld, Rushikesh Kamalapurkar, and Warren E Dixon. Approximate dynamic programming: Combining regional and local state following approximations. IEEE Iransactions on neural networks and learning systems, (9(6)215442166 2018.
David Q Mayne and Hannah Michalska. Receding horizon control of nonlinear systems. IEEE Transactions on automatic control, 35(7):814-824, 1990.
Hanna Michalska and David Q Mayne. Robust receding horizon control of constrained nonlinear systems. IEEE Transactions automatic control, 38(1 I): 1623-16333, 1993.
David Q Mayne, James B Rawlings, Christopher V Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789-8 2000.
Chris J Ostafew, Angela P Schoellig, and Timothy D Barfoot. Learning-base basedd nonlinear model predictive control to improve vision-based mobile robot path-tracking in challenging outdoor environments. In 2014 IÉEE International Conference on Rubotics and Automation (ICRA), pages 4029-4036. IEEE, 2014.
Anil Aswani, Humberto Gonzalez, S Shankar Sastry, and Claire Tomlin. Provably safe and robust learning-basse model predictive control. Automatica, 49(5): 1216-1266, 2013.
Anil Aswani, Neal Master, Jay Taneja, David Culler, and Claire Tomlin. Reducing transient and steady state electricity consumption in hvac using Searning-basee model -ree---red control. Proceedings of the IEEE, 100(1):240-2)) 2011.
Patrick Bouffard, Anil Aswani, and Claire Tomlin. Learning-baat based model predictive control on quadrotor: Onboard implementation and experimental results. In 2012 IEEE International Conference on Robotics and Automation, pages 279-284. IEFE, 2012.
Stefano Di Cairano, Daniele Bernardini, Alberto Bemporad, and Ilya Kolmanovsky. Stochastic mpc with learning for driver-predictive vehicle control and its application hev energy management. IEEE Transactions on Control Systems Technology, 2013):001-1 2013.
Ruxandra Valentina Bobiti. Sampling- driven stability domains computation and predictive con- trol of constrained nonlinear systems. PhD thesis, PhD thesis, 2017. URL https://pure. tue. nl/ws/flees78455403. ...., 2017.
Ruxandra Bobiti and Mircea Lazar. Automated-sampling-aae stability verification and doa es- timation for nonlinear systems. IEEE Transactions Automatic Control, (5((1))3659-3777 2018.
Shaikhet. Necessary and sufficient conditions of asymptotic mean square stability for stochastic linear difference equations. Applied Mathematics Letters, 10(3):11-115, 1997.
Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability. Springer Science & Business Media, 2012.
Richard Sutton, Hamid R Maei, and Csaba Szepesvári. A convergent o(n) temporal-difference algorithm for off-policy learning with linear function approximation. In Advances in neural information processing systems, pages 1609-1616, 2009.
Nathaniel Korda and Prashanth La. On (0) with function approximation: Concentration bounds and a centered variant with exponential convergence. In International Conference Machine Learning, pages 226-634, 2015.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. arXiv preprint arXiv:/8066 02450, 2018.
Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. In Advances in Neural Information Processing Systems, pages 8665-8675, 2019.
Ole Peters. The ergodicity problem in economics. Nature Physics, 15(12)1121--1221, 2019.
D. V Anosov. Ergodic properties of geodesic flows on closed riemannian manifolds of negative curvature. In Dokl Akad Nauk Sssr, 2010.
13

---

Under review as a conference paper at ICLR 2021
Sun Woo Park. An introduction to dynamical billiards, 2014.
Francisco S Melo, Sean P Meyn, and M Isabel Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th international conference on Machine learning. pages 664 2008.
2008.
David A Levin and Yuval Peres. Markov chains and mixing times, volume 107. American Mathemat- ical Soc., 2017.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation, 2018.
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George J. Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks, 2019.
Dongmian Zou, Radu Balan, and Maneesh Singh. On lipschitz bounds of general convolutional neural networks. IEEE Transactions on Information Theory, 6638) 1759, Mar 2020, ISSN 1557-9654. doi: 10. 20199999.21992911111 URL http: doi www.orrggl0.. 1ttp://..11111111111 1109/TIT 2019. 2961122
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2-3):209-232 2002.
Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L Littman. Pac model- free reinforcement learning. In Proceedings of the 23rd international conference Machine learning, pages 881 881-888, 2006.
Chi Jin, Zeyuan Allen Sebastien Bubeck, and Michael Jordan. Is q-learning provably efficient? In Advances in Neural Information Processing Systems, pages 4863-4873, 2018.
Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606:00600000155550 2016.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/200500 2018.
Peter S Swain, Michael B Elowitz, and Eric D Siggia. Intrinsic and extrinsic contributions to stochasticity in gene expression. Proceedings of the National Academy of Sciences, 99(20): 12795-12800, 2002.
Arren Bar-Even, Johan Paulsson, Narendra Maheshri, Miri Carmi, Erin O' O'Shaa Yitzhak Pilpel, and Naama Barkai. Noise in protein expression scales with natural protein abundance. Nature genetics, 38(6):636-64) 2006.
Wenjie Shi, Shiji Song, and Cheng Wu. Soft policy gradient method for maximum entropy deep reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3425-3431. AAAI Press, 2019.
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum mntropy-regularizee multi-goal goal reinforcement learning. International Conference on Machine Learning, pages 7553-7562, 2019.
Michael Elowitz and Stanislas Leibler. A synthetic oscillatory network of transcriptional regulators. Nature, 403(6767):335, 2000.
Natalja Strelkowa and Mauricio Barahona. Switchable genetic oscillator operating in quasi-stable mode. Journal of The Royal Society Interface, 7(48): 2010.
Aivar Sootla, Natalja Strelkowa, Damien Ernst, Mauricio Barahona. and Guy-Bart Stan. On periodic reference tracking using batch-mode reinforcement learning with application to gene regulatory network control. In 52nd IEEE conference on decision and control, pages 4096-4091. IEFE, 2013.
14


Reviews:
[{"review_id": "thMXefYeSP4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**General overview:** The paper studies guaranteeing the closed-loop stability of a Markov decision process (MDP) using a given a policy, based on a finite number of trajectories each containing finite number of steps. Both the state and the action spaces are assumed to be subsets of finite dimensional Euclidean spaces. Particularly, the concept of mean square stability (MSS) is used which is guaranteed based on the properties of certain Lyapunov functions. Theoretical results on probabilistic MSS guarantees are provided based on finite samples. A variant of the standard REINFORCE policy gradient method is also presented which searches for a policy having MSS guarantees. Finally, numerical experiments on a simulated cart-pole problem comparing the suggested L-REINFORCE method with the soft actor-critic (SAC) off-policy RL algorithm are shown.\n\n**Pros:**\n- Though, the concept of stability is central to classical control theory, it is often neglected in the RL literature. The paper studies such a stability in continuous state and action spaces. \n- Having a stochastic guarantee based on a finite sample (of finite length trajectories) is a practically motivated problem. \n- The paper presents a Lyapunov function based approach and corresponding theorems to have a probabilistic guarantee on MSS. \n- A policy gradient type method is introduced, as well, which searches for a policy that has the MSS property with high probability.\n- Some numerical experiments are also presented comparing the results to a recent off-policy RL method.\n\n**Cons:**\n- Although, the general objective of the paper (namely, to have finite sample stochastic stability guarantees in RL) is nice and should be further studied, the paper in its current form seems premature.\n- The used assumptions, for example, the ergodicity of the induced Markov chains and Assumption 2, are strong and could be hard to check in practice. For example, how should one verify Assumption 2 for a given problem? The paper should present some specific examples to show how should the conditions be checked in practice.\n- The theorems, like Theorem 2, themselves are not surprising and seem like relatively simple consequences of standard concentration inequalities, such as the Hoeffding inequality.\n- Also, a practically more relevant form of this result, namely to give a lower bound on M (number of trajectoris) and T (number of steps), given a confidence level, which guarantees MSS with that confidence, should also be included.\n- It would be more elegant to use only one long trajectory, instead of assuming that we can generate arbitrary number of trajectories to test the stability of the system.\n- The L-REINFORCE algorithm only searches for a stable controller, but it does not care about its optimality. The authors mention that an optimal controller might not be stable. Nevertheless, it would make much more sense finding the optimal stable controller, namely, the best controller (for example, with respect to the total expected reward criterion) which is also stable, in the MSS sense.\n- In the light of the previos remark, it is questionable what do the experimental results demonstrate, as the SAC method of Haarnoja et al. aims at optimizing an expeted total reward criterion, while the suggested L-REINFORCE method aims at finding a stable controller (apples vs oranges). \n- The paper claims that REINFORCE is a special case of L-REINFORCE. This claim is a bit strange, as the authors use a very specific choice of cost function c(s), defined on page 3 after Q2, while in the classical REINFORCE method the immediate-cost function can be arbitrary.\n- Some assumptions are important, but not emphasized. For example, the Markov chain induced by the policy is ergodic with a unique stationary distribution (see page 3) should be highlighted and labeled, similarly to Assumptions 1 and 2.", "extra_scores": {}, "extra_text": {}}, {"review_id": "T9mttw-BB4e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides insight and results for probabilistic stability certification using a finite set of trajectories and a Lyapunov function based on a neural network plus a norm prior function. Then, a link to REINFORCE is estabilished to learn policies that are stabilising (once optimality is reached and the RL is finished). \n\nPros: The theoretical results seem correct. The link to REINFORCE is nice. The small example illustrates the point of stability. \n\nCons: \n\n0) Some relevant citations are missing:\n\n`Bobiti, R. V. (2017). Sampling–driven stability domains computation and predictive control of constrained\nnonlinear systems. Technische Universiteit Eindhoven.`\n\nIn this thesis, a probabilistic stability result is formulated based on Lyapunov functions. Part of this work was used by some of your references (the deterministic bit) and the stochastic part was used in another missing citation:\n\n`Marco Gallieri and Seyed Sina Mirrazavi Salehian and Nihat Engin Toklu and Alessio Quaglino and Jonathan Masci and Jan Koutník and Faustino Gomez (2019). Safe Interactive Model-Based Learning. In NeurIPS 2019 workshop on Safety and Robustness in Decision-Making`\n\nThese works are very closely related to the present paper and should be cited. It would be important to see where your results stand with respect to these related works. In particular, your sample bound based on the log probability seems almost the same as the one derived in Chapter 5 of Bobiti (2017). What is the advantage of using your bound? Is it possible to estimate the level sets as well as the Lyapunov function for local stability as done in Gallieri et al (2019)? Your function is also quite close to the one used in Gallieri et al (2019) since you have a prior norm term and you make sure it is positive definite by construction which was not done in other related work on Lyapunov networks. What are the limit of these approaches for learning and verification when the system is stochastic and how does your approach overcome them if it does? \n \n1) The assumptions are not very clear and seem strong at times. Do you assume the origin to be an \"equilibrium point\" (in expectation at least). This is never stated. The definition of Lyapunov function in the introduction is wrong, you need the derivative to be negative definite in order to converge to the origin. You have that in the later definition for your special case but this is also true in the general case. Isn't it a bit too strong in Assumption 2 to ask for a single $\\gamma<1$ for all values of T (all possible sequence lenghts)? Does this mean that there is effectively a contration in the system or that probability distributions converge quickly to the limit distribution? How does that fit with the claim that the system under analysis is \"highly stochastic\" (made in the introduction)? \n\n2) It is claimed that the system to deal with is \"highly stochastic\". Then the approach is tested on a cartpole, which is deterministic. \nCould you provide an example of a system that is \"highly stochastic\"?  I think it would then make sense to try your approach also on such a system and not just a cartpole. \n\n3) Does your result hold for any distribution of the initial states? You have a bound on the probability mass but it seems like the distribution does not affect your results in any ways? \n\n4) Can you scale the approach to a more complex example then the cartpole? It seems rather simple and deterministic. Your technique seems like an overkill for the cartpole. That can be solved by a linear deterministic controller.\n Since you compare to SAC and use REINFORCE, you should have all the machinery in place to address an example that cannot be controlled via LQR, right? There shouldn't be any further coding required... \n\nNotation is a bit hard to follow at times, in particular when discussing closed-loop and open loop distributions the same symbols are used. There are also a few typos that should be checked: Lemma 1 \"denote\", above Lemma 3 \"affect\". \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4V0sQCTAPXT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Summary\n\nThe paper describes a finite sample approximation for estimating stability of\ncontrol policy. They propose to use stability in mean squared setting with\nfinite sample errors on estimating Lyapunov function. The experimental results\nare thin and based only on balancing a cart-pole. \n\n### Pros\n\n1. Take aways from the paper are intuitively reasonable. Any sampling based\n   method would suffer from low samples M or high variance due to lower T in\n   sequence based estimates.\n\n2. Authors in related works highlight the importance of missing research in\n   stability analysis in recent advances in RL. THe points covered in related\n   work are valid and relevant\n\n### Cons/Questions\n\n1. I disagree that Lyapunov analysis is always impossible. Finding a Lyapunov\n   candidate function is very challenging task, but it doesn't not mean there\n   isn't a function. Impossible --> implies there is no such function.\n2. One typically addresses the infinite horizon requirements with discounting.\n   One can obtain an error bound for horizon truncation see [Section III Ref 1.] \n3. From the cart pole results which a fairly linear system around the vertical\n   stabilising point, the numbers for M and T look very large. Authors have not\n   mentioned sampling rate for the cart-pole system, to judge what exactly the\n   number T implies. \n\n### Language\n\n1. The paper in general would benefit from a language review. This does not\n   factor in the review decision, however, authors may prefer to ease readers\n   burden by getting the work reviewed for language alone.\n\n\n### Ref\n\n1. D. Ernst, M. Glavic, F. Capitanescu and L. Wehenkel, \"Reinforcement Learning\n   Versus Model Predictive Control: A Comparison on a Power System Problem,\" in\n   IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics),\n   vol. 39, no. 2, pp. 517-529, April 2009, doi: 10.1109/TSMCB.2008.2007630.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3aOyupweJB9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the probabilistic stability guarantee of control systems. In general, hard stability guarantee is difficult with only finite samples. The authors instead focus on developing probabilistic stability conditions. High probability bound is derived in terms of the number of trajectories and the length of them. This also leads to a practical policy gradient style algorithm, which is applied to the Cartpole task with desired performance.\n\nOverall, I think the approach makes sense. With only finite sample, the best one can hope for, without further assumptions, is to understand the probabilistic behavior. The method seems intuitive and the writing is generally clear. It is interesting to see the connection of the resulting algorithm and the classical REINFORCE algorithm. This seems to suggest that \"proper\" policy gradient methods can lead to stability. It would be interesting to further understand if other policy gradient style methods can be also made to exhibit such property.\n\nWhile the paper contains one experiment, I think it is not enough to fully support the overall theoretical results. Since the authors already have the code available, it should be fairly easy to include more tasks with varying difficulties. In addition, since the motivation is to solve complex task, it might be interesting to understand how the overall approach scales when the dimension increases. Overall, having only one experiment makes me wonder whether the results are cherry-picked and the parameters are heavily tuned.\n\n-----------------------------\nPost rebuttal:\nI thank the authors for addressing my comments and updating the manuscript. I plan to keep my positive score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the reviewers like the general idea of the paper, there are concerns regarding the clarity of the statements, especially in stating the main assumptions, referring to related work, and how well the experiments support the results of the paper. Although the authors' long response addressed some of the issues/comments raised by the reviewers, not all of them are convinced that the paper carries enough novelty and is ready for publication. I would suggest that the authors take the reviewers' comments into account, revise the paper, and make it ready to be submitted to an upcoming conference. 
Decision: Reject