A Policy Optimization Method Towards Optimal-time Stability
Authors: Shengjie Wang
Lan Fengb
Xiang Zheng
Yuxue Cao
Oluwatosin OluwaPelumi Oseni
Haotian Xu
Tao Zhang
Yang Gao
Venue: CoRL 2023 Poster (2023)
Keywords: Reinforcement Learning
Robotic Control
Stability

Abstract: In current model-free reinforcement learning (RL) algorithms, stability criteria based on sampling methods are commonly utilized to guide policy optimization. However, these criteria only guarantee the infinite-time convergence of the system's state to an equilibrium point, which leads to sub-optimality of the policy. In this paper, we propose a policy optimization technique incorporating sampling-based Lyapunov stability. Our approach enables the system's state to reach an equilibrium point within an optimal time and maintain stability thereafter, referred to as "\textit{optimal-time stability}". To achieve this, we integrate the optimization method into the Actor-Critic framework, resulting in the development of the Adaptive Lyapunov-based Actor-Critic (ALAC) algorithm. Through evaluations conducted on ten robotic tasks, our approach outperforms previous studies significantly, effectively guiding the system to generate stable patterns.

A Policy Optimization Method Towards Optimal-time Stability
Shengjie Wang!. 1,2.3
Fengbo Lan' Haotian Xu
Xiang Zheng'
Yuxue Cao
Oluwatosin Oseni
Tao Zhang"
Yang Gao 12,3,
Tsinghua University SShnngaai Artificial Intelligence Laboratory CCity University of Hong Kong
"Shanghai Qi Zhi Institute
Covenant University
f Corresponding author
Abstract: In current model-free reinforcement learning (RL) algorithms, stability criteria based on sampling methods are commonly utilized to guide policy opti- mization. However, these criteria only guarantee the infinite-time convergence of the system's state to an equilibrium point, which leads to sub-optimality of the policy. In this paper, we propose a policy optimization technique incorporating sampling- based Lyapunov stability. Our approach enables the system's 's state reach an equilibrium point within an optimal time and maintain stability there- after, referred to as "optimal-time stability". To achieve this, we integrate the optimization method into the Actor-Critic framework, resulting in the development Crttr-Critic (ALAC) algorithm. Through evalua-
of the Adaptive Lyapunov-based tions conducted on ten robotic tasks, our approach outperforms previous studies significantly, effectively guiding the system to generate stable patterns.
Keywords: Reinforcement Learning, Robotic Control, Stability
Introduction
Model-free reinforcement learning (RL) con- trollers have achieved excellent performance in a large variety of robotic tasks [1. 2]. How- ever, current methods lack a stability guarantee, which poses additional risks both the robots and their environments, especially in the pres- ence of external disturbances [3]. Therefore, ensuring stability in RL-based methods is a cru- cial requirement.
Optimal trajectory wi stabilly
Unstable trajectory
State space
S (5)
In -ontrol-theoretic methods, there exists an effective tool, Lyapunov functions, assess the stability [4]. Recently, researchers have employed neural networks to search for fea- sible Lyapunov functions [5, 6, 7, 8]. No- tably, model-free methods have integrated pol- icy and Lyapunov function learning based on discrete sampling-basee based stability, demonstrat- ing promising results in robotic control tasks (9, 10, 11, 12, 13] (See Appendix for the dis- cussion of related works). Current research pro- poses different conditions to speed up the search of a feasible I Lyapunov function [9, 10, 11], but these methods only guarantee the eventual conver- gence of the system's 's state to an equilibrium point. Figure illustrates a limitation of this guarantee,
Sub-optimal trajectory w/ stability
Diverging trajectory
Figure I: Intuitive example showing the optimality and stability. The stability condition is the mean cost
stability defined in Definition 2.1. The figure shows the relationship between three sets, like the whole state
space, {s (x) < b) and (s c (s) = 0}. The red line represents a diverging trajectory. The yellow line represents trajectory without stability. The trajectories coloured cyan and purple remain stable, whereas the state in the purple trajectory reaches the set {s cx cx(s) 0) more quickly.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

where two current methods, despite the purple trajectory exhibiting better stability convergence efficiency. In our experiments, we find that this loose guarantee fails to provide adequate guidance for policy learning. Inspired by finite-time stability in continuous-time control theory [14], we propose an improved objective where the state converges to the equilibrium point within an optimal time. which we refer to as "optimal-tine" -imee stability" in this paper. This optimal condition facilitates rapid convergence the equilibrium point while ensuring stability in the system. Our objective is to develop policy capable of generating the purple trajectory depicted in Figure 1.
state trajectories (colored eyan and purple) cannot be differentiated by
To ensure optimal-time stability in the system, we introduce a sampling-bar Lyapunov stability certification, which guarantees the fulfillment of the mean cost stability condition. Subsequently, we design policy optimization method that facilitates the gradual convergence of the policy towards the optimal point, where the system achieves optimal-time stability. By leveraging this policy optimization technique, we develop the Adaptive Lyapunov-based Actor-Critic algorithm (ALAC) for learning both a Lyapunov function and policy. Experimental results validate the effectiveness of our approach in ensuring optimality and stability across various robotic control tasks, including legged robot walking and free-floating space robot trajectory planning. Specifically, our method significantly outperforms baselines ten robotic control environments, and the trained I Lyapunov function provides effective guidance for policy learning compared previous methods'.
Problem Formulation
robotic system can be modelled as a Markov Decision Process (MDP). MDP mainly consists of five elements, S, A, P. C and p. Here, S is the state space, A is the action space, P the dynamic transition function, and C. is the cost function. Besides, the distribution of starting state denotes so p. At timestep t, S, E S represents the state the robot observes. Then, a, E A is the action executed by the agent (robot). Note that a, is sampled from the agent's policy (aa]ss)). According to P(s (st oz), the state of system transfers to the next state st+1 with a certain probability. At the same time, the agent receives the cost C(se, 0z). And then, we can define the state distribution T:
T(slp.m, + = / / x(a ((s as) da T(slp, n. t) ds
Note that T(slp, n, 0) = p holds. An MDP system corresponds to a continuous and discrete- time dynamical system with state space S and action space A. Generally speaking, the system's stability can be verified by a classical tool, Lyapunov's stability function (Appendix B.1). However, the classical definition should be satisfied in the whole state space S, it is unsuitable for sampling optimization, especially for model-fres RL methods. To extend the stability reasonable case in freel-free RL, we introduce the Mean Cost Stability as the stability condition in this paper.
Definition 2.1 (Mean Cost Stability [9]). A robotic system remains stable in mean cost when satisfying the following equation, where cx (8x) Ea-aC(st, C(s,, ax) and b is constant.
lim B cr (8c) 0, Cz (5p) b 80000
Concretely, for most robotic tasks, the mean cost stability is related to the stability of the closed-loop system [9]. Our aim is to reach an equilibrium point within an optimal time and behave stably around it, to achieve the optimal-time ttiee stability. Thus, we construct the problem formulation represented as follows:
8
min P.
c (st)
s.t. lim BA,TCC = 100
1=0
Specifically, the constraint part can facilitate the system's 's state to converge to an cquilibrium point. The objective of the task to minimize the sum of discounted costs. When the sum of discounted
For more information, please visit our project page at http:://stes.ooggl..co/mvvvv//a Iyapunov- actor-critic.
2

---

costs becomes smaller, the system converges to an equilibrium point more quickly. Therefore, by minimizing the sum of discounted costs, we ensure that the system's 's state converges to an equilibrium point within an optimal time.
3 Policy Optimization with Sampling-based Stability
Considering the difficulty of calculating the stability condition in problem (3), we present novel stability theorem in sampling-based format suitable for the Reinforcement Learning (RL) setting. Additionally, we transform the original problem (3) into a new optimization problem shown as Equation 8, which progressively seeks the optimal convergence time while ensuring stability.
3.1 Sampling-based Stability Certification
Our approach reveals that ensuring the stability condition in Equation (3) corresponds to finding the Lyapunov function L. The I. Lyapunov function has long been used for stability analysis, and more details can be found in Appendix B.I. Unlike classical methods that rely on the entire state space, we present certification for sampling-basee based Lyapunov stability.
Theorem 3.1 (Sampling-basee Lyapunov Stability). An MDP system is stable with regard to the mean cost as shown Definition 2.1, if there exists a function S + I. meets the following conditions:
ac*(s) S ((s) S (cx
((s) (s) + XE L(s')
(5)
Egget [R.. [()) S k: R (B(8) A [(s')]
(6)
where a, 3. and positive constants. Among them, U, lim T Et= T(s P. n, represents the stationary distribution of the state, and Pa (s'ls) SA ( s,) da represents the stationary distribution of state transition. It is based Appendix C.I. For proof. please see Appendix C.2.
mild assumptions as shown in
In practice, the theorem reveals that the guarantee of stability in Equation (3) corresponds to the existence of the Lyapunov function L under the constraints given by Equations (4), (5), and (6). Intuitively, Equation (4) represents the lower and upper bounds of [(s). Equation (5) shows the relationship of C at two steps under the transition distribution. Equation (6) shows the constraint on the expectation of the state stationary distribution. It is worth noting that our method extends the more general case, illustrated in Appendix C.3. To mitigate the difficulty of searching for L under multiple constraints, meets the constraints (4) and (5), given by L. (s) E, E 7 (st) so s). Please Appendix B.2 for a detailed demonstration. In summary, the stability condition can be expressed as finding feasible policy т, under which the Lyapunov candidate C satisfies the constraint in Equation (6).
previous approach
propose Lyapunov candidate L (s), which naturally
3.2 Policy Optimization Towards Optimal-time Stability
Recalling the optimization problem (3), we observe that the constraint part is equivalent to Equation (6) since the introduction of the Lyapunov candidate (s) eliminates the constraints (4) and (5). Meanwhile, the objective part can be represented as ming Ee (s) based on the definition of the Lyapunov candidate. Therefore, the optimization problem can be rewritten as:
min (x)pC((s)
S.I. Ex-it [EgN LT'(') (s) < - (Cx (Sr Cx
potential approach tackle the problem (7) is formulate unified objective function that removes the constraints. However, directly combining the objective with constraints through simple addition may lead to sub-optimal policy an invalid Lyapunov function. Therefore, propose policy
3

---

Igspumov Learning
Eyngunes
Vef-)
TD-Ske Leerning
4y 4)
4x(0)
Replay
Eygusso
Lagrangian bhased Pollcy Learning
Optinal Polcc with Meaz Cost Stability
Optinication Problen
Policy
Iralia (y.
Lazrange Muttipleer
Maapite Farters
Figure 2: Architecture illustrating the practical implementation ALAC. The RL, agent is optimized by Actor-Critic framework. The main contents consist of two parts. For the Lyapunov critic learning, the constrained Lyapunov network is updated by the TD-like learning with respect to the Lyapunov candidate. During the policy Learning, we the Lagrangian-based method to solve the optimization problem shown in Equation (8). Meanwhile, the parameters À and l: are adjusted by the Lagrange multiplier X adaptively.
optimization method that progressively seeks the optimal policy while ensuring stability by making the constraints' parameters À and & learnable. The method outlined as follows.
max 1u (cr 2,K
s.t. π E (T|EESe, DCr (8) 50}
(8)
where
ACF (Bx))) Lo - +|C=(()) -
(9)
Intuitively, the constraint part corresponds the constraints defined in Equation (7). The objective seeks find optimal values for À and k to maximize Ez-ut AL, (s). Alternatively, we can view this objective enhancing the strength of the constraints by maximizing Eu-ur (s) By gradually improving the constraints, minimizing (s) can be achieved in Equation (7). Because, as the constraints become strict, the policy should ensure that C (s') at the next state becomes smaller, as dictated by the constraint part in Equation (8). Considering the sampling distribution Un, C, becomes simultaneously smaller at every state along the state trajectories. This also applies to E, under the initial state distribution p. When the strength of the constraints reaches maximum value, the minimization of Ca (s) is achieved. In addition, we offer comprehensive illustration to explain the decrease of (s') using a classical tracking task, which is provided in Appendix C.5. Interestingly, the form of Equation (9) bears resemblance to the finite-time tracking method in continuous-time systems (Appendix C.4).
Adaptive Lyapunov-based Actor-Critic Algorithm
In this section, we propose the Adaptive Lyapunov-based Actor-Critic algorithm (ALAC) to solve the optimization problem (8). According to the variables shown in Equation (8), we introduce two learnable parameters X and kt along with two neural networks, namely the Lyapunov critic Ca a) and the actor (ala))) The Lyapunov critic Lo(s, a) serves dual role in both stability analysis and actor learning, and it depends Lo,s, a) (s) to compute (s) in Equation (8). Furthermore, the actor (a(s)) maps a given state 8 to a distribution over actions. This action distribution is modeled as a Gaussian distribution with state-dependent mean (*(()) and diagonal covariance matrix (x).
both the state and the action We leverage the relationship
Figure shows the architecture of the ALAC algorithm. Specifically, we employ the Lagrangian- based method to handle the solution of the policy under the constraint in (8). Additionally, the parameters À and & are adjusted by the Lagrange multiplier x to satisfy the objective in (8).

---

4.1 Lyapunov Critic Learning
For the training of Lo(s, a), we typically update based on the TD error in an actor-critic learning process. In our framework, the target Lyapunov value can be approximated as ea + (s' (-1s/)) Thus, the update rule for 0 can be written as:
0 0 0g + oxo (V(Co(ss (cz + 76', (((ss()))))
(10)
where the number of iterations. L' and n' are the target networks parameterized by O' and o', respectively. To ensure stable learning. the parameters /' and ø' are typically updated using an exponentially moving average of weights, controlled by hyper- hypee--aramettr encourage accurate and efficient learning, we construct constrained critic network. The constrained network ensures that the output is non-negative and the Lyapunov value should be zero when the state is an equilibrium point. For the details, please see Appendix C.6.
(0, 1). In order to
4.2 Lagrangian-bast based Policy Learning
Policy learning aims search feasible parameters of X, k and the policy network to solve the optimization problem defined in Equation (8). According Equation (9) and Lyapunov critic, denote ALr, (s) as Co(s', TT( s')) Co(s. a) + I(Co(ss,) a) ALo (s', TT(- s'))]-
First, we solve the sub-problem of Equation (8), finding Ta when satisfying the constraint with arbitrary A. Applying the Lagrangian-barn -asaad method [15], the parameters of T are updated by:
Φk+1 Φk + as aol.iV V
(11)
where Ogg is the learning rate of Φ. A represents the Lagrange multiplier of the constraint. During the training, D is updated by gradient ascent to maximize AL (s, a).
x-+1= = A + AC,
(12)
Note that A should always be positive. axx is the learning rate. It is worth noting that x is clipped by and to bound the value, which is exploited in optimization methods [16].
According to Equation (8), need to find suitable values for À and k to maximize AL., as. We observe that À should decrease towards 0 to enlarge AL=, (s, a). Furthermore, it should be between and O, with the range of À restricted based on Theorem 3.1. Enlarging ALT, (s. corresponds to strengthening the constraints in Equation (8). Notably, we find that the Lagrange multiplier A decreases as the constraints are satisfied. Thus, we use the rule À + miinAr,3 7) to gradually improve the constraints, As for the selection of k. we adjust its value based on the Lagrange multiplier, setting k <- A By increasing k, the strength of the constraints gradually improves throughout the training process. Finally, when À and l stabilize, indicating that the Lagrange multiplier \ remains constant, we can observe that ALr, (s, a) approaches O according to Equation (12). This implies that the objective of maximizing AL (s, a) is achieved since the maximum value of ALr, (s, a) does not exceed 0 due to the constraint part in Equation (8).
In addition, to improve the exploration efficiency, we add a constraint about the minimum entropy as the same as the maximum entropy RL algorithms. Until now, we have designed the complete ALAC algorithm, and the pseudo-code is provided in Appendix D.
4.3 Theoretical Analysis
Theorem 3.1 has assumed that the expectation is obtained perfectly, but this is not the case in practical settings due to finite samplings. Thus, we derive the bias between the practical computing and theoretical analysis about U,x
To be concrete, we need an infinite number of trajectories with infinite time steps to estimate the distribution Uz Whereas in practice, only M trajectories of T time steps are accessible. To
better illustrate the issue, we define a finite sampling distribution Ut apparently where UL
$ Ei= T(s p, t). First, we provide a quantitative bound of expectation from U, and Ut
5

---

Table 1: Performance evaluations of the cultivated costs and stability constraint violations ten environments compared with six baselines. All quantities are provided in a scale 0.1. Standard errors are provided in brackets. (if the mean constraint violations are less than 0.2, the sign is | else indicates the algorithm does not contain the stability constraints.)
Metrics
ALAAC
SAC-enst
SPPO
LAC
LAC
POLYC
L.PPO
TNL.
Carppoee-cost
Cost Return
26,2nm
22.7,0
102,3s4n
31.0aa
31.550
104,801
205.3
33.5
Violation
4
Point-circle cost
Cost Return Violation
111.as
111.8a
247.9Gm
958.6155
112,0sm
207.0
122.
145.8:ao
Cost Return Violation
1.7m
16.6mm
144.0
119.5om
188s
168,8
37.8
6.5a.
Halfchectah-cost
Swimmer-cost
Return Violation
44.6.n
$3.7,0
52.5
47.5k
44.8
104.7
52.30
46.50
Cost Return Violation
101.0e
155.2cm
255.0
166.9m
125.6m2
259 8mm
114.6zen
186.8:00
Ant-cost
Humanoid-cost
Cost Return Violation
354.6mm
441.9m
531.8a29
431.30
368.3
490
452.4mm
317.7
Minitaur-uss
Coat Return Violation
493.0umz
692.2
950.0mm
612.2oo
666.6096
608. 3iss
838.30x
382.9
Spacereacb-oost
Cost Return Violation
16b2
8,9
19,4a
35.2
1,80
125.7
31.0mm
112, Joue
Cost Return Violation
2.3am
38.4
53.2m37
33.9
2,8
112,8cko
35.8
85.9
Spacerandom-ood
Spacedualarm cool
Crst Return Violation
26.1
36, 1ac
201. 9um
66.3rmm
63,6
140.6ma
37.8n
280
Theorem 4.1, Suppose that the length of sampling trajectories is T. then the bound can be expressed as:
Ba-u, (L Eggue AL (+)] < (k 1
(13)
1-7
where Eg is the maximum of cost and q is a constant in (0, 1). For proof. please see Appendix C.7.
Next, we take the number of trajectories into consideration and derive the probabilistic bound of the difference of AL, (s) estimated by UL, distribution and M trajectories.
Theorem 4.2. Suppose that the length of sampling trajectories is T and the number of trajectories is M, then there exists the following upper bound:
1 M. T
Ma²(1 -1 71 - -BNnE AC)(s) 2 a) S 2expl - (A1= + (k 1)2)
PO
Σ (s)
MT
TIL
(14)
where $7 represents the state in the m-th trajectory at the time t. For proof. please see Appendix C.8.
Theorems 4.1 and 4.2 highlight the theoretical gap between infinite and finite samples in practical usage. In addition, they provide valuable insights into the choice of k. Theorem 4.1 suggests that when i approaches 0, the gap becomes small in practice. On the other hand, Theorem 4.2 indicates that le is better to be to 1. This implies that the optimal choice of k lies within the range of 0 1.
5 Experiments
In this section, we demonstrate empirical evidence that ALAC captures improved trade-off off between optimality (sum of costs) and stability compared to the baseline approaches. We test our method and baselines in ten robotic control environments. Details of ten environments are given in Appendix E. Furthermore, we benchmark the ALAC method against five algorithms with neural Lyapunov function, including POLYC [10], LBPO[17]. TNLF[11], SPPO[18] and LAC [9] 2 We also take the SAC-cost [19] method into account because the method is very close to method without the stability condition. For the detailed hyper-parameter settings see Appendix E.2. and E.2.2
We find LAC with large ag (see Appendix 1.2.1) performs better, so we call it LAC" for the distinction between them.

---

Caripole-cost
HaliChectab-cost
Minitaur-cost
Spacerandom-cost
good
grals
Steps
Line
Values
Visualization of states for ALAC method by t-SNE and phase trajectory techniques. The top row of the figure depicts the t-SNE dimension reduction technique. (Cartpole- is visualized within 2 dims while others within 3 dims.) The bottom row shows the phase trajectories and Lyapunov-value surfaces. vp and & denotes the angular position and velocity respectively.
Figure 3:
Munar scheert
Mistarcaed
Figure 4:
Ablation studies of the sampling base stability we propose. ALAC(original) shows comparable or the best performance compared with other certifications on each task.
5.1 Comparing with Baselines
In this part, we evaluate the optimality and stability of our methods and baselines. According to the definition of optimal-time stability in Eq. (3), we use the accumulated cost the metric of optimality and the stability constraint violations as the stability metric in a testing episode (further details can be found in Appendix E.3). The results demonstrate that ALAC achieves the lowest accumulated cost and stability violations. To fairly evaluate the performance of the methods mentioned above, we run the experiments over rollouts and seeds for all algorithms. Table shows the performance on all tasks, and the training curves for different algorithms are in Appendíx E.3. Although LAC" using tighter constraints achieves comparable performance with method in contrast to LAC, the stability violations in LAC" remain at high level on many tasks. Admittedly, TNLF achieves lower cost than ALAC on Minitaur-cost, but TNLF converges to suboptimal policies on many tasks. According to Figure 10, we notice that in TNLF the trained Lyapunov function is close to 0 quickly. Hence, it does not provide dense guidance for the policy, thus leading it to a suboptimal solution.
5.2 Ablation Studies
To demonstrate the effectiveness of AC,, S O, we do the ablation studies about different constraints in ALAC. We compare the performance of the original ALAC with a version that uses AC! (A = 0) and A (A = 1), and with version where & is a constant throughout training, The descriptions of ablation algorithms are given in Appendix E.4. Figure 4 and Figure 9 (see Appendix D.4) depict the accumulated cost and constraint violations on all tasks. ALAC(A achieve lower performance than ALAC, while ALAC(AC)) performs the tasks comparably with ALAC. Nevertheless, more
7

---

Table 2: Average evaluation score and standard deviation on environments for ALAC with and without the errors under different biases of goals. (w/ errors means using errors between the desired and achieved goals as extra elements of states for the agent)
Task
Point-cirele-csst
Halfcheetah-cnst
Spacereach-com
Biases of goals
-20%
0%
20%
-20%
0%
20%
-20%
0%
20%
ALAC w/ errors
85, 84.2
110.1
148.
3,90al
2,5iom
8,3,67)
6,4c.z)
2.4:14)
8.7
ALAC
178.8
118.9(
247.5 8.1.
10.
3,3
13.421
11,9
1.602)
11.5a0
SAC-cost w/ errors
84,2,2)
109.3a2
140
60.1225
81.6 31.6:00)
1
21.9112)
21.19
20.6
SAC-cost w/o errors
180.9
115.3
240.3
15.9us
16.7
33.5
16.1
8.8
15.0czz)
strict constraints (ALAC(AC; )) negatively affect its performance on constraint violations, as shown in Figure 4. This is because there doesn't It exist a reasonable policy that meets such strict constraints. Moreover, the results of ALAC(k = 0.1) comparing with ALAC demonstrate that the heuristic updating of k is effective during the training.
5.3 Evaluation Results
First, use the t-SNE method to indicate the visualization of the state in 3 dimensions in order to illustrate better the stability of the system learned by ALAC (Cartpole-cost in 2 dimensions). As we can see, the top row of Figure 3 shows the states in the final stage of an episode converge to a point or circle. Basically, we recognize that those patterns happen in a stable system. The second row of Figure shows the phase trajectories with variance according to the state pairs of joint angular position and velocity. In practice, experts can judge system's stability from a phase space of the system. Concretely, y and & represent an angular position and velocity, respectively. The angular velocity starts from 0 to O, and the angular position starts from the beginning to an equilibrium point. Based on the above phenomenons, it suggests the trained systems using the ALAC method satisfy focal stability or stable limit cycles. Furthermore, the Lyapunov value exhibits significant changes in the state space, as depicted in Figure (bottom row). indicates that the Lyapunov value can effectively guide the policy towards discovering stable patterns in the system. More implementation details for t-SNE and phase trajectories are given in Appendix E.5.
Analysis of Visualization:
Furthermore, some experiments verify that the policy can generalize well to follow previously unseen desired goals, such as different desired velocities in Halfcheetah-cost. When evaluating, we add -20% bias to the desired goals in three environments, as illustrated in Table 2. The definition of desired and achieved goals is shown in Appendix E.6.2. Specifically, when we introduce the error between the desired and achieved goals as additional information the state, ALAC w/ errors gains remarkable performance improvement on generalization compared to SAC-cost w/ errors. This indicates that the I Lyapunov function successfully captures the relationship between error and stability. Even though the desired goal experiences some shifting, the Lyapunov function can guide the policy to reach the goal. Finally, we also add external disturbances with different magnitudes in ten environments and observe the performance difference. Figure 8 (see Appendix 1. shows that in all scenarios, ALAC enjoys superior performance over other methods.
Generalization:
6 Discussion
Conclusion. We propose sampling-basee based Lyapunov stability condition to meet the mean cost stability. Based on the condition, the policy optimization with sampling based stability is proposed to gradually find the optimal policy which maintains the optimal-tine -ime stability we propose. Based on the Actor-Critic framework and Lagrangian-based optimization, We present a practical algorithm, namely the Adaptive Lyapunov-based Actor-Critic algorithm (ALAC).
Limitation. Despite the great success in simulated tasks, our method has not been evaluated in practical scenarios. An important direction for future work is to apply our method real-world robotic control tasks, such as locomotion and navigation. While this work primarily focuses on providing stability guarantees, it is essential to consider another critical requirement, safety.
8

---

Acknowledgments
This work is supported by the Ministry of Science and Technology of the People 's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), This work is also supported by the National Key R&D Program of China (2022ZD0161700).
References
[1] Hwangbo, Lee, A. Dosovitskiy, D. Bellicoso, Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, x(26))caau877, 2019.
[2] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The Interna- tional Journal of Robotics Research, 39(1):3-20, 2020.
[3] W. Jin, 7. Wang, Z. Yang, and S. Mou. Neural certificates for safe control policies. arXiv preprint arXiv:2006.08460 2020.
[4] La Salle and S. Lefichetz. Stability by Liapunov's Direct Method with Applications by Joseph L Salle and Solomon Lefschetz. Elsevier, 2012.
[5] H. Dai, B. Landry, L. Yang. M. Pavone, and R. Tedrake. Lyapunov-stable neural-network control. arXiv preprint arXiv:2109.14152 2021.
[6] S. Lale, Y. Shi, G. Qu, K. Arizzadeneshell, A. Wierman, and A. Anandkumar. Kerl: Krasovskii- constrained reinforcement learning with guaranteed stability in nonlinear dynamical systems. arXiv preprint arXiv:2206.00 20744, 2022.
[7] N. Gaby, F. Zhang, and X. Ye. Lyapunov-net: A deep neural network architecture for Iyapunov function approximation. arXiv preprint arXiv:2109.13: 13359, 2021.
[8] R. Zhou, Т. Quartz, H. De Sterek, and J. Liu. Neural дyарunov control of unknown nonlinear systems with stability guarantees. arXiv preprint arXiv:2206.01913, 2022.
[9] M. Han, L. Zhang, Wang, and W. Pan. Actor-critic reinforcement learning for control with stability guarantee. IEEE Robotics and Automation Letters, 5(4):6217-6224, 2020.
[10] Y.-C. Chang and S. Gao. Stabilizing neural control using self-learned almost Iyapunov critics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1803-1809, IEEE, 2021.
[11] Z. Xiong, J. Eappen, A. H. Qureshi, and S. Jagannathan. Model-free neural Iyapunov control for safe robot navigation. arXiv preprint arXiv:2203.01190. 2022.
[12] Zhao, Wang. Lu, and Du. Neural łyapunov control for power system transient stability: deep learning-basee based approach. IEEE Transactions Power Systems, 37(2)955-966, 2021.
[13] M. Han, Y. Tian, L. Zhang, J. Wang, and W. Pan. Reinforcement learning control of constrained dynamic systems with uniformly ultimate boundedness stability guarantee. Automatica, 129: 109689, 2021.
[14] P. Bhat and D. S. Bernstein. Finite-time stability of continuous autonomous systems. SIAM Journal on Control and optimization, 38(3):751-766, 2000.
[15] A. Stooke, Achiam, and P. Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference Machine Learning, pages 9133-9143. PMLR, 2020.

---

[16] B. Peng, Y. Mu, J. Duan, Y. Guan, S. E. Li, and J. Chen. Separated proportional-intege lagrangian for chance constrained reinforcement learning. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 193-199. IEEE, 2021.
[17] H. Sikchi, W. Zhou, and D. Held. Lyapunov barrier policy optimization. arXiv preprint arXiv:2103.09230, 2021.
[18] Y. Chow, O. Nachum, A. Faust, E. Bnenez-Guzman, and M. Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprini arXiv: 2019.
[19] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[20] S. M. Richards, F. Berkenkamp, and A. Krause. The Iyaрunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, pages 4666477..7 PMLR, 2018.
[21] Y.-C. Chang, N. Roohi, and S. Gao. Neural Iyapunov control. Advances in neural information processing systems, 32, 2019.
[22] M. Mittal, M. Gallieri, Quaglino, S. M. Salchian, and J. Koutník. Neural lyapunov model predictive control. 2020.
[23] H. Dai, B. Landry, M. Pavone, and R. Tedrake. Counter-example guided synthesis of neural network Iyapunov functions for piecewise linear systems. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1274 1274-1281. IEEE, 2020.
[24] M. Lechner, D. Zikelić, K. Chatterjee, and T. A. Henzinger. Stability verification in stochastic control systems via neural network supermartingales. arXiv preprint 1eXiv:2112.0949, 2021.
[25] P. L. Donti, M. Roderick, M. Fazlyab, and J. Z. Kolter. Enforcing robust control guarantees within neural network policies. In International Conference Learning Representations, 2020.
[26] K. Kashima, Yoshiuchi, and Y. Kawano. Learning stabilizable deep dynamics models. arXiv preprint 2rXiv:2203.097100 10, 2022.
[27] S. Chen, M. Fazlyab, M. Morari, G. for nonlinear systems. 2021 60th IEEE Conference on Decision and Control (CDC), pages 6477-6484 6484. IEEE, 2021.
J. Pappas, and V. M. Preciado. Learning region of attraction
[28] N. Lawrence, P. Loewen, M. Forbes, J. Backstrom, and B. Gopaluni. Almost surely stable deep dynamics. Advances in Neural Information Processing Systems, 13:12922-1895, 2020.
[29] A. Schlaginhaufen, P. Wenk, A. Krause, and F. Dorfler. Leamning stable deep dynamics models for partially observed delayed dynamical systems. Advances in Neural Information Processing Systems, 34:11870-11882, 2021.
[30] Berkenkamp. M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
[31] C. Dawson, Z. Qin, S. Gao, and C. Fan. Safe nonlinear control using robust neural Iyapunov- barrier functions. In Conference on Robot Learning, pages 1724-12 1735. PMLR, 2022.
[32] Y. Chow, O. Nachum, E. Ouenez-Guzmant and M. Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
[33] M. Han, Z. Zhou, L. Zhang, J. Wang, and W. Pan. Reinforcement learning for control with probabilistic stability guarantee. 2020.
10

---

[34] T. Huang, S. Gao, X. Long, and L. Xie. A neural Iyарunov approach to transient stability assessment in interconnected microgrids. In HICSS, pages 1-10, 2021.
[35] X. Li, J. Xiao, Y. Cheng, and H. Liu. An actor-critic learning framework based on Iyapunov stability for automatic assembly. Applied Intelligence, pages 1-12, 2022.
[36] I.. Zhang, R. Zhang. T. Wu, Weng, M. Han, and Y. Zhao. Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles. IEEE Transactions on Neural Networks and Learning Systems, 32(12):5435-5444 2021.
[37] R. M. Murray, Z. Li, and S. Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017.
[38] S. Zou, T. Xu, and Y. Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems, 32, 2019.
[39] S. Wang, Y. Cao, X. Zheng, and T. Zhang. Collision-free trajectory planning for 6-dof free- floating space robot via hierarchical decoupling optimization. IEEE Robotics and Automation Letters, 7(2):4953-4960, 2022.
[40] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 2016.
[41] Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. International conference on machine learning, pages 22-31. PMLR, 2017.
[42] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[43] Schulman, S. Levine, Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learming, pages 1889-1897. PMLR, 2015.
[44] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, arXiv preprint arXiv: (00.006777..3444 2017.

---

A Related Work
Learning- baaning--assedde controllers have achieved excellent performance in non-linear dynamic systems [1. 2]. However, lack of stability introduces additional risks to the agents and environments [3]. Fortunately, there exists an effective tool, Lyapunov functions, assess the stability. Lyapunov functions can be designed for linear system with specific criteria in the form of quadratic positive-definite function. But how to find suitable Lyapunov function remains open challenge in the non-linear dynamic system [4].
A.1 Model-based RL Lyapunov Learning
Due the difficulty of manual design, constructing a Lyapunov neural network has become increasingly popular in non-linear dynamic system. For the model-known situation, the approaches jointly learn Lyapunov function and controller [20, 21, 22, 23, 24, 25, 7]. But it restricts the application of complex systems which are hard to obtain accurate models. Therefore, some researchers present model-learned methods with a stability guarantee, in which Gaussian Process or Neural Network approximates the model. The model-learned method can be separated into two types. The first one learning dynamics models guided by learnable Lyapunov function, in which policies are inherently included or learned by LQR method [26, 27, 28, 29]. Another approach learnable policy network updated by neural Lyapunov function, thereby satisfying the stability of system [30, 31, 8, 5, 6]. However, notice that most model-learned methods are only verified in relatively easy environments. A possible reason is that the coupling of the Lyapunov function and dynamic model makes learning unstable or incompatible due to interdependency.
construct
A.2 Model-free RL & Lyapunov Learning
A promising direction is study model-free methods with a stability guarantee. Recently, large variety of methods have been proposed to address the issue. One method is that the policy is updated by mixed objective with respect to the neural Lyapunov function and Q function. POLYC [10] introduced the necessary conditions required by the Lyapunov function into objectives to optimize the policy network. LBPO [17] applied the logarithmic barrier function based on the form of the Lyapunov function. TNLF [11] constructed Lyapunov and Q functions trained by the stability certification. The other form is policy optimization with Lyapunov constraints. Chow al. [32] designed constrained RL algorithm project policy in trust region with Lyapunov stability. In those previous study, there still exist main drawback. The discrete Lyapunov condition they used did not meet the demand for sampling based stability guarantee in RL. Therefore, Han et al. [9] considered sampling- bbsed stability condition they proposed as a constraint and then used the prime-dual method to modify the constraint. Their latter work verified them in both on-policy and off off-policy settings [13, 33). Nevertheless, their method can only find an existing policy with the demand for stability. In contrast, our method can satisfy the sampling- based stability and search for the optimal policy adaptively. The optimal policy can facilitate the state to reach the equilibrium point rapidly.
A.3 Applications
Furthermore, worth noting that current RL-based methods with stability guarantee have been applied in some practical problems successfully, such as monitoring the security of interconnected microgrids [34], power system control [12], automatic assembly [35] and motion planning of au- tonomous vehicles [36].
12

---

B Preliminary Remarks
B.1 Lyapunov function
The method we employ constructing or finding a Lyapunov function, denoted as : + RR with the property that its difference along the state trajectory negative definite. This ensures that the state moves in direction that decreases the value of the Lyapunov function, eventually leading to convergence at the origin. The Lyapunov method has well-established history of application in stability analysis and controller design within control theory.
Definition B.1 (Equilibrium Point). A state is equilibrium point if 3 action de A such that [(S., de) [37]
Definition B.2 (Stabilizable in the sense of Lyapunov). A system is stabilizable if Ve 0, 38 such that for all S such that |s00- sell S 6, there exists ( such that the resulting {st ) satisfies ||s sell S vt 2 0.[37]
Definition B.3 (Lyapunov Function). A continuous and radially unbounded function : is Lyapunov function if the following conditions hold:
1. vs E S, 3a E A. s.t. ((s) 2 (+)
2. vs 0, L 0; (()) 0.
If Lyapunov function exists, discrete-time system achieve stability in the sense of Lyapunov without considering the physical energy.
B.2 Lyapunov Candidate Bound
In this part, we show that the Lyapunov candidate L (s) meets the property in Theorem 3.1, which can be formulated as:
Ca (s) + AB s C(s) S Bcx (s)
(15)
where omit the lower bound ac,, (s) which is naturally satisfied by L. (s).
Firstly, according to the definition of (s),,, we have
8
(x) Ex
Σ
Y (x)(00) = =s]
[x(c((80) +
7 (80)
(16)
= =c((s) + ET (x
oc
cx(()) + TEA..o'
(n:)(s)) =
1=0
= cx(ss + -.SPD. L(8')
Considering the left-hand side Equation (4), we can find that A S y holds, the lower bound of the Lyapunov function can be satisfied. This because the Lyapunov candidate L, (s) is positive each state. Furthermore, the right- hand side of Equation 4 illustrates the higher bound of the Lyapunov function exists. The condition is also guaranteed for Lyapunov candidate shown in the following
13

---

process.
(x(8) E,
7 (s))sss
1=
(st))o0
(17)
1=0
1-7
Note that & denotes the maximum cost. The second row of the inequality holds due to Jensen
inequality. Only if the maximum cost exists, 33 E 14+, 1 S (cx(s) holds.
C Details of Theoretical Analysis
C.1 Assumptions of Theorem 3.1
Assumption C.1 (Region of Attraction). There exists a positive constant such that p(s) > 0, Vs {s (s) < 6}. Assumption C.2 (Ergodic Property). The Markov Chain driven by the policy is ergodic, wx (s) lim;-x (( p, t).
The first one ensures that the starting state is sampled in the region of attraction. The second one is the existence of the stationary state distribution.
C.2 Proof of Theorem 3.1
Theorem C.3 (Sampling-based Lyapunov Stability). An MDP system is sluble with regard to the mean cost, if there exists function L S + R meets the following conditions:
ac>(s) S L(s) < (cz(s)
(18)
((s) 2 ca (s) + XE C(s')
(19)
Eywut BBw,,, L(8)) L(s)] < Ek [[(s) - AB, L(s')]
(20)
where a, 3, and positive constants. Among them, Ut limy T ET T(s
p. a,
represents the stationary distribution of the state, and Pa (5'18) SA (()))) ( da represents the stationary distribution of state transition.
Proof. Firstly, we simplify the left side of the Equation (20) with reference [9]. Introducing the definition of Un (s) leads
Eanile Ev'p, Cx(s') Lx(ss)
(21)
Σ T(s p. t) (s)))) (s) T-0
lim
T
Due to the boundedness of C. we apply the Lebesgue's Dominated convergence theorem. To be specific, when (s)] Bss), Vs E S, Vn holds, have
/ lim Fs(s)ds
lim ==
F.(ssdss
(22)
noc
14

---

Hence, we get
Ex [Bs' La((') -xr)))
T 5 70000
lim Σ
T(s I Lz
T
t=0
lim
T(s T(s [(slp, p,
Pa(s')s) Cx (23)
5 T
t=0
lim
E 11
Σ (T
T T
1
lim
(1) O) Cx(s))
T T T
Note that Tslpp n, t- = 0) is equal to p. Since the expectation of C (s) is a finite value, the left side of Equation (20) zero.
Now, we turn to the right side of Equation (20). According to the Equation (23), we have
- -k|E, [C(s) - (EL 2 Eu (R(s) - ABseap, [(s')] <
(24)
Since [(s) 2 (s) + AENNP,L(s") holds, we get
Eg Gr (8) 50
(25)
Based on the Abelian theorem, we know there exists
(s)s) lim Σ p,
T
(26)
lim (ss)p,
(s)
Thus, we get
Eswwe 1c (8)] 0
(27)
The last row of inequality holds because of Equation (26). Based on the definition of wn (s), we have
lim ET (x( < 100
(28)
Suppose that there exists a starting state 50 E (50 (50) < b} and a positive constant d such that lim C (s) co. Consider that p(sg) > for all starting states in {so c, (so) S b) (Assumption C.I), then lim C 0, which contradictory with Equation (28). Thus Vsp (so (x()) S b), lime C= 0.
lim, Ca
Thus the system meets the mean cost stability by Definition 2.
C,3 Comparison to Existing Methods
Intuitively, the previous method is a special case when L(s) = Cs (s) + ALE L(s') holds [9], as indicated in the following equation.
Eg-2t [LE''=P, L(s') L(s)] < - k[EE (C(s) AB C(s')]
Our methou
(29)
Han al. 2020 [9]
15

---

That means we extend the previous method to a more general case. To be specific, the introduction of enlarges the solution space of the policy. Thus, it facilitates the policy to find the optimal point while maintaining the system's stability.
C.4 Fimite-Time Feedback Tracking Method
Lemma C.4 (Fimite-Time Feedback Tracking Method), In a continuous-time system, trajectory (()) tracks the reference R((). (L)) can truck the reference within finite time T, such that R(t) = W (t), 2 T. if the following conditions holds.
V.W()) S (() - R())), VI [0,T]
(30)
Note that the gradient of R(t) is bounded, meaning that V R, S μ holds.
Proof. First, build the mean square error (tt) between them.
= 2 (w (t) R(t))²
(31)
Then, we can derive the difference of (t) as follows
V.V (W R)(VWW R.
< (W R)(-k(W -R) R.
(32)
< kkWW R| (W - R
Introducing the Assumption that the bounded gradient of R(t), we have
W
R
W R
V.V < 2k
24
2
2
(33)
2kV V2FVV
Observe that the above formulation belongs to a form of the Bernoulli differential equation. In this case, we can reduce the Bernoulli equation to linear differential equation by substituting 2 = VV. Then, the general solution for z is
V2p + -ki 2 k
== VV
(34)
Applying the initial condition V(t = 0) = Utg we have
/
C = /uto +
(35)
2 k
Finally, the convergence time T can be represented as:
2
TTo
T
In
+ to
(36)
k
22
k
C.5 Illustration of the Feedback Tracking
First, we denote (tt) as (s),, W(t + 1) as (s') and R(t) as (s)), where we omit the expection operator for simplicity. Specifically, at time t + 1. the value of W 1) should decrease by (t) - R(t)). The change of À and & results (( - R(t)) increases correspondingly. Consequently, ( + 1) needs decrease further to meet the requirement. Recalling the definition. L, (s') become smaller. Additionally, the form of Equation (9) is similar finite-time tracking method in continuous-tin time system which we depcit in Appendix C.4.
16

---

C.6 Constrained Lyapunov Critic Network
Concretely, we denote the output of a neural network as f(s, And then, Lo(8, a) can be described
by:
Co(s, (G.((G
(37)
where Gg is a linear transformation, which guarantees Gæ=e, (f(s = se: a)) = 0 (se is an equilibrium point defined in Definition B.1.). Note that G, contains no parameters be learned, so the operator does cause harm the representation ability of the neural network.
Concretely, the output of the neural network of the Lyapunov critic is described by:
fs,, a) ho (hoo--i--... (....22hhi
(2 (< (< >)))
(38)
where each (2)2) has the same form:
(z) (W. (W. oz + b,)
(39)
Here, O represents the number of layers, and the is the non-linear activation function used in the o-th layer. Furthermore, { (Wo, bo} is the weight and bias of the o-th layer.
First of all, to meet the demand of Lo( a) = O, we introduce a linear transformation G., one of whose possible forms can be
f
f
f. f. f.
f
f
(f)
882
ser
(40)
Σ 58g te
fi
f
where I denotes the number of elements of the state, and e is the number of units of the output layer. is constant close to 0 to avoid singularity. Note that ds
which indicates the difference between the current state and an equilibrium point. As we can see, when each element of 8s is zero, the multiplication of matrices is zero. Thus, G.=a, (f(s = Se; a)) = holds. Furthermore, it brings another benefit having no impact on the training networks.
C.7 Proof of Theorem 4.1
Theorem C.5. Suppose that the length of sampling trajectories is T, then the bound be expressed
Ea-u, ALr(s) Enwuz (s), 2 (k + 1) T
(41)
1-7
where is a constant in (0, 1).
Proof. First, we can get the following equation by introducing the definitions of Ur and Ut
Exntl ALx(s) - Exnuy (*
1 (U+(s) T
T(s 1,,1 (s)
(42)
t=1
Σ
((<(s)) T(s p, (tC
t=1
Then, eliminating the integral operator, we obtain
E Egyue AL (s) Ex-uz AC (s)
T
(43)
Un (s) T(s) a, (oT.)))ilIAC.. (+)]
Thus, the next step is to get the bounds of [i (s) T(s T. t) and |A
17

---

For the first part, we introduce the assumption that first is mentioned in [38], shown as follows:
T
Σ (s) - T(s p, S 27" AA E 2+, 3g E (0,
(44)
Frankly speaking, the assumption easily satisfied because the I.1 distance between two distributions bounded by At the same time, ((s p. n, t) converges to 2,, (s) with time approaching.
For the second part, we can get the bound of (8) according to Equation 17.
(
Ey--,, Lx(6)) - Cr (8) + k(Cx (8) - AE (s'))
CT 0 k(
CT
(45)
0)
1-7
--7
Then, we have
C
( S +
(46)
Adding results in Equation 47, we finally get
(k+ 1)0x 52
Egnet ACr(s) Enwuy (Cr <2
T
(47)
1-7
C.8 Proof of Theorem 4.2
Theorem C.6. Suppose that the length of sampling trajectories is T and the number of trajectories is M, then there exists the following upper bound:
M
T
P(
(s - E, Egwery (sr a)
MT
m=
t=1
(48)
Ma² - 7)
2 expl
- + (k 1)
where st" represents the state in the m-th trajectory at the timestep
Proof. First, eliminating AC (s) by Equation 9, we rewrites the left side of Equation 48 as
M
P(
AC (s7)) BaNuy (s)
MT
m=1 t=1
M
T
= IP(
(Cx (8++1) - Cx((e) + (Cx((8)) - >Lx((1+1)) Eener ALr 2 a))
MT
m
T
IP MT
Σ Σ
(11
kA) + (k(())((ss)) EaNye (s)
m=lta t=1
(49)
M Here Eynur (s) is expected value of NT EM ΣL As+() In addition, the bounds of (1 () and (k (s,) can be obtained easily by Equation 17. Thus, obtain the Theorem 4.2 by applying Hoeffding's inequality.
2M2a²
5 < exp(
M((1 - kA)² + (k: - 1)2) (T-)
(50)
Ma? (1 7)22
< <2exp(-
((1 k2)2 + (Ir - 1)
18

---

D Details of Algorithms
As mentioned in the main text, we introduce minimum entropy as a constraint in policy optimization and apply the primal-dual method update the policy and the Lagrange multiplier Åe. To be specific, the constraint can be expressed
log TT(aas) S - where 2g the minimum value of policy entropy, usually, Z. corresponds to the dimension of action space the environment.
(51)
Algorithm 1: Adaptive Lyapunov-based Actor-Critic Algorithm (ALAC)
Orthogonal initialize the parameters actor and critic networks with ø, 0 Initialize replay buffer D) and λı de, À and k Initialize the parameters of target network with ø' t Φ and t for episode
1, M do Sample an initial state 30 for step 0, T. do
Sample an action from (a(a)))))
Execute the action de and observe state 81+1
Store < St, end for for iteration 1, N do
S141
into D
Sample minibatch B from the replay buffer
Update 0 according to (10) using minibatch B
Update 6, A₁, Ae according to (1) (1) using minibatch
Update adaptive factors and k
Update the parameters target networks, 0'
end for end for
E
Details of Experiments
Cartpole-cost
Swimmer-cost
Halfchectah-cost
Ant-cost
Humanoid-cost
Point-cirele-cost
Minitaur-cost
Spacereach-cost
Spacerandom-cost
Spacedualarm-coost
Figure 5: Overview of environments.
We test our method and baselines in ten robotic control environments, including Cartpole- cost, Portt--irrccle--ott Halfcheetah-cost. Swimmer-cost, Ant-cost, Humanoid-cost. Minitaur-cost. Spacereach-cost, Spacerandom-cost and Spacedualarm--cost Most tasks environments goal-oriented, tracking target position speed, which corresponds to most control tasks. Further- more, the latter four environments involve models of practical robots like quadruped robot and
19

---

robotic arm, making them relatively more difficult. It is worth noting that the task of Spacedualarm cost is trajectory planning of a free-floating dual-arm space rohot. The coupling property of the base and the robotic arms brings hardship for both traditional control and RL-based methods [39].
E.1 Environmental Design
Cartpole-cost This task aims to maintain the pole vertically at a target position. The environment is inherited from [33]. The state and action space are the same as the default settings in OpenAI
2
Gym[40], so we omit the description. The cost function c
+ 20
where
Tihresbold 10 and dreshold 20 The other settings can be found in Table 3.
Point-circle-cost This task aims to allow sphere to track a circular trajectory. The environment is inherited from [41]. The sphere is initialized at the original point. The cost function is represented as c = d, where denotes the distance between the current position and the reference. The other settings can be found in Table 3.
Table 3: Hyper-paane parpee-prameterss of non-linear dynamic environments
Sepper-paramters
Cartpole-cost
Point-cirele-cost
State shape
4
7
Action shape
2
2
Length of an episode
250 steps
65 steps
Maximum steps
3e5 steps
3e5 steps
Actor network
(64, 64)
(64, 64)
Critic network
(64, 64, 16)
(64, 64, 16)
Halfcheetah-cost The goal of this task to make a HalfCheetah (a 2-legged simulated robot) to track the desired velocity. The environment inherited from [33]. The state and action space are the same as the default settings OpenAI Gym[40], so we omit the description. The cost function is - 1)?, where represents the desired velocity. The other settings can be found in Table 4.
Swimmer-cost This task aims to make multi-joint snake robot to track the desired velocity. The environment is inherited from [33]. The state and action space are the same as the default settings in OpenAl Gym[40], so we omit the description. The cost function is c = (v 1 where 1 represents the desired velocity. The other settings can be found in Table 4.
Ant-cost This task aims make an Ant (a quadrupedal simulated robot) track the desired velocity. The environment is inherited from [40]. The state and action space the same the default settings in OpenAl Gym [40], we omit the description. The cost function is c = (v 1)?, where 1 represents the desired velocity. The other settings can be found in Table 4.
Humanoid-cost This task aims to make a humanoid robot to track the desired velocity. The environment inherited from [40]. The state and action space are the same as the default settings in OpenAl Gym [40], so we omit the description. The cost function c = (v- 1)2, where I represents the desired velocity. The other settings can be found in Table 4.
Minitaur-cost This task aims to control the Ghost Robotics Minitaur quadruped to run forward at the desired velocity. The environment is inherited from [42]. The state and action space are the same as the default settings in PyBullet environment(2) [42], so we omit the description. The cost function is e = (v - 1)2, where represents the desired velocity. The other settings can be found in Table 5.
Spacereach-cost This task aims make a free-floating single-arm space robot's end-effector reach fixed goal position. Since the base satellite is uncontrolled, collisions will cause system instability once collisions occur. Therefore, it is critical to plan collision-free -free path while maintaining the
20

---

Table 4: Hyper-parameters of mujoco environments
Hyper-parameters
Swimmer-cost
Halfchectah-cost
Ant-cost
Humanoid-cost
State shape
27
376
Action shape
2
8
8
Length of an episode
250 steps
200 steps
200 steps
500 steps
Maximum steps
3e5 steps
1e6 steps
1e6 steps
1e6 steps
Actor network
(64, 64)
(64, 64)
(64, 64)
(256, 256)
Critic network
(64, 64, 16)
(256, 256, 16)
(64, 64, 16)
(256, 256, 128)
Table 5: Hyper-parameter of robotic environments
Minitaur-cost
Spacereach-cost
Spacerandom-cost
Spacedualarm-cost 54
State shape
27
18
18
Action shape
6
6
12
Length of episode
500 steps
200 steps
200 steps
200 steps 5e5 steps
Maximum steps
1e6 steps
3e5 steps
5e5 steps
Actor network
(256, 256)
(256, 256)
(256, 256)
(512, 5T2)
Critic network
(256, 256, 16)
(256, 256, 128)
(256, 256, 128)
(512, 512, 256)
stability of the base. The agent can obtain the state, including the angular positions and velocities of joints, the position of the end-effector, and the position of the reference point. Then, the agent outputs the desired velocities of joints. In low-level planning, a PD controller converts the desired velocities into torques, and then controls the manipulator. The cost function is defined as c = d, where d is the distance between the goal and end-effector. The other settings can be found in Table 5.
Spacerandom-coot This task aims make a free-floating single-arm space robot's end-effector reach a random goal position. The agent can obtain the state, including the angular positions and velocities of joints, the position of the end-effector, and the position of the reference point. Then, the agent outputs the desired velocities of joints. In low-level planning, PD controller converts the desired velocities into torques to control the manipulator. The cost function is defined = d, where d is the distance between goal and end-effector. The other settings can be found in Table 5.
This task aims to make a free-floating dual-arm space robot's end-effectors reach random goal positions. The complexity of the task increases dramatically due to two arms' coupling effects on the base. The agent can obtain the state, including the angular positions and velocities of joints, the positions of end-effectors, and the position of target points of two manipulators. Then, the agent outputs the desired velocities of joints. In low-level planning, PD controller converts the desired velocities into torques to control the manipulators. The cost function is defined follows: = do + d1, where de is the distance between goal and end-effector of Arm-i. The other settings can be found in Table 5.
Spacedualarm-cost
E.2 Implementation Details
E.2.1
Baselines
SAC-cost Soft Actor-Critic (SAC) is an off-policy maximum entropy actor-critic algorithm [19]. The main contribution is to add maximum entropy objective into standard algorithms. The soft Q and V functions are trained to minimize the soft Bellman residual, and the policy can be learned by directly minimizing the expected KL-divergence. The only difference between SAC and SAC-cost replacing maximizing reward function with minimizing cost function. The haper-parameters of SAC-cost is illustrated in Table 6.
SPPO Safe proximal policy optimization (SPPO) is Lyapunov-based safe policy optimization algorithm. The neural Lyapunov network is constructed to prevent unsafe behaviors. Actually, the
21

---

Table 6: Hyper-parameters of SAC-cost
Hyper-parameters
SAC-cost
Learning rate of actor Learning rate of critic
I.e-4
3.e-4
Optimizer
Adam
ReplayBuffer size
106
Discount (?) Polyak (1 - 7) Entropy coefficient
0.995
0.995
1
Batch size
256
safe projection method is inspired by the TRPO algorithm [43]. In this paper, modify it to apply the Lyapunov constraints on the MDP tasks, similar to the process in [9]. The hyper-parameters of SPPO illustrated in Table
Table 7: Hyper-parameter of SPPO
Hyper-paree pereer-prrameters
SPPO
Learning rate of actor
1.e-4
Learning rate of Lyapunov
3.e-4
Optimizer
Adam
Discount (7)
0.995
GAE parameter (A)
0.95
Clipping range
0.2
KL constraint (6)
0.2
Fisher estimation fraction
0.1
Conjugate gradient steps
10
Conjugate gradient damping
0.1
Backtracking steps
10
Timesteps per iteration
2000
LAC Lyapunov-based CritiCCrii)L algorithm is an actor-critic RL-based algorithm jointly learning a neural controller and Lyapunov function [9]. Particularly, they propose a data-driven stability condition on the expected value over the state space. Moreover, they have found that the method achieves high generalization and robustness. The hyper-parametent of LAC is illustrated in Table 8, Among them, 0g is 0.1 in LAC, while it is changed as 1 in LAC".
Table 8: Hyperparameters of LAC
Hyperparameters
LAC
Learning rate actor
I.c4
Learning rate of Lyapunov
3.e-4
Learning rate of Larange multiplier
3.e-4
Optimizer ReplayBuffer size Discount (7)
Adam
106
0.995
Polyak (1 r)
0.995
Parameter of Lyapunov constraint (og)
0.1
Batch size
256
POLYC Policy Optimization with Self-Learned Almost Lyapunov Critics (POLYC) algorithm is built the standard PPO algorithm [44]. Introducing a Lyapunov function without access the cost allows the agent to self-learn the Lyapunov critic function by minimizing the Lyapunov risk. The hyper-parameters of POLYC is illustrated in Table 9.
22

---

Table 9: Hyper-parameters of POLYC
POLYC
Learning rate actor
I.c-4
Learning rate of critic
3.e-4
Learning rate of Lyapunov
3.
Optimizer
Adam
Discount (n)
0.995
GAE parameter (A)
0.95
Weight of Lyapunov constraint (3)
0.1
Clipping range
0.2
Timesteps per iteration
2000
Lyapunov Barrier Policy Optimization (LBPO) algorithm [17] is built on SPPO algorithm [18]. However, the core improvement uses Lyapunov-based barrier function to restrict the policy update to safe set for each training iteration. Compared with the SPPO algorithm, the method avoids backtracking to ensure safety. For the implementation our paper, the process is similar to that of SPPO algorithm. The hyperparameters LBPO is illustrated in Table 10.
LBPO
Table 10: Hyperparameters of LBPO
Hyperparameters
LBPO
Learning rate of actor
I.c-4
Learning rate of critic
Le-4
Learning rate of Lyapunov Optimizer
3.e-4
Adam
Discount (t))
0.99
GAE parameter (A)
0.97
Clipping range
0.2
KI. constraint
0.012
Fisher estimation fraction
0.1
Conjugate gradient steps
Conjugate gradient damping
0.1
Backtracking steps
Weight of Lyapunov constraint (3)
0.01
Timesteps per iteration
2000
TNLF Twin Neural Lyapunov Function (TNLF) algorithm is proposed to deal with safe robot navigation in [11]. Different from other approaches, the TNLF method defines a Lyapunov V function and Lyapunov Q function, which are trained by minimizing the Lyapunov risk. In effect, the Lyapunov risk is similar that of [10]. Since the Lyapunov function strietly decreases over time, the robot starting with any state in Region of Attraction (RoA) will always stay in the RoA in the future. It should be pointed out that
environments only support the cost function, the objective, except for Lyapunov risk, is to minimize the cumulative return of cost. The hyper-parameterd of TNLF is illustrated in Table 11.
E.2.2
Our method
Our method offers a significant advantage in contrast to baselines, which to use fewer hyperparameters. The main hyperparameters are illustrated Table 12. We notice that these parameters control networks' learning without including the parameters of constraints. The reason is they are automatically updated according to Lagrange multipliers, X₁, and Xe The initial value of Lagrange multipliers is set to 1, common usage previous constrained methods.
ALAC
23

---

Table 11: Hyper-parameters of TNLF
Hlyper-parameters
TNLF
Learning rate actor
1.e-4
Learning rate of critic
3.e-4
Learning rate of Lyapunov V functiob
3.e-4
Learning rate of Lyapunov functiob
3.c-4
Optimizer ReplayBuffer size
Adam
10
Discount (?)
0.995
Polyak (1-7)
0.995
Weight of Lyapunov constraint (a)
0.1
Variance of noise distribution
Batch size
256
Table 12: Hyper-parameters of ALAC
sepper-aaramttes
ALAC
Learning rate of actor
1.c-4
Learning rate of Lyapunov
3.e-4
Learning rate of Lagrange multipliers (Ar and Xe)
3.e-4
Optimizer
Adam
ReplayBuffer size
10
Discount (?)
0.995
Polyak (1 7)
0.995
Batch size
256
E.3 More Results on Comparison
Figure 10 shows the learning curves of the accumulated cost and constraint violations of ALAC and other baselines in ten environments.
Note that Eq. (3) illustrates the definition of optimal-time stability. We can observe that when the system satisfies the stability condition and the accumulated cost minimal, the system achieves optimal-time stability. Therefore, in experiments, we the accumulated cost testing episode the metric of optimality and the stability constraint violations the stability metric. As shown in Table 1, our method achieves both minimal accumulated costs and minimal stability violations compared with the baselines.
violation of stability conditions. We measure the stability of each algorithm by evaluating the violations of stability conditions in a trajectory (episode). Since different algorithms may have different requirements for Lyapunov functions, metric. However, all algorithms aim to minimize the stability constraints. When the value of these constraints is not close to 0, it indicates that the algorithm does not satisfy the designed stability conditions. In such cases, the algorithm cannot guarantee the convergence of the system's 's state to an equilibrium point.
cannot
single metric function the stability
E.4 More Results on Ablation Study
We provide the specific formulation of AC} and AC? Compared with AC,, in Equation we intuitively find that ACt and AC2 are lower and higher bound of ACr, respectively. In other words, AC1 represents the strongest constraint, while AL represents the loosest constraint. The comparison between them demonstrate that the sampling-bate based Lyapunov stability (ACn,) can search for the optimal policy with stability guarantee due to the adaptive updating of X.

---

1.component==2 or 3,
early exaggeration=
[earning_rate=20000 rate n_iter=1000,
W_iter_wwttout_pprogress =300,
min_grad_norm=le-1,
perplexity=30.
metric="euullidenn,
n_jobs=None.
random_state=42,
11
verbose=True.
init='pca'
Table 13: Other hyper-parameters t-SNE method.
ACT, (8, Co(s',Ta('10 - Lo(s, (f((s) - 0]
AC Tu (s, a) Ta(s',Ta(11
+ Co(s', n (+(sss)))
(52)
ACro (8,4) La(s', T.( 8')) - La(8, + ((Co(8, a) ALa TT( s'))]
Furthermore, we also conduct the algorithm with a constant = 0.1. The comparison demonstrates that the heuristic-based updating of an effective method. The ablation experiments on other tasks shown in Figure 9.
E.5 Details of Visualization
Our RL-based policy optimization method guided by adaptive stability is difficult to express the latent laws of states in the convergent process of different environments as the high- high-diiension states-space. To find and show the state's ' change laws in the convergent process:
We use the t-SNE dimension reduction technique to visualize the state-space.
We plot the phase trajectory with variance according to the state pairs of joint angular position and velocity.
We plot the Lyapunov-value surface and its shadow with the phase trajectory and values in the convergence process.
The top row of Figure shows the results of the t-SNE state plot- ting with SciKit-Learn bobsfiesttemm.mmniflll... TSNE function) with varying parameters(e.g, eettlyexaggeration min_ other environments with n_components=3. The hyper-parameters for t-SNE are shown in Table 13.
T-SNE Visualization
Cartpole-Cost is visualized with n components==2 while
Phase Trajectories Systems We select the angular position and velocity of a joint the state space in each environment and plot the phase trajectory with variance in Figure 6. The convergent process is shown the angular velocity starts from 0 to O, and the joint angle starts from the beginning to the convergence position.
Lyapunov Functions of Systems We visualize the change of Lyapunov-value dimensions based on the phase trajectory. The second row of Figure 3 shows the I Lyapunov-value surface. The curves of values along the phase trajectory are mapped the whole plane with down-sampled and smoothed by Gaussian filter; add the values and the phase trajectory shadows correspondingly simultaneously.
25

---

Cartpele-cost phase trajectory
Halfcheetah phase trajectory
16-
Angular Velocity
Loguia Veloco
-10
-15
0.16
8.40
Angular Position
Angular Position
Minitaur-cost phase trajectory
Spacerandom-cost phase trajectory
0.2
Angular Velocity
Angular AngularVelocin Velocity
-63
13-
-02
155
1.
14
20
2A
-2.75
-2.0
-2.25
-1.03
75
-0.50
-025
0.00
Angular Position
Angular Position
Figure 6: Phase trajectories of the systems trained by ALAC. (we report the results of 20 trials and select joint to graph the phase trajectory in each task.)
E.6
More Results on Evaluation
E.6.1
Robustness
Generally speaking, stability has potential relationship with robustness to some extent. Thus, add external disturbances with different magnitudes in each environment and observe the performance difference. Specifically, when evaluating, 50 steps. The magnitudes determine the range of disturbances. For example, magnitude equal to 0.5 means that the noise is randomly picked from [-0.5, 0.5]. Figure 8 shows that in all scenarios, ALAC outperforms other methods under different disturbance magnitudes. Furthermore, we omit the algorithms that do not converge to a reasonable solution each task.
introduce external disturbance
actions every
E.6.2 Generalization
We verify that ALAC achieves excellent generalization with the feedback of the error. First, introduce
variable in the state, which represents the error between the desired goal and the achieved goal. To be specific, the desired goal can be the desired velocity position of the agent, and the achieved goal is the current velocity position of the agent. During training, we use fixed value for the desired goal, while in evaluation, we add +-20% bias to the desired goal. The results show that ALAC generalizes well to previously unseen desired goals, as depicted in Figure 8. Furthermore, we observe that errors have a negative impact on the performance of SAC-cost. The reason could be that SAC-cost does not capture the error information without the guidance of a Lyapunov function. Notably, the number of environment steps in Halfcheetah-cost is 5e5 in this section.
26

---

Magntide
Magerads
May
Ming
Figure 7: Performance of ALAC method and other baselines under persistent disturbances with different magnitudes. (The X-axis indicates the magnitude the applied disturbance. We evaluate the trained policies for 20 trials in each setting.)
Pointzirde
HalfChectah
Spaceresch
ALAC Ai an1r snar
ALAC NE BITET
ALAG aN UrtE unir SAC-aai
wa
Coal RERtUUT
Cost Sutuurn
425
15
Blasse
Baaess
Blases
Figure 8: Evaluation of ALAC and SAC-cost methods in the presence different biases of goals. (The X-axis indicates the magnitude of the applied shifting. We evaluate the trained policies for 20 trials in each setting.)
27

---

Cartellccc
Treatto
AAC 1-111
Auda Ao
2001 221
Thino
Secorachoort
Spaceer Sorverr
auichee
Tinsdag
Figure 9: Ablation studies our method. ALAC(original) shows comparable or the best performance compared with other certifications each task.
28

---

Tranno
Treatoo
Timettoo
Thnuts
MOUG
Roo
POUS
Secomsadir cal
Spureh orticer
Specefasire.co cod
Figure 10: Performance comparison on ten tasks. The ALAC method finds good trade-off between minimizing the accumulated cost and constraint violations contrast to their rivals.
29


Reviews:
[{"review_id": "yo6S-C1ZYs", "reviewer": "Reviewer_M8UP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a novel model that addresses sampling-based stability issues by employing Lyapunov stability certification. Additionally, the model combines this approach with the actor-critic algorithm to achieve optimality and stability in various simulated robotics tasks, outperforming the baselines.", "summary_of_recommendation": "I cannot recommend this paper as I faced difficulties in understanding the main math definitions and theorems presented in this work. The lack of comprehensive motivation and explanations hinders my ability to grasp the work's contribution and judge its significance.\n\nNevertheless, it is evident that this paper builds upon a very solid mathematical foundation and includes well-structured proofs. If other reviewers with expertise in control theory and stability analysis hold differing opinions, I would suggest considering their recommendations more seriously than mine.\n\n$\\textbf{Updated review after rebuttal:}$\n\nAfter reading the revised version of the paper, I recommend this paper now. The authors spent a lot of the effort to improve the readability of the paper and allow the researchers in a broad area to understand this work. Combining these updates with the solid math fundamentals, I recommend this work should to be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Unfortunately, I encountered difficulties in comprehending the paper starting from page three. As such, I anticipate the revised version to include comprehensive explanations for both the mathematical formulations and the experiments.\n\n\n$\\textbf{Updated review after rebuttal:}$\n\nMost of my concern has been addressed by the authors. I am in favor of the paper now.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n- This paper presents solid mathematical formulations supported by numerous math proofs in the appendix.\n\n- The authors have invested significant effort in presenting math equations, tables, and figures with a professional and graceful appearance.\n\nCons:\n\n- Unfortunately, I found it challenging to follow the paper's storyline. In my opinion, the main manuscript lacks self-containment, with insufficient descriptions and explanations. To comprehend the paper, readers are expected to possess the same knowledge and research experience as the authors.\n\n- The main manuscript lacks a connection to related works. While I understand that an 8-page limitation is restrictive for a theory-heavy paper, the absence of a proper discussion of related works makes it difficult to establish a knowledge framework and differentiate this work from previous research. Although the authors include related works in the Appendix, it cannot be assumed that reviewers will read it.\n\n- The paper lacks a clear motivation for this research and its connection to standard Reinforcement Learning (RL). For instance, it would be helpful to explain the difference between using cost and reward in the MDP description and how Cartpole-cost differs from the standard Cartpole task in RL. Additionally, a more detailed description of the costs used in the experiments would be valuable.\n\n- Some of the math definitions and theorems appear to be directly taken from other papers without proper citation. For instance, \"Mean Cost Stability\" in Definition 2.1 can also be found in Definition 1 of [9], yet the authors state in line 68 that they introduce it as the stability condition in this paper.\n\n- Many of the mathematical formulations in sections 3 and 4 lack proper explanations, making the paper challenging to understand, especially for readers with limited knowledge in the field. For example, from line 86 to line 100 (theorem 3.1), it is unclear what this theorem represents, why it is necessary, and its connection to the formulation in line 98. More reader-friendly explanations are needed to provide insight and intuition, even though the math proofs seem solid. \n\n- Please provide additional descriptions of the experiments, explaining the task settings and the rationale behind their selection in this paper.\n\n- Despite these drawbacks, I believe this paper is founded on solid mathematical principles. However, the missing part blocks conveying these concepts to the readers. Therefore, I suggest the authors consider whether CORL is the appropriate venue for this work, given the heavy emphasis on theory and math proofs in your work. A journal with a main manuscript length of 20+ pages might be a more suitable choice, allowing for more extensive explanations to ensure a technically smooth and coherent narrative.\n\n\n\nCritical issue:\n\n- CORL explicitly requires a limitation section in the main manuscript, which this paper failed to include, putting it at risk of desk rejection (https://www.corl2023.org/instructions-for-authors/). I recommend that the authors carefully review the submission guidelines in the future.\n\n\n$\\textbf{Updated review after rebuttal:}$\n\nMost of my concern has been addressed by the authors. I am in favor of the paper now.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lHKDmXscvrd", "reviewer": "Reviewer_fiUo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a Lyapunov-based approach for policy optimization. The state-action value function (critic) is replaced by a Lyapunov candidate, the expected discounted sum of costs, under certain MDP stability conditions. The critic can then be trained by optimizing the standard TD-learning objective. The policy update is formulated to achieve cost optimality such that the cost incurred at a step tends to zero while satisfying stability conditions added with suitable Lagrange multipliers. The multipliers are updated to increase the strength of the constraints gradually and reach the equilibrium state. A constrained critic network is used such that the Lyapunov value of state action is always positive, zero if the state is an equilibrium point. The authors compare the proposed setup with prior lyapunov-based methods on ten simulated environments. Ablations are shown to study the importance of the strength of the stability constraint.", "summary_of_recommendation": "I believe that the paper introduces an interesting direction in using lyapunov-based formulations for policy optimization in RL. It introduces new stability conditions and establishes maximum deviation bounds w.r.t. practical implementation. The comparison results are also extensive. However, the results are not convincing from a robotics perspective, as for both Minitaur and Humanoid, the method fails to surpass prior works.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See review above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a generalized approach to sThe paper presents a generalized approach to incorporating convergence stability in policy optimization. The lyapunov critic follows standard Q-function formulation but with cost (instead of reward) and the goal is to minimize the state-action value. The Lagrangian constraint-based policy update rule is intuitive and theoretically justified.\n\nThe algorithm is compared with relevant baselines using Lyapunov based formulations. The analysis on stability constraint violations is not explained. Is the same metric used for all the algorithms? What is the metric?\n\nThe paper mentions optimal time convergence, how is the convergence time optimal? Does that mean ALAC is sample efficient as compared to the baselines? Such a conclusion cannot be drawn from Fig 10. Can you clarify?\n\nFor generalization experiments, how do you calculate the errors like what is a desired goal and an achieved goal? What is a reference signal? Can you please clarify further, maybe, with an example from an environment?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "CYUr3s_jdP", "reviewer": "Reviewer_bi85", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors propose a novel model-free reinforcement learning (RL) algorithm. This algorithm (ALAC) borrows techniques from Lyapunov stability analysis to achieve optimal behavior under some stability constraints. The proposed approach is integrated within the Actor-Critic framework. The method is motivated by a theoretical analysis, the algorithm implementation is derived from the theoretical analysis. Finally, the algorithm is successfully implemented on 10 RL tasks related to robotics problems. Comparisons with existing RL algorithms baselines are provided.  ", "summary_of_recommendation": "This paper propose a new algorithm to solve RL problems under stability constraints. The proposed approach is backed by theoretical results and is benchmarked under a number of baselines on a set of robotics tasks in simulation. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "To improve the clarity of the paper, it would be great if the authors could explain which algorithm design choices are directly stemming from the theoretical analysis and the ones that are only loosely related to the theory section or simple heuristics. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides strong and detailed theoretical results on optimization of accumulated cost under stability constraints. \nThe paper demonstrates the superiority of the proposed algorithm compared to existing baselines on a series of RL robotics tasks in simulated environments. \n\nIt would be extremely valuable if the author could clearly state what are the approximations that are made when transitioning from the theory to the actual algorithm. What are the choices dictated by the theoretical analysis and which choices are arbitrary, or heuristics that tend to work well in practice. \n\nFor instance, why is the Lagrange multiplier lambda clipped between 0 and 1? Enforcing positive multiplier is expected for inequality constraints, however it is unclear why the algorithm requires clipping multiplier values larger than 1. The choice seems somewhat arbitrary. It would be great if the authors could motivate this choice with additional details.\n\nOverall, it is hard for the reader to distinguish between the algorithm design choices that are directly stemming from the theoretical analysis and the ones that are only loosely related to the theory section or simple heuristics. \n\nMinor issues:\ncultivated costs  <- accumulated costs\nsatisfy focal stability <- local stability", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a rigorous mathematical framework for policy optimization with sampling-based Lyapunov stability. The initial concerns of the reviewers regarding the clarity of the presentation was well addressed by the rebuttal. All reviewers now agree that the paper should be accepted  due to its rigorous mathematical derivation and the convincing results with comparison to relevant baselines. I agree that the paper contains interesting contributions and is of high quality. I therefore recommend acceptance.
Decision: Accept (Poster)