Title: Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lyapunov Stability

Reinforcement Learning

Generalized Lyapunov Stability

Stability of Reinforcement Learning Policies
Abstract: Establishing stability certificates for closed-loop systems under reinforcement learning (RL) policies is essential to move beyond empirical performance and offer guarantees of system behavior. Classical Lyapunov methods require a strict stepwise decrease in the Lyapunov function but such certificates are difficult to construct for learned policies. The RL value function is a natural candidate but it is not well understood how it can be adapted for this purpose. To gain intuition, we first study the linear quadratic regulator (LQR) problem and make two key observations. First, a Lyapunov function can be obtained from the value function of an LQR policy by augmenting it with a residual term related to the system dynamics and stage cost. Second, the classical Lyapunov decrease requirement can be relaxed to a generalized Lyapunov condition requiring only decrease on average over multiple time steps. Using this intuition, we consider the nonlinear setting and formulate an approach to learn generalized Lyapunov functions by augmenting RL value functions with neural network residual terms. Our approach successfully certifies the stability of RL policies trained on Gymnasium and DeepMind Control benchmarks. We also extend our method to jointly train neural controllers and stability certificates using a multi-step Lyapunov loss, resulting in larger certified inner approximations of the region of attraction compared to the classical Lyapunov approach. Overall, our formulation enables stability certification for a broad class of systems with learned policies by making certificates easier to construct, thereby bridging classical control theory and modern learning-based methods.

Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions

Abstract

Establishing stability certificates for closed-loop systems under reinforcement learning (RL) policies is essential to move beyond empirical performance and offer guarantees of system behavior. Classical Lyapunov methods require a strict stepwise decrease in the Lyapunov function but such certificates are difficult to construct for learned policies. The RL value function is a natural candidate but it is not well understood how it can be adapted for this purpose. To gain intuition, we first study the linear quadratic regulator (LQR) problem and make two key observations. First, a Lyapunov function can be obtained from the value function of an LQR policy by augmenting it with a residual term related to the system dynamics and stage cost. Second, the classical Lyapunov decrease requirement can be relaxed to a generalized Lyapunov condition requiring only decrease on average over multiple time steps. Using this intuition, we consider the nonlinear setting and formulate an approach to learn generalized Lyapunov functions by augmenting RL value functions with neural network residual terms. Our approach successfully certifies the stability of RL policies trained on Gymnasium and DeepMind Control benchmarks. We also extend our method to jointly train neural controllers and stability certificates using a multi-step Lyapunov loss, resulting in larger certified inner approximations of the region of attraction compared to the classical Lyapunov approach. Overall, our formulation enables stability certification for a broad class of systems with learned policies by making certificates easier to construct, thereby bridging classical control theory and modern learning-based methods.

1 Introduction

Designing and certifying stable control policies is a core challenge in both control theory and reinforcement learning (RL). Although RL methods are capable of learning policies that optimize long-term performance, their stability guarantees are often lacking. The classical Lyapunov theory [Lyapunov, 1992] offers a principled framework for certifying stability but constructing a function that satisfies the required stepwise decrease condition is often challenging in practice. While value functions from optimal control or RL encode long-term cumulative cost, they do not naturally satisfy the Lyapunov stability conditions.

We propose certifying stability using a generalized notion of Lyapunov function [Fürnsinn et al., 2023, 2025] that replaces the stepwise decrease condition with a multi-step, weighted decrease criterion, allowing temporary increases over a finite horizon. As a result, constructing a generalized Lyapunov stability certificate is easier, enabling stability verification for a broader class of policies and systems.

For linear systems with quadratic cost under discounted optimal control, the value function typically fails to satisfy classical Lyapunov conditions due to the influence of the discount factor. Postoyan et al. [2017] show that augmenting the value function with a quadratic residual term allows certifying

stability via a linear matrix inequality (LMI), although the guarantees are conservative with respect to the discount factor. Leveraging this insight, we show that augmenting the value function with a quadratic residual leads to a valid generalized Lyapunov function, which certifies stability for a boarder range of discount factors.

Building on the success of our approach in the linear setting, we extend it to nonlinear systems, where classical Lyapunov analysis is substantially more difficult. We augment value functions learned by RL algorithms (e.g., PPO [Schulman et al., 2017], SAC [Haarnoja et al., 2018]) with a neural residual term to form generalized Lyapunov functions, and jointly learn state-dependent multi-step weights to satisfy the generalized decrease condition. This joint learning scheme enables certification of nonlinear RL policies in settings where classical Lyapunov functions are challenging to construct by learning generalized certificates that tolerate non-monotonic behavior along trajectories.

In addition to showing stability of fixed policies, we extend our approach to support the joint synthesis of stable neural controllers and generalized Lyapunov certificates. Prior works [Chang et al., 2019, Wu et al., 2023, Yang et al., 2024] frame this problem as learning a controller-certificate pair that maximizes the volume of a certifiable region, while iteratively training on counter-examples where Lyapunov conditions fail. We incorporate our generalized multi-step condition into this formulation and show that it enables larger formally verifiable inner approximations of the region of attraction compared to standard one-step Lyapunov training.

Contributions. In summary, we make the following contributions:

We introduce an approach for certifying stability of reinforcement learning policies by constructing generalized Lyapunov functions, composed of the policy's value function and a learned neural residual. Empirically, our formulation certifies policies on Gymnasium [Towers et al., 2024] and DeepMind Control Suite [Tassa et al., 2018] benchmarks where classical single-step Lyapunov methods fail.

We show analytically that, for the linear quadratic regulator (LQR) problem, the validity of the proposed generalized Lyapunov function can be certified by a set of LMIs.

We extend the formulation to jointly learn a control policy and a stability certificate using a generalized multi-step Lyapunov loss, resulting in larger certified regions of attraction compared to classical Lyapunov approaches.

We provide an open-source implementation of our method at https://github.com/ ExistentialRobotics/Generalized_Policy_Stability .

Related Work. Optimal control [Bertsekas, 2012] and reinforcement learning (RL) [Sutton et al., 1998] offer a variety of principled techniques for designing control policies to optimize the performance of intelligent agents and dynamical systems. In the special case of linear systems with quadratic rewards, the optimality and stability of LQR policies [Anderson and Moore, 1990] are well understood. However, despite a variety of methods for synthesizing policies (e.g., PPO [Schulman et al., 2017], SAC [Haarnoja et al., 2018], and DDPG [Lillicrap et al., 2015] are among the most widely used) for nonlinear systems with nonlinear costs, very few techniques are available for certifying the stability of the resulting closed-loop system.

Several works have explored connections between Lyapunov stability [Lyapunov, 1992] and RL, mainly in the context of safe RL. Perkins and Barto [2002] proposed a safe RL formulation by learning to switch among hand-designed Lyapunov-stable controllers. Berkenkamp et al. [2017] developed a model-based RL algorithm that uses Gaussian process dynamics and Lyapunov verification to safely improve policies while guaranteeing stability with high probability. Chow et al. [2018] formulate a Lyapunov-based method for safe RL in constrained Markov decision processes, using linear constraints to ensure safety during training. Hejase and Ozguner [2023] regularized policy training by penalizing violations of a learned Lyapunov condition to encourage stability in autonomous driving. Mittal et al. [2020] learn a Lyapunov function from suboptimal demonstrations and incorporate it as a terminal cost in a one-step model predictive control formulation to improve stability.

In parallel, there has been growing interest in jointly learning policies and Lyapunov functions from data to obtain stability guarantees for nonlinear systems. Chang et al. [2019] trained Lyapunov certificates with formal verification via satisfiability modulo theories. Other works constrain neural architectures to satisfy Lyapunov conditions structurally [Gaby et al., 2022, Boffi et al., 2021], or apply mixed-integer programming [Dai et al., 2021]. Long et al. [2023] studied neural distributionally robust Lyapunov functions under additive uncertainty. Wu et al. [2023] learn provably stable controllers

for discrete-time nonlinear systems by jointly training a policy and Lyapunov function, with formal verification via mixed-integer linear programs. Yang et al. [2024] use gradient-guided falsification for training a controller-certificate pair, and apply formal verification via α, β -CROWN [Wang et al., 2021] to certify stability post training. See also Dawson et al. [2023] for a comprehensive survey.

2 Problem Formulation

Consider the discrete-time system:

where X is open and f : X × U → X is locally Lipschitz. A control policy π : X → U for the system can be obtained by solving an infinite-horizon discounted optimal control problem:

where γ ∈ (0 , 1) is a discount factor and ℓ ( x , u ) is a stage cost (or reward in the case of maximization), specifying the performance criterion.

Optimal control and reinforcement learning methods usually solve a problem like (2) to obtain a policy π and associated value function J π γ . We consider a deterministic problem in our theoretical development for simplicity but apply our approach to stochastic problems in the evaluation (Section 5). Note that, while reinforcement learning methods work with stochastic control policies during training, only the mode of the final trained policy is used at test time, allowing us to treat it as deterministic.

Given a policy π , we are interested in certifying whether it stabilizes the closed-loop system:

We assume π ( 0 n ) = 0 m , f ( 0 n , 0 m ) = 0 n , ℓ ( 0 n , 0 m ) = 0 , and 0 n ∈ X , so that the origin is an equilibrium point of the system. Stability can be certified by identifying a Lyapunov function.

Definition 2.1 ( Lyapunov Function ) . Consider the closed-loop system (3). A continuous function V : X → R ≥ 0 is a Lyapunov function if it satisfies:

Lyapunov functions certify the asymptotic stability of the system, as stated in the following result. Theorem 2.2 ( Asymptotic Stability via a Lyapunov Function [Khalil, 1996, Theorem 3.3]) . If there exists a Lyapunov function V as in Definition 2.1, then the origin x = 0 n is an asymptotically stable equilibrium of the system (3) .

When π arises from (2), the corresponding value function J π γ is also available. We are interested in whether J π γ can itself certify the stability of (3) or assist in constructing a certificate.

3 Lyapunov Stability Analysis for Linear Quadratic Problems

To understand the relationship between a discounted value function J π γ obtained from (2) and a Lyapunov function V certifying stability of the closed-loop system (3), we first study a simple setting with a linear system and quadratic stage cost.

Consider the discrete-time linear system: x k +1 = Ax k + Bu k , where x k ∈ R n , u k ∈ R m , and A , B are known constant matrices.

Choosing the stage cost in (2) as ℓ ( x , u ) = x ⊤ Qx + u ⊤ Ru with Q = C ⊤ C ⪰ 0 and R ≻ 0 leads to the discounted LQR problem [Anderson and Moore, 1990]. The optimal value function J ∗ γ ( x ) = x ⊤ P γ x is quadratic, where P γ solves the discounted Algebraic Riccati Equation (ARE):

The corresponding optimal feedback gain is K ⋆ γ = -( γ B ⊤ P γ B + R ) -1 B ⊤ P γ A , which yields the policy π ∗ γ ( x k ) = K ⋆ γ x k . Under this policy, the closed-loop system evolves as

Although J ∗ γ satisfies the discounted Bellman equation, it is well understood that it does not guarantee Lyapunov decrease due to γ ∈ (0 , 1) [De Farias and Van Roy, 2003]. Moreover, verifying stability by directly analyzing the eigenvalues of F ∗ γ and solving the discounted ARE (5) is nontrivial due to its nonlinear dependence on γ . This motivates the idea [Postoyan et al., 2017] of constructing valid Lyapunov functions by augmenting J ∗ γ with a residual term. To formalize this, we first state standard assumptions that ensure the existence of a stabilizing policy and a well-defined value function.

Assumption 3.1. The pair ( A , B ) is stabilizable and the pair ( A , C ) is detectable.

The following result shows the existence of a Lyapunov certificate derived from the value function.

Theorem 3.2 ( Lyapunov Stability via LMIs for Discounted LQR [Postoyan et al., 2017]) . Suppose Assumption 3.1 holds. Consider an optimal policy π ∗ γ and value function J ∗ γ obtained from the LQR problem with discount γ . Let P denote the solution to the undiscounted ARE in (5) (with γ = 1 ). If there exist symmetric positive definite matrices S 0 , S 1 ≻ 0 and scalars ϖ,α > 0 such that:

then the function V ( x ) := J ∗ γ ( x ) + 1 ϖ x ⊤ S 0 x is a Lyapunov function for the closed-loop system (6) , and certifies global exponential stability for any discount factor satisfying γ > ϖ ϖ + α .

Theorem 3.2 shows that while the discounted value function J ∗ γ may not satisfy the Lyapunov condition on its own, it can be modified into a valid certificate when γ is sufficiently large. The required residual term and a computable lower bound on γ are obtained by solving a set of LMIs. However, this bound is often conservative compared to the true stability threshold γ ∗ obtained by analyzing the closed-loop dynamics via the discounted ARE, as illustrated in the following example.

Example 3.3 ([Postoyan et al., 2017, Example 1]) . Consider the scalar system x k +1 = 2 x k + u k with stage cost ℓ ( x, u ) = x 2 + u 2 . The optimal policy is u k = K ⋆ γ x k , where

The closed-loop multiplier is F ∗ γ = 2+ K ⋆ γ , and the origin is globally exponentially stable if and only if | F ∗ γ | < 1 , which is equivalent to γ > γ ∗ = 1 / 3 . However, applying the LMIs from Theorem 3.2 yields a feasible solution for γ > 0 . 8090 , which is significantly more conservative.

Example 3.3 highlights the limitation of classical Lyapunov analysis and motivates the development of an alternative formulation with less conservative conditions for certifying stability.

4 Generalized Lyapunov Stability and Applications to Linear Systems

Building on the observation from Example 3.3, we introduce a generalized notion of Lyapunov stability [Fürnsinn et al., 2023, Definition 2.2] that relaxes the classical pointwise decrease condition.

Definition 4.1 ( Generalized Lyapunov Function ) . Consider the closed-loop system in (3). A continuous function V : X → R ≥ 0 is a generalized Lyapunov function if it satisfies (4a) and there exist an integer M ∈ N > 0 and state-dependent non-negative weights σ 1 ( x ) , . . . , σ M ( x ) such that

and, for any x ∈ X \ { 0 n } , the following generalized decrease condition holds:

where x i +1 = f ( x i , π ( x i )) with x 0 = x .

Condition (9) generalizes the decrease requirement in (4b) by allowing temporary increases in V over individual steps, provided that its weighted average decreases over a finite horizon of length M . We state the stability guarantee provided by a generalized Lyapunov function in the next theorem.

Theorem 4.2 ( Asymptotic Stability via a Generalized Lyapunov Function ) . Consider the closedloop system in (3) , and assume the policy π is Lipschitz on X . If there exists a generalized Lyapunov function V : X → R ≥ 0 as in Definition 4.1, then x = 0 n is an asymptotically stable equilibrium.

See Appendix A for the proof (adapted from [Fürnsinn et al., 2023, Theorem 2]), and Appendix B for an explicit construction of a classical one-step Lyapunov function from a generalized Lyapunov function when the step weights { σ i } i M =1 are state-independent.

Remark 4.3 ( Relation to k -Inductive Verification ) . The generalized Lyapunov condition in (9) can be viewed through the lens of k -step inductive verification [Brain et al., 2015, Anand et al., 2022, Wooding and Lavaei, 2024]. In particular, by selecting weights σ 1 = · · · = σ k -1 = 0 and σ k = M , condition (9) reduces to V ( x t + k ) ≤ (1 -α ) V ( x t ) , which is analogous to a k -step Lyapunov decrease condition used in k -inductive reasoning. Classical k -induction requires a strict decrease at the final step while permitting bounded increases in intermediate steps. In contrast, Definition 4.1 enforces a decrease in a weighted average across multiple steps, with the weights σ i chosen adaptively or learned during training. This distinction provides additional flexibility, enabling certificates that may be easier to construct for nonlinear systems with learned controllers.

Returning to the LQR setting in Section 3, we consider certifying the stability of (6) via a generalized Lyapunov function. Building on Theorem 3.2, we augment J ∗ γ with a quadratic residual and require the composite function to satisfy (9), leading to a new set of LMIs for stability certification.

Theorem 4.4 ( Generalized Lyapunov Stability for Discounted LQR via LMIs ) . Suppose Assumption 3.1 holds. Consider an optimal policy π ∗ γ and value function J ∗ γ obtained from the LQR problem with discount γ . Let P denote the solution to the undiscounted ARE in (5) (with γ = 1 ). If there exist symmetric positive definite matrices S 0 , S 1 , . . . , S M , scalars ϖ,α 1 , . . . , α M > 0 , and weights σ 1 , . . . , σ M ≥ 0 satisfying ∑ M i =1 σ i ≥ M , such that:

is a generalized Lyapunov function in the sense of Definition 4.1 (with constant weights), and certifies that the origin is globally exponentially stable for the closed-loop system (6) , provided that:

Remark 4.5 ( Feasibility of Multi-Step LMIs ) . Note that when σ 1 = M and σ i = 0 for all 2 ≤ i ≤ M , Theorem 4.4 recovers the classical Lyapunov result in Theorem 3.2. Therefore, if the original one-step LMIs (7) are feasible, then feasible solutions for (10) also exist. Empirically, the multi-step formulation can certify stability in additional cases where the one-step condition is infeasible, effectively enlarging the set of stabilizable systems. Formal characterization of this weaker feasibility property is left for future work.

Remark 4.6 ( Choice of σ i Weights ) . While the multi-step formulation enables optimizing the weights σ 1 , . . . , σ M to minimize the certified lower bound on γ in (12) , finding globally optimal weights is challenging due to its non-convexity nature. In practice, heuristics such as grid search for small M and random sampling with local refinements for larger M are sufficient to achieve noticeable improvements over the one-step baseline.

then the function

Example 4.7 ( Example 3.3 Revisited ) . We illustrate the benefits of the generalized multi-step LMIs from Theorem 4.4 in Example 3.3. We first focus on the case M = 2 . Figure 1 shows how the certified bound on γ varies with the choice of weights σ 1 and σ 2 , constrained by σ 1 + σ 2 = 2 . Since the undiscounted value function ( γ = 1 ) satisfies the Lyapunov condition, this bound is capped at γ = 1 . The classical Lyapunov setting ( σ 1 , σ 2 ) = (2 , 0) yields a conservative bound of γ > 0 . 8090 , whereas optimizing over σ 1 and σ 2 improves the bound to γ > 0 . 6229 at ( σ 1 , σ 2 ) = (1 . 54 , 0 . 46) . Another important observation is that increasing M significantly reduces the certified lower bound on γ , progressively approaching the true stability threshold γ ⋆ = 1 / 3 , as shown in Figure 2.

1.0

Bound

0.9

人

Certified

0.8

01+02=2

0.7

Min Certifiedy

Classical LF(o=2)

0.6

0.0

0.5

1.0

1.5

2.0

01

0.8

punoc

0.7

0.6

b

>0.5

0.4

0.3

2

Certified y bounds

Trueybound

4

6

8

Number of steps (M)

5 Nonlinear Systems: Stability Certification of RL Policies

Inspired by the treatment of linear systems described in Section 4, we first formulate a post-hoc approach to certify stability of policies for nonlinear systems obtained by RL using a generalized Lyapunov function. The key observation from Theorem 4.4 is that augmenting the optimal value function with a residual term can result in a valid generalized Lyapunov function, as in (11). Here, we use the same idea to form generalized Lyapunov functions for nonlinear systems with unknown dynamics, which is a problem considered by RL. The value function and policy obtained by an RL algorithm are typically parameterized by neural networks. Let π RL be a pre-trained RL policy (e.g., obtained by [Schulman et al., 2017, Haarnoja et al., 2018, Hansen et al., 2022]), and let J π RL γ denote its corresponding learned value function. We consider a generalized Lyapunov candidate as:

where φ ( x ; θ 1 ) is a neural residual correction. To allow more flexibility in the generalized Lyapunov condition (9), we introduce a step-weighting network σ ( x ; θ 2 ) ∈ R M ≥ 0 that outputs non-negative weights over a rollout horizon of length M . The weights are required to satisfy ∑ M i =1 σ i ( x ; θ 2 ) ≥ M . These weights are then used to construct the generalized Lyapunov decrease condition:

where ¯ α ∈ (0 , 1) is a user-specified decay parameter. We train the networks φ ( · ; θ 1 ) and σ ( · ; θ 2 ) jointly by minimizing a loss that penalizes violations of the generalized Lyapunov condition:

where { x ( i ) k } N i =1 are sampled initial states.

Remark 5.1 ( Network Architecture ) . In practice, the learned value function J π RL γ may not be optimal, hence, may not satisfy J π RL γ ( 0 n ) = 0 . To ensure that the generalized Lyapunov candidate satisfies V ( 0 n ; θ 1 ) = 0 and is positive definite, we modify (13) as:

̸

where β > 0 is a small constant (the term β ∥ x ∥ 2 enforces strict positive definiteness for x = 0 n and improves numerical stability near the origin.) The step-weighting network σ ( x ; θ 2 ) ∈ R M ≥ 0 ends with a softmax layer scaled by M , ensuring the output weights satisfy 1 M ∑ M i =1 σ i ( x ; θ 2 ) = 1 .

10

Remark 5.2 ( Equilibrium Behavior ) . In practice, RL policies often converge to a neighborhood near the origin rather than the origin itself. Thus, we train and verify the generalized Lyapunov condition over the set X \ B ( 0 n ; δ ) , where B ( 0 n ; δ ) denotes a ball of radius δ around the origin.

Training and Evaluation Setup. We evaluate our method on two standard RL control benchmarks from Gymnasium [Towers et al., 2024] and the DeepMind Control Suite [Tassa et al., 2018]. RL policies π RL and their corresponding value functions J π RL γ are trained using implementations from Raffin et al. [2021], Hansen et al. [2022]. We collect a total of N rollout trajectories by simulating the closed-loop system under π RL, with each trajectory τ ( i ) = { x ( i ) k } M k =0 starting from a randomly sampled initial state x ( i ) 0 . Additional training and architecture details are provided in Appendix D.

Inverted Pendulum Swingup. We consider the inverted pendulum environment from Towers et al. [2024] with parameters m = 1 , l = 1 , g = 10 , and control limits | u | ≤ mgl 5 = 2 . The state space is [ -π, π ) × [ -8 , 8] . Due to tight torque limits, the pendulum must swing back and forth to build momentum before reaching the upright position. This makes it challenging to synthesize a policy with a Lyapunov certificate valid over the entire state space. Prior work [Yang et al., 2024, Long et al., 2024] has shown that regions where stability can be verified are typically restricted to small neighborhoods near the upright position, failing to cover the full swing-up trajectories. In contrast, modern RL policies can discover effective swing-up behaviors but lack stability guarantees. Our method bridges this gap: we take a trained SAC policy Haarnoja et al. [2018] and apply our generalized Lyapunov function training with M = 15 . Figure 3 shows the generalized Lyapunov function values along several trajectories from different initial states. As expected, the function exhibits non-monotonic behavior but shows an overall decline over the planning horizon. Figure 4 visualizes the Lyapunov function across the state space. Figure 5 plots the residual F ( x k ) defined in (14), verifying that the generalized decrease condition is satisfied throughout the domain.

250

8

270

8

0.0

Generalized Lyapunov Value

6

240

6

12.5

200

25.0

4

210

Dot (rad/s)

4

37.5

150

2

180

2

-50.0

150

0

-62.5

100

120

Theta

-2

-2

75.0

90

50

4

60

-4

-87.5

-100.0

0

-6

30

-6

112.5

0

25

50

75

Time Step

100125

150

175

200

-8

-6

-4

2

Theta (radians)

0

2

4

6

0

-8

-6

4

Theta (radians)

2

0

2

4

6

Figure 5: Generalized Lyapunov decrease condition.

Cartpole Swingup. We consider the cartpole swing-up task from Tassa et al. [2018]. The system state is represented as [ x, cos( θ ) , sin( θ ) , ˙ x, ˙ θ ] , where x is the cart position, θ is the pole angle, and the remaining terms are their velocities. The goal is to swing the pole upright θ = 0 and stabilize the position at x = 0 . We use a TD-MPC policy [Hansen et al., 2022] and apply our certificate training (15) with M = 20 . Figure 6 visualize the generalized Lyapunov condition across three representative 2D slices of the state space, with the remaining two states fixed at zero. The generalized decrease condition is satisfied throughout these slices, suggesting asymptotic stability of the RL policy.

3

10.0

2

15

7.5

-15

4

-20

(m)

Pos

1

0

-30

45

-60

75

(S/pe

Dot(

(rac

5.0

0.0

2.5

-30

-60

45

Velocity (m/s)

2

0

-40

-60

-80

Theta

75

-90

2.5

-100

1

-105

-5.0

-90

-2

120

-105

-2

135

120

7.5

120

-4

140

-3

6

-4

-2

Theta (radians)

0

2

4

6

-10.0

-6

4

Theta (radians)

-2

0

2

4

6

-3

-2

-1

Pos (m)

0

1

2

3

-160

In addition to qualitative visualizations, we quantitatively evaluate our learned certificates by sampling N test states from the full state space and checking whether (14) is satisfied. For each environment, we train certificates for multiple RL policies (the PPO policy fails to stabilize the cartpole from some initial states). As shown in Table 1, the condition holds for all test states in both environments.

Concentration of Learned Step-Weights. To further analyze the learned generalized Lyapunov function, we examine the distribution of the learned step-weights across the horizon. For each trained policy, the step-weight network was evaluated over 10 , 000 uniformly sampled states from the state space, and the normalized weights were averaged across samples. The weights were then grouped into five equal bins corresponding to different segments of the Lyapunov horizon. Table 2 summarizes the fraction of the total weight assigned to each bin.

As shown in Table 2, the consistent concentration of weights toward the end of the horizon reveals a key strategy learned by our method. By assigning lower weights to initial steps and higher weights to later ones, the network effectively learns to tolerate initial non-monotonic transients in the Lyapunov candidate, focusing instead on ensuring a decisive, weighted decrease over the full horizon. This behavior empirically realizes the central benefit of the generalized Lyapunov condition: providing a flexible yet formal basis for certifying complex neural policies.

6 Joint Synthesis of Stable Neural Policies and Certificates

So far, we addressed the certification of stability for fixed pre-trained control policies. In this section, we consider joint synthesis of neural controllers and Lyapunov certificates and employ formal verification inspired by Chang et al. [2019], Dai et al. [2021], Wu et al. [2023], Yang et al. [2024]. Compared to prior methods, our generalized Lyapunov framework naturally extends this setting by replacing the standard one-step decrease condition with the proposed multi-step, weighted formulation in (9).

Definition 6.1 ( Region of Attraction ) . The region of attraction (ROA) R ⊆ R n of the (locally asymptotically stable) origin for the discrete-time system (3) is the set of all points from which the system trajectory converges to the origin, i.e., x 0 ∈ R implies lim k →∞ x k = 0 n .

In general, precise characterizations of R are challenging to obtain and one instead looks for suitable approximations. Here, we seek to jointly learn a policy π ( x ; ϕ ) and a certificate function V ( x ; θ 1 ) such that the closed-loop system x k +1 = f ( x k , π ϕ ( x k )) is provably asymptotically stable and obtain at the same time an inner approximation S of the ROA R . Yang et al. [2024] formalized it as a constrained optimization that maximizes the volume of a Lyapunov sublevel set S = { x ∈ R n : V ( x ) ≤ ρ } under Lyapunov constraints

where ¯ α ∈ (0 , 1) is a user-specified decay parameter. To utilize a generalized Lyapunov function, we replace (17c) with the generalized M -step condition for each x k ∈ S , with ∑ M i =1 σ i ( x k ) ≥ M :

where σ ∈ R > 0 is a uniform lower bound on the weights. The condition (18) enables learning certificates that tolerate non-monotonic behavior along a trajectory, but it no longer guarantees that S is forward invariant . However, as Theorem 6.2 shows, S still defines a valid inner approximation of the ROA and guarantees asymptotic stability of the origin.

Theorem 6.2 ( Asymptotic Stability Relative to S ) . If the optimization problem (17) is solved with the generalized condition (18) instead of (17c) , then S ⊂ R .

See Appendix E for the proof.

Training Formulation. Following [Yang et al., 2024], we reformulate the problem (17) into a learning objective by sampling states x k ∈ X and penalizing violations of the Lyapunov conditions using soft constraints. For each x k ∈ D , we simulate M forward steps under the closed-loop dynamics and compute the generalized Lyapunov residual F ( x k ) as defined in (18).

To enforce stability and domain constraints, we define the stability loss:

where H ( x k ) := ∑ M i =1 ∥ ReLU ( x k + i -x up ) + ReLU ( x lo -x k + i ) ∥ 1 penalizes violations of the bounded domain X = { x | x lo ≤ x ≤ x up } . The inner min ensures that the generalized Lyapunov decrease condition is only enforced inside the certified region S .

To expand the certifiable region, Yang et al. [2024] proposed a surrogate region loss L region := ∑ N j =1 ReLU ( V ( x j ; θ 1 ) /ρ -1) , where the candidate states x j are obtained via random boundary sampling or projected gradient descent (PGD) to minimize V ( · ; θ 1 ) . PGD is also used for falsification by maximizing stability violations, generating additional training states for D .

The final training objective is L ( θ 1 , ϕ ) := ∑ x k ∈D L stab ( x k ) + c 1 L region + c 2 ∥ θ 1 , ϕ ∥ 1 , where D contains states sampled randomly and generated by falsification.

Remark 6.3. Although the theoretical framework (Theorem 4.2) supports trainable weights σ i ( x k ) , including neural network parameterizations (see Section 5), certifying stability under jointly learned controllers, Lyapunov functions, and weights remains computationally challenging with existing tools [Wang et al., 2021, Gao et al., 2013]. In practice, we fix σ i during training. For small values of M , we select the best-performing weight configuration via a simple grid search.

Verification Formulation. After training, we verify that the generalized decrease condition holds within X using the α -β -CROWN verifier [Wang et al., 2021]. We formally certify Lyapunov stability of the origin if the following holds for all x k ∈ X :

As Theorem 6.2 implies, if (20) holds for all x k ∈ X , then S is an inner approximation of R .

Evaluation. We evaluate our generalized Lyapunov synthesis approach on three systems presented in [Yang et al., 2024]: inverted pendulum, path tracking, and 2D quadrotor (with a 6D state space). Detailed system dynamics and neural network architectures used are provided in Appendix F.

Figures 7 and 8 show the certified stability regions S for different horizon lengths M . We use fixed step weights ( σ 1 , σ 2 , . . . ) selected via grid search: for the inverted pendulum, (0 . 4 , 1 . 6) for M =2 and (0 . 3 , 1 . 5 , 1 . 2) for M =3 ; for path tracking, (0 . 4 , 1 . 6) and (1 . 2 , 1 . 2 , 0 . 6) , respectively. As shown in the figures, multi-step training and verification yields consistently larger certifiable ROAs.

Table 3 reports the quantitative results, including the volume ( S ) and the verification time. The volume is estimated via Monte Carlo integration: we sample n = 10 6 points in X and compute the fraction satisfying V ( x ) ≤ ρ , averaged across trials and scaled by the total domain volume. To assess

10

1-step

(s/pe

2-step

(rac

5

3-step

AngularVelocity

0

-5

-10

-10

-5

0

5

10

Angle (rad)

3

1-step

2

2-step

(rad)

3-step

1

AngleError(

0

-1

-2

-3

3

-2

-1

0

1

2

3

DistanceError (m)

robustness, we repeat each experiment 10 times with different random seeds and report the mean and standard deviation. The results confirm that our generalized stability formulation consistently certifies larger ROA volumes compared to the classical one-step Lyapunov approach. This improvement comes at the cost of longer verification times, since bounding multiple intermediate states and their weighted combinations increases the complexity of bound propagation and branching decisions.

(a) Certified ROA volume ( S ).

7 Conclusion, Limitations, and Future Work

We presented a new approach for constructing stability certificates for closed-loop systems under policies obtained from optimal control and RL. We also showed that our method can be used for joint synthesis of neural policies and stability certificates. Rather than learning certificates from scratch, a key insight is that the value function of a considered policy can be augmented with a residual term to obtain a certificate candidate. We also replaced the classical Lyapunov decrease condition with a generalized multi-step Lyapunov formulation, which provides asymptotical stability guarantees while improving learning flexibility. Our theoretical and empirical results show that this perspective enables construction of more effective stability certificates compared to traditional Lyapunov approaches.

Our approach has several limitations. First, the certification horizon M is fixed during training, and it remains unclear how to determine the minimal viable M for a given system. Second, the method has not yet been applied to high-dimensional systems such as humanoids or dexterous manipulators. Third, formally verifying neural policies (e.g., from RL), along with learned state-dependent weights and certificate functions, remains computationally challenging with existing tools.

This work opens up several promising directions for future research. One key avenue is to develop partial-state stability certification, providing guarantees for task-relevant components such as torso height or velocity in humanoid locomotion rather than for the full system state, which can improve both scalability and interpretability. A second direction is to extend the generalized Lyapunov formulation to certify robust stability under bounded perturbations and stochastic disturbances, taking inspiration from input-to-state and robust Lyapunov theory. Another important direction is to relax the requirement of knowing the equilibrium state, allowing stability analysis when the equilibrium is unknown or non-isolated, as often occurs in reinforcement learning, by inferring equilibria from data or from the reward structure. Finally, investigating the connection between generalized Lyapunov conditions and optimality objectives may yield new insights for reward design and policy refinement in reinforcement learning.

(b) Verification time (seconds).

Acknowledgments

We gratefully acknowledge support from ONR Award N00014-23-1-2353 and NSF Award CCF2112665 (TILOS).

References

Mahathi Anand, Vishnu Murali, Ashutosh Trivedi, and Majid Zamani. K-inductive barrier certificates for stochastic systems. In ACM International Conference on Hybrid Systems: Computation and Control , pages 1-11, 2022.

Brian D. O. Anderson and John B. Moore. Optimal control: linear quadratic methods . Prentice-Hall Inc., 1990.

Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems , volume 30, page 908-919, 2017.

Dimitri Bertsekas. Dynamic programming and optimal control: Volume I , volume 4. Athena scientific, 2012.

Nicholas Boffi, Stephen Tu, Nikolai Matni, Jean-Jacques Slotine, and Vikas Sindhwani. Learning stability certificates from data. In Conference on Robot Learning , pages 1341-1350. PMLR, 2021.

Martin Brain, Saurabh Joshi, Daniel Kroening, and Peter Schrammel. Safety verification and refutation by k-invariants and k-induction. In International Static Analysis Symposium , pages 145-161. Springer, 2015.

Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Lyapunov control. In Advances in Neural Information Processing Systems , volume 32, 2019.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Lyapunovbased approach to safe reinforcement learning. In Advances in Neural Information Processing Systems , volume 31, 2018.

Hongkai Dai, Benoit Landry, Lujie Yang, Marco Pavone, and Russ Tedrake. Lyapunov-stable neural-network control. In Robotics: Science and Systems , Virtual, July 2021.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics , 39(3):1749-1767, 2023. doi: 10.1109/TRO.2022.3232542.

Daniela Pucci De Farias and Benjamin Van Roy. The linear programming approach to approximate dynamic programming. Operations research , 51(6):850-865, 2003.

Annika Fürnsinn, Christian Ebenbauer, and Bahman Gharesifard. Relaxed feasibility and stability criteria for flexible-step MPC. IEEE Control Systems Letters , 7:2851-2856, 2023. doi: 10.1109/ LCSYS.2023.3289150.

Annika Fürnsinn, Christian Ebenbauer, and Bahman Gharesifard. Flexible-step model predictive control based on generalized Lyapunov functions. Automatica , 175:112215, 2025. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2025.112215. URL https://www.sciencedirect. com/science/article/pii/S0005109825001074 .

Nathan Gaby, Fumin Zhang, and Xiaojing Ye. Lyapunov-net: A deep neural network architecture for Lyapunov function approximation. In IEEE Conference on Decision and Control , pages 2091-2096, 2022. doi: 10.1109/CDC51059.2022.9993006.

Sicun Gao, Soonho Kong, and Edmund M Clarke. dreal: An smt solver for nonlinear theories over the reals. In International Conference on Automated Deduction , pages 208-214. Springer, 2013.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning , pages 8387-8406. PMLR, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE International Conference on Computer Vision , pages 1026-1034, 2015.

Bilal Hejase and Umit Ozguner. Lyapunov stability regulation of deep reinforcement learning control with application to automated driving. In American Control Conference , pages 4437-4442, 2023.

Hassan Khalil. Nonlinear Systems . Prentice Hall, 1996.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Kehan Long, Yinzhuang Yi, Jorge Cortes, and Nikolay Atanasov. Distributionally robust Lyapunov function search under uncertainty. In Learning for Dynamics and Control Conference , pages 864-877. PMLR, 2023.

Kehan Long, Jorge Cortés, and Nikolay Atanasov. Distributionally robust policy and lyapunovcertificate learning. IEEE Open Journal of Control Systems , 3:375-388, 2024. doi: 10.1109/ OJCSYS.2024.3440051.

Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control , 55(3):531-534, 1992.

Mayank Mittal, Marco Gallieri, Alessio Quaglino, Seyed Sina Mirrazavi Salehian, and Jan Koutník. Neural Lyapunov model predictive control: Learning safe global controllers from sub-optimal examples. arXiv preprint arXiv:2002.10451 , 2020.

Theodore J Perkins and Andrew G Barto. Lyapunov design for safe reinforcement learning. Journal of Machine Learning Research , 3(Dec):803-832, 2002.

Romain Postoyan, Lucian Bu¸ soniu, Dragan Neši´ c, and Jamal Daafouz. Stability analysis of discretetime infinite-horizon optimal control with discounted cost. IEEE Transactions on Automatic Control , 62(6):2736-2749, 2017. doi: 10.1109/TAC.2016.2616644.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. URL https://arxiv.org/ abs/1707.06347 .

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 , 2018.

Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 , 2024.

Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. Advances in Neural Information Processing Systems , 34, 2021.

Ben Wooding and Abolfazl Lavaei. Learning k-inductive control barrier certificates for unknown nonlinear dynamics beyond polynomials. arXiv preprint arXiv:2412.07232 , 2024.

Junlin Wu, Andrew Clark, Yiannis Kantaros, and Yevgeniy Vorobeychik. Neural Lyapunov control for discrete-time systems. In Advances in Neural Information Processing Systems , volume 36, pages 2939-2955, 2023.

Lujie Yang, Hongkai Dai, Zhouxing Shi, Cho-Jui Hsieh, Russ Tedrake, and Huan Zhang. Lyapunovstable neural control for state and output feedback: A novel formulation. In International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=3xPMW9JURD .

A Proof of Theorem 4.2

Proof. Attractivity. We first show attractivity. Fix any initial condition x 0 ∈ X \ { 0 } and let ( x k ) k ≥ 0 be the corresponding trajectory. For each k ≥ 0 choose an index

whose existence is guaranteed by (9). Define the strictly increasing sequence of return indices

for which, by construction,

Due to the positive-definiteness of V , applying standard Lyapunov arguments on { V ( x k j ) } , we conclude that lim k j →∞ V ( x k j ) = 0 .

Because the gaps satisfy k j +1 -k j ≤ M and V is continuous, we have

Finally, the positive-definiteness of V yields x k → 0 n as k →∞ . Thus the origin is attractive .

Stability. Since both the system dynamics f and the policy π are assumed to be Lipschitz, the closed-loop map x ↦→ f ( x , π ( x )) is also Lipschitz on the forward-invariant set X . Let L > 0 denote the Lipschitz constant. Then, for all 0 ≤ i < M and k ≥ 0 ,

Fix an arbitrary ε > 0 and choose

Consider any initial state with ∥ x 0 ∥ < δ . By construction of the return indices k j in (22) we have ∥ x k j +1 ∥ ≤ ∥ x k j ∥ for all j , so in particular

For an arbitrary time index k pick j such that k j ≤ k < k j + M . Applying (25) with i = k -k j < M and (27) yields

Because ε was arbitrary, the ε -δ condition holds and the origin is Lyapunov stable.

In conclusion, the origin is both attractive and Lyapunov-stable on X ; therefore it is asymptotically stable relative to X .

Remark A.1 ( Exponential Stability ) . Suppose the function V : X → R ≥ 0 satisfies the positive definiteness condition in (4a) , and there exist constants β 1 , β 2 , p > 0 such that

Suppose further that V satisfies the strict generalized decrease condition for all x k ∈ X ,

where α : R ≥ 0 → R ≥ 0 is a classK function. Then the origin x = 0 n is exponentially stable.

B Construction of a Classical Lyapunov Function from a Generalized One

This appendix shows that when the generalized M -step Lyapunov decrease in Definition 4.1 holds with state-independent step weights, one can explicitly construct a classical one-step Lyapunov function, following a similar idea to classical constructions that convert multi-step (non-monotone) decrease conditions into a one-step Lyapunov decrease ? . In contrast, when the step weights depend on the state, obtaining an explicit closed-form classical Lyapunov function from the generalized certificate appears non-trivial; while asymptotic stability still follows from Theorem 4.2, deriving a simple one-step Lyapunov function construction in the state-dependent case is left for future work.

Assumption B.1 ( Constant step weights ) . The step weights in Definition 4.1 are constant, i.e., there exist σ 1 , . . . , σ M ≥ 0 (independent of x ) such that for all x ∈ X \ { 0 } ,

where x 0 = x and x i +1 = f ( x i , π ( x i )) for i = 0 , . . . , M -1 .

Lemma B.2 ( Explicit construction of a classical Lyapunov function ) . Suppose V : X → R ≥ 0 satisfies Definition 4.1 and Assumption B.1. Without loss of generality, assume

where f j denotes j -fold composition of the closed-loop map x ↦→ f ( x , π ( x )) . Then W is a classical Lyapunov function:

̸

and it decreases strictly in one step:

̸

Proof. By (31) and (32), we have a 0 = 1 and a j ≥ 0 for all j . Hence

which implies (34). If V is radially unbounded, then so is W by (36).

Let { x k } k ≥ 0 be any closed-loop trajectory and define V k := V ( x k ) . Using (33),

By (32), a j -1 -a j = σ j /M for j = 1 , . . . , M -1 and a M -1 = σ M /M , and by (31) we have a 0 = 1 . Substituting into (37) gives

̸

The right-hand side of (38) is strictly negative for all x k = 0 by (30), proving (35).

Define coefficients and the function

Scaling the weights. If ∑ M i =1 σ i > M , define ˜ σ i := M ∑ j M =1 σ j σ i . Then ˜ σ i ≥ 0 and ∑ M i =1 ˜ σ i = M . Since M ∑ j M =1 σ j ∈ (0 , 1) , the left-hand side of (30) is scaled down while V ( x ) is unchanged, so the strict inequality in (30) remains valid. Hence (31) is without loss of generality.

C Proof of Theorem 4.4

For clarity, we present the detailed proof for the case M = 2 . The general M -step case follows analogously by extending the argument to multiple future steps.

Proof. We consider the generalized Lyapunov decrease condition over two steps:

where σ 1 , σ 2 ≥ 0 and σ 1 + σ 2 ≥ 2 .

Using the discounted Bellman equation for the optimal value function V γ ,

we can recursively expand the two-step generalized decrease condition for V γ . Direct computation yields:

However, due to the presence of the discount factor γ ∈ (0 , 1) , the value function V γ does not naturally satisfy the required decrease condition.

Following the approach in [Postoyan et al., 2017], we introduce an auxiliary quadratic function:

where S 0 ≻ 0 and ϖ > 0 are design parameters.

Define the composite function:

Our goal is to ensure that Y satisfies the generalized Lyapunov decrease condition in (39).

We compute the change in the auxiliary function W ( x )

γ over two steps:

Expanding the system dynamics

and substituting into (44), we express the change in W as

To upper bound this change, we introduce positive definite matrices S 1 , S 2 ≻ 0 and impose:

Combining the changes in V γ and W , we obtain:

Thus, to ensure strict decay of Y γ , it suffices to require:

This leads to the requirement that γ must satisfy:

where α 1 , α 2 > 0 satisfy the comparison inequalities:

Furthermore, by comparing (46) and (47), the auxiliary matrices must satisfy the following LMIs:

This concludes the proof.

D RL Certificates Training Details

D.1 Network Architectures

We use feedforward neural networks with LeakyReLU activations for both the residual network φ ( · , θ 1 ) and step-weighting network σ ( · , θ 2 ) . The architectures are summarized in Table 4. The residual network is initialized using Kaiming initialization [He et al., 2015], and all biases are initialized to zero.

D.2 Training Configuration

We train all networks using the Adam optimizer with an initial learning rate of 5 × 10 -4 , a ReduceLROnPlateau scheduler (factor 0.5, patience 500), and a batch size of 256 for 1000 epochs. Gradients are clipped at 5.0. We set the decay parameter to ¯ α = 0 . 02 and the slack to β = 0 . 01 . As discussed in Remark 5.2, we exclude a small ball around the origin during both training and evaluation. Specifically, we set δ = 0 . 05 for the inverted pendulum and δ = 0 . 5 for the cartpole.

D.3 Compute Resources

All experiments were run on a single workstation with an NVIDIA RTX 4090 GPU, AMD Ryzen 9 7950X CPU, and 64 GB RAM. Training the Lyapunov certificate for the inverted pendulum takes approximately 40 minutes, and for the cartpole environment around 4 hours. These durations include all rollouts and training steps for both the residual and step-weight networks.

E Proof of Theorem 6.2

For clarity, we prove the result for M = 2 and assume σ 1 , σ 2 are constants.

Proof. We restate the assumptions. Let the closed-loop system be:

and V : R n → R ≥ 0 is a continuous function satisfying

with σ 1 , σ 2 > σ > 0 and σ 1 + σ 2 ≥ 2 .

Attractivity. The attractivity proof follows as in Appendix A, we conclude lim k →∞ x k = 0 n .

Stability. We follow the classical ε -δ Lyapunov proof in [Khalil, 1996, Thm. 3.3]. Fix an arbitrary radius ε > 0 and define

Because the ε -sphere E = { x ∈ R n : ∥ x ∥ = ε } is compact and V is continuous, the infimum is achieved on E . Positive-definiteness of V further implies m ε > 0 .

Define

Because V is continuous at the origin and V (0) = 0 , for every threshold η > 0 there exists a radius δ ( η ) > 0 such that

Choosing η = m ε /c and setting δ = δ ( m ε /c ) , gives

Now, pick any index k with x k ∈ S and ∥ x k ∥ < δ . From (58), we have the bound

Using the two-step Lyapunov condition (56) and the fact that σ 1 , σ 2 ≥ σ , we obtain

Combining (60) with (59) gives

By definition of m ε , V ( x ) < m ε implies ∥ x ∥ < ε ; hence

Equation (61) shows that, starting from any state inside the δ -ball, the trajectory remains inside the prescribed ε -ball.

Moreover, due to the strict decrease enforced by (56), it is not possible for both V ( x k +1 ) ≥ V ( x k ) and V ( x k +2 ) ≥ V ( x k ) to hold simultaneously. Therefore, at least one of the two future states must

satisfy V ( x k + i ) < V ( x k ) for i ∈ { 1 , 2 } , hence the trajectory enters a strictly smaller sublevel set of V within two steps, allowing the generalized Lyapunov condition to be re-applied.

We now distinguish two cases to formalize the re-application logic:

Case 1: x k +1 ∈ S . Then the generalized condition can be re-applied at x k +1 , and since V ( x k +1 ) < ϵ , the next steps x k +2 , x k +3 will also lie within the ϵ -ball by repeating the same argument.

Case 2: x k +1 / ∈ S . Then x k +2 ∈ S , and again V ( x k +2 ) < ϵ ⇒∥ x k +2 ∥ < ϵ , allowing the condition to be re-applied at step k +2 .

Conclusion. In both cases, the trajectory remains within the ϵ -ball for all time, and the generalized decrease condition continues to hold along the sequence. Therefore, for all ϵ > 0 , there exists δ > 0 such that:

proving Lyapunov stability of the origin relative to S .

Therefore, we conclude that S is a valid inner approximation of the ROA.

F System Dynamics for Formal Verification

We describe below the continuous-time dynamics, control constraints, and verification domain X for each benchmark system used in our evaluations. All systems are discretized using explicit Euler integration with a time step of ∆ t = 0 . 05 s.

F.1 Inverted Pendulum

The inverted pendulum is modeled as a single-link rigid body rotating about its base, subject to bounded torque control. The continuous-time dynamics are:

where θ is the pendulum angle (upright at θ = 0 ), ω is the angular velocity, and u is the control input. We use the following parameters: gravity g = 9 . 81 m/s 2 , mass m = 0 . 15 kg, length ℓ = 0 . 5 m, and damping coefficient β = 0 . 1 .

The control input is saturated to u ∈ [ -6 , 6] Nm. The state vector is x = [ θ, ω ] ⊤ , and the verification region is defined as:

F.2 Path Tracking

We consider a kinematic model for a ground vehicle tracking a circular path at constant forward speed v . The system state is x = [ d e , θ e ] ⊤ , where d e is the lateral distance to the reference path and θ e is the heading error. The control input u is the steering angle. The continuous-time dynamics are:

where L = 1 mis the wheelbase length, R = 10 mis the radius of the reference path, and v = 2 m/s is the constant forward speed. The control input is bounded by | u | ≤ 0 . 84 .

The verification domain is defined as:

F.3 2D Quadrotor

The 2D quadrotor models a rigid body with six states: horizontal and vertical positions x, z , pitch angle θ , and their corresponding velocities ˙ x, ˙ z, ˙ θ . The system is controlled by two rotor thrusts

u 1 , u 2 , and the continuous-time dynamics are:

where m = 0 . 486 kg, ℓ = 0 . 25 m, I = 0 . 00383 kg · m 2 , and g = 9 . 81 m/s 2 . The state vector is x = [ x, z, θ, ˙ x, ˙ z, ˙ θ ] ⊤ . The control inputs are constrained to be within [0 , 2 . 5 · u eq ] , where u eq = mg 2 . The verification domain is defined as:

F.4 Neural Network Architectures and Training Parameters

We adopt the same network structures used in Yang et al. [2024]. The configurations are summarized in Table 5. All networks use leaky ReLU activations.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately state the paper's two main claims: (1) a stability certificate can be constructed by augmenting the value function of with a residual term, and (2) the classical decrease condition in Lyapunov theory can be relaxed to a generalized multi-step Lyapunov condition. These claims are supported by theoretical analysis and validated through examples and simulation results in Sections 4 to 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are stated with their full assumptions and are either proved in the appendix or directly adapted from cited references with proper attribution.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental details needed to reproduce our results are provided in the paper and appendices. In particular, Appendix D details the training configuration for RL certificate experiments, and Appendix F specifies details for the joint synthesis part.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We open source our code and implementation details.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All training and evaluation details are provided in the Appendix D and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our main results are based on verification of stability conditions, not stochastic evaluations or statistical comparisons. Therefore, notions such as error bars or statistical significance do not apply.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide full compute resource details in Appendix D and in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research does not involve human subjects, sensitive personal data, or potentially harmful applications. We have carefully reviewed the NeurIPS Code of Ethics and confirm that our work adheres to its guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work proposes a theoretical and algorithmic framework for stability certification of optimal control and reinforcement learning policies. It is foundational in nature and does not involve deployment, human interaction, or domain-specific applications that would raise immediate societal concerns.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release any models or datasets with potential for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All external assets used in this work are properly cited. This includes simulation environments from Gymnasium [Towers et al., 2024] and the DeepMind Control Suite [Tassa et al., 2018], as well as baselines and architectures from Yang et al. [2024]. For formal verification, we reference and use tools from Wang et al. [2021], with appropriate credit and citations.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve human subjects or crowdsourcing and therefore does not require IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development does not involve LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a less conservative formulation for certifying stability in learning enabled systems. Traditionally, existence of a Lyapunov function is functional proof of stability. Conditions of a Lyapunov function can be conservative in certain systems, it requires strict decrease w.r.t evolution of the system. Authors relax this condition by allowing increase in the value, but the value should decrease over a horizon.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1-Authors provide a less conservative condition for Lyapunov function, which in my experience, helps when you have multiple integrators in a system.\n\n2-Overall, paper is well written and easy to follow. \n\nWeaknesses:\n\n1- Though authors provide a new formulation for Lyapunov function, this idea is not novel, and has been studied extensively in barrier certificates and software verification. I believe following work deserve citation:\n\nk-Inductive Barrier Certificates for Stochastic Systems, HSCC 2022.\n\nt-Barrier Certificates: A Continuous Analogy to k-Induction, ADHS 2018.\n\nAutomatic synthesis of k-inductive piecewise quadratic invariants for switched affine control programs, CLSS 2017.\n\nSafety Verification and Refutation by k-invariants and k-induction, SAS 2015.\n\nSoftware Verification Using k-Induction, SAS 2011.\n\nI believe if authors frame this problem in a k-inductive manner, it would strengthen the paper.\n\n2- Experiments are not convincing. I do understand that methods that provide formal guarantee do not scale well, however I believe authors could have done a better job at selling their method by providing more experiments. Furthermore, there is no comparison with state-of-the-art methods, such as Yang et al (which this work is heavily inspired by), or other model-based methods (since the verification part requires exact mathematical model of the system). There are works that provide formal guarantee using Lipschitz continuity that have more complicated experiments, and they do not require mathematical model of the system. \n\n3- Contribution seems incremental compared to Yang et al.\n\n4- Bulk of mathematical vigor of the paper is borrowed from previous work, and I am not sure if the proof for Theorem 6.2 is correct (which is the main theoretical contribution of the paper). How does equation 56 imply 62? Moreover, the claim in line 532 by authors does not guarantee ``progress\" towards origin. If the increase is substantially more than decrease, then you cannot say you are making progress towards origin (hence I suggested to use k-induction formulation). \n\n5- There is not discussion on how does verification scale with respect to number of layers or neurons, controllers in the paper are relatively simple (since systems are low-dimensional or simple).\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nContribution seems quite incremental, especially compared to the work mentioned in the weaknesses. \nI am not quite sure how this method scales with higher dimensional systems, since it is model-based, I expect more complicated systems. I also suspect this method does not scale well with number of layers and neurons of the controller.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a method to cetificate the stability of a RL policy by learning a Lyaponv value function based on the value function of RL policy and a resudual network. It also introduce a generalized multi-step fomulation of the lyanapov condition to extend the bound of the disounted factor which guarantees the stability. The paper provides comprehensive mathmatical proof of the proposed theorms and the examples to illustrate the performance of the proposed theory.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses the important problem of bridging the stability guarantees of Lyapunov functions with value functions in RL, helping clarify the gap between provably stable control methods and learning-based approaches.\n\n2. It provides a concrete example by extending the analysis to the LQR setting, illustrating the necessity of the proposed generalized Lyapunov function.\n\n3. The paper presents a complete pipeline to incorporate the generalized Lyapunov function into the RL framework, including the formulation of a loss function and a thorough analysis of experimental results.\n\nWeaknesses:\n\n1. The proposed method is parameterized by M and corresponding weights. It would be helpful to analyze how the grid search range for these weights affects the results. In particular, since there is no explicit upper bound on the sum of weights (which is set equal to M in the experiments), further explanation of this choice is needed.\n\n2. The generalized Lyapunov function allows the value to temporarily increase over short horizons. Is there any theoretical bound on this temporary increase? As M grows, such increases may become more significant—does this affect convergence speed or stability?\n\n3. In the RL application, is the policy trained with a fixed discount factor λ? If so, how do the authors ensure that the resulting policy satisfies the certification condition under that λ? In other words, how can we guarantee that the chosen λ lies within the bounds established by the proposed method?\n\nQUESTIONS:\nThe questions are essentially the same as those listed in the Weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops generalized Lyapunov certificates for verifying stability of reinforcement learning policies. The generalized Lyapunov functions are constructed by augmenting the value function with a neural residual correction term. The method is shown to successfully verify stability of RL policies on Gymnasium and DeepMind control benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Certifying stability of RL policies is an important consideration the RL community has ignored for a long time. In this regard, the authors make a strong and timely contribution.\n2. The authors invoke two clever insights in the development: i) use of generalized Lyapunov condition which requires a Lyapunov decrease only on an average over multiple timesteps instead of a single step, ii) using neural residual error correction term in addition to the value function. These insights solve the difficult problem of certifying stability of RL policies.\n3. The writing is engaging to the reader. The intuition discussing stability of LQR controller with discounted cost provides a strong motivation for their approach.\n\nWeaknesses:\n1. The Lyapunov certificates lack guarantees on robustness to small (bounded) perturbations. For typical RL controllers, robustness to disturbances is a bigger concern that just the stability of the policy. Modifying the generalized Lyapunov condition to account for bounded perturbations and proving stability of a corresponding neighborhood of the origin would make the result stronger. \n2. The discrete-time system in (1) does not account of stochastic effects, e.g., as in Markov processes typically considered in RL problems. Extending the approach to stochastic systems by developing stochastic Lyapunov certificates would make the contribution stronger.\n\nHowever, the strengths far outweigh the weaknesses in my opinion.\n\nQUESTIONS:\nI do not have any additional questions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper addresses the important issue of certifying stable control of deep reinforcement learning policies. It starts by showing that with a small modification, the value function can be used as a Lyapunov function. They then show that stable control can be certified with a relaxed form of the step-wise convergence property, which converges on average.

STRENGTHS AND WEAKNESSES:
Strengths
- Overall, the paper is very clear and easy to follow.
- The paper provides a detailed account of the problem statement and related work before introducing the new method.
- The paper provides strong theoretical foundations for the proposed method


Weaknesses
- This paper only evaluates on a limited set of environments. 
- It would be good to see confidence intervals for the quantitative metrics in Table 2, as you are making comparisons between different finite horizons (M).
- It would be good to see how your proposed method compares to alternative methods for stabilising stability.
- It would be good to see the influence of the step weights rather than only using the weights selected via grid search.


Typography errors
- Can you include brackets for Equations 9, 14, and 18 so it is clear what is being summed over?
- Currently, the Y labels in figures 4, 5 & 6 are displayed as “Theta_dot”, instead of “Theta Dot” or $\dot{\theta}$.

QUESTIONS:
- What do you mean by detectable in Assumption 3.1?
- Could this method be extended to work with higher-dimensional control systems (e.g. Walker Stand)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-01

comments: SUMMARY:
The paper considers identifying a Lyapunov function which is defined as addition of $M$-step value function induced by a policy $\pi$ and a residual term. The Lyapunov function is adopted from the genearlized Lyapunov concept by Furnsinn et al. and the residual term is motivated from solving the discounted LQR problem by Postoyan et al.

STRENGTHS AND WEAKNESSES:
**Strength**
1. Theoretical contribution to the study of discounted LQR: The result of Furnsinn and Postoyan yields relaxed condition on $\gamma$ in Theorem 4.3. 

2. Solid experimental results : The authors show that the proposed method indeed learns a valid Lyapunov function in simple experiments including Cartpole and inverted pendulum problems

**Weakness**

1. Even though deriving a relaxed condition on $\gamma$ is a contribution, it is a result of  Furnsinn and Postoyan rather than proposing a new concept.

2. The required method needs to know the equilibrium state (origin) to define Lyapunov function. In contrast to classical control, RL takes advantages when the equilibrium state is unknown. 

3. The experiments are quite limited to simple control problems, which follows from Weakness 2.

QUESTIONS:
1. Remark 4.4 : The statement would benefit if it presents stronger result than feasibility other than that of classical Lyapunov problem. Can  we guarantee a weaker version of feasibility?

2. How did the authors choose $M=15$ or $M=20$ in the experiments? 

3. What are the final values of $\sigma_i(x)$ in the experiments? Does the final values of $\sigma_i(x)$ provide any further insights?

LIMITATIONS:
yes;
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02