SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At this point, all of the constraints can be linearized as before with the exception of Constraint (3d), which involves the nonlinear dynamics  $f$ . To deal with this, we relax the verification problem by replacing  $f$  with linear lower and upper bounds. To obtain tight bounds, we divide  $\mathcal{R}(\gamma)$  into a collection of subdomains  $\{\mathcal{R}_k\}$ . For each  $\mathcal{R}_k$ , we obtain a linear lower bound  $f_{lb}(x)$  and upper bound  $f_{ub}(x)$  on  $f$ , and relax the problematic Constraint (3d) into  $f_{lb}(x) \leq \bar{z}_0 \leq f_{ub}(x)$ , which is now a pair of linear constraints. We can then solve Problem (3) for each  $\mathcal{R}_k$ .
Computing Linear Bounds on System Dynamics Recall that  $f: \mathbb{R}^n \mapsto \mathbb{R}^n$  is the Lipschitz-continuous system dynamic  $x_{t+1} = f(x_t, u_t)$ . For simplicity we omit  $u_t = \pi_\beta(x)$  below. Let  $\lambda$  be the  $\ell_\infty$  Lipschitz constant of  $f$ . Suppose that we are given a region of the domain  $\mathcal{R}_k$  represented as a hyperrectangle, i.e.,  $\mathcal{R}_k = \times_i[x_{i,l}, x_{i,u}]$ , where  $[x_{i,l}, x_{i,u}]$  are the lower and upper bounds of  $x$  along coordinate  $i$ . Our goal is to compute a linear upper and lower bound on  $f(x)$  over this region. We bound  $f_j(x): \mathbb{R}^n \mapsto \mathbb{R}$  along each  $j$ -th dimension separately. By  $\lambda$ -Lipschitz-continuity, we can obtain  $f_j(x) \leq f_j(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) + \lambda \sum_i (x_i - x_{i,l})$  and  $f_j(x) \geq f_j(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) - \lambda \sum_i (x_i - x_{i,l})$ . The full derivation is provided in the Supplement Section B.
Alternatively, if  $f_{j}$  is differentiable, convex over  $x_{i}$ , and monotone over other dimensions, we can restrict it to calculating one-dimensional bounds by finding coordinates  $x_{-i,u}$  and  $x_{-i,l}$  such that  $f_{j}(x_{i},x_{-i,l})\leq f_{j}(x_{i},x_{-i})\leq f_{j}(x_{i},x_{-i,u})$  for any given  $x$ . Then  $f_{j}(x)\leq f_{j,i,ub}(x)\equiv f_{j}(x_{i,l},x_{-i,u}) + \frac{f_{j}(x_{i,u},x_{-i,u}) - f_{j}(x_{i,l},x_{-i,u})}{x_{i,u} - x_{i,l}} (x_{i} - x_{i,l})$  and  $f_{j}(x)\geq f_{i,lb}(x)\equiv f_{j}(x_{i}^{*},x_{-i,l})+$
$f_{j}^{\prime}(x_{i}^{*},x_{-i,l})(x_{i} - x_{i}^{*})$ , where we let  $x_{i}^{*} = \min_{x_{s}\in [x_{i,l},x_{i,u}]} \int_{x_{i,l}}^{x_{i,u}}[f_{j}(x_{i},x_{-i,l}) - f_{j}(x_{s},x_{-i,l}) +$ $f_{j}^{\prime}(x_{s},x_{-i,l})(x_{i} - x_{s})]dx_{s}$  to minimize the error area between  $f_{j}(x)$  and  $f_{i,lb}(x)$ . Note that the solution for  $x_{i}^{*}$  can be approximated when conducting experiments, and this approximation has no impact on the correctness of the bound. A similar result obtains for where  $f_{j}(x)$  is concave over  $x_{i}$ . Furthermore, we can use these single-dimensional bounds to obtain tighter lower and upper bounds on  $f_{j}(x)$  as follows: note that for any  $c^l,c^u$  with  $\sum_i c_i^l = 1$  and  $\sum_i c_i^u = 1$ ,  $\sum_i c_i^lf_{j,i,lb}(x)\leq f_j(x)\leq \sum_i c_i^uf_{j,i,ub}(x)$ , which means we can optimize  $c^l$  and  $c^u$  to minimize the bounding error. In practice, we can typically partition  $\mathcal{R}(\gamma)$  so that the stronger monotonicity and convexity or concavity assumptions hold for each  $\mathcal{R}_k$ .
Note that our linear bounds  $f_{lb}(x)$  and  $f_{ub}(x)$  introduce errors compared to the original nonlinear dynamics  $f$ . However, we can obtain tighter bounds by splitting  $\mathcal{R}_k$  further into subregions, and computing tighter bounds in each of the resulting subregions, but at the cost of increased computation. To balance these considerations, we start with a relatively small collection of subdomains  $\{\mathcal{R}_k\}$ , and only split a region  $\mathcal{R}_k$  if we obtain a counterexample in  $\mathcal{R}_k$  that is not an actual counterexample for the true dynamics  $f$ .
Computing Bounds on ReLU Linearization Constants In linearizing the ReLU activations, we supposed an existence of lower and upper bounds  $L$  and  $U$ . However, we cannot simply set them to some large negative and positive number, respectively, because  $V_{\theta}(x)$  has no a priori fixed bounds (in particular, note that for any  $\epsilon$ -Lyapunov function  $V$  and constant  $a > 1$ ,  $aV$  is also a  $\epsilon$ -Lyapunov function). Thus, arbitrarily setting  $L$  and  $U$  makes verification unsound. To address this issue, we use interval bound propagation (IBP) [Gowal et al., 2018] to obtain  $M = \max_{1 \leq i \leq n} \{|U_i|, |L_i|\}$ , where  $U_i$  is the upper bound, and  $L_i$  is the lower bound returned by IBP for the  $i$ -th layer, with inputs for the first layer the upper and lower bounds of  $f(\mathcal{R}(\gamma))$ . Setting each  $L = -M$  and  $U = M$  then yields sound verification.
Computing Sublevel Sets The approaches above verify the conditions (a)-(c) of  $\epsilon$ -stability on  $\mathcal{R}(\gamma)$ . The final piece is to find the  $\mathcal{R}(\gamma)$ -invariant sublevel set  $\mathcal{D}(\gamma, \rho)$ , that is, to find  $\rho$ . Let  $B(\gamma) \geq \max \left(\max_{x \in \mathcal{R}(\gamma)} \|f(x, \pi_{\beta}(x))\|_{\infty}, \gamma\right)$ . We find  $\rho$  by solving
$$
\min  _ {x: \gamma \leq \| x \| _ {\infty} \leq B (\gamma)} V (x). \tag {4}
$$
We can transform both Problem (4) and computation of  $B(\gamma)$  into MILP as for other problems above.
Theorem 3.1. Suppose that  $V$  and  $\pi$  are  $\epsilon$ -stable on  $\mathcal{R}(\gamma)$ ,  $\| \pi(0) - u_0 \|_{\infty} \leq \epsilon$ , and  $\gamma \geq L_f \max \{1, L_\pi + 1\} \epsilon$ . Let  $V^*$  be the optimal value of the objective in Problem (4), and  $\rho = V^* - \mu$  for any  $\mu > 0$ . Then the set  $\mathcal{D}(\gamma, \rho) = \{x : x \in \mathcal{R}(\gamma), V(x) \leq \rho\}$  is an  $\mathcal{R}(\gamma)$ -invariant sublevel set.
Proof. If  $\| x \|_{\infty} > \epsilon$ ,  $V(f(x, \pi(x))) < V(x) \leq \rho < V^{*}$  by  $\epsilon$ -stability of  $V$  and  $\pi$ . Suppose that  $\bar{x} = f(x, \pi(x)) \notin \mathcal{R}(\gamma)$ . Since  $V(\bar{x}) < V^{*}$ ,  $V^{*}$  must not be an optimal solution to (4), a contradiction. If  $\| x \|_{\infty} \leq \epsilon$ , the argument is similar to Theorem 2.2 (ii).
4 Learning  $\epsilon$ -Lyapunov Function and Policy
We cast learning the  $\epsilon$ -Lyapunov function and policy as the following problem:
$$
\min  _ {\theta , \beta} \sum_ {i \in S} \mathcal {L} \left(x _ {i}; V _ {\theta}, \pi_ {\beta}\right), \tag {5}
$$
where  $\mathcal{L}(\cdot)$  is a loss function that promotes Lyapunov learning and the set  $S\subseteq \mathcal{R}(\gamma)$  is a finite subset of points in the valid region. We assume that the loss function is differentiable, and consequently training follows a sequence of gradient update steps  $(\theta^{\prime},\beta^{\prime}) = (\theta ,\beta) - \mu \sum_{i\in S}\nabla_{\theta ,\beta}\mathcal{L}(x_i;V_\theta ,\pi_\beta)$ .
Clearly, the choice of the set  $S$  is pivotal to successfully learning  $V_{\theta}$  and  $\pi_{\beta}$  with provable stability properties. Prior approaches for learning Lyapunov functions represented by neural networks use one of two ways to generate  $S$ . The first is to generate a fixed set  $S$  comprised of uniformly random samples from the domain [Chang and Gao, 2021, Richards et al., 2018]. However, this fails to learn verifiable Lyapunov functions. In the second,  $S$  is not fixed, but changes in each learning iteration, starting with randomly generated samples in the domain, and then using the verification tool, such as dReal, in each step of the learning process to update  $S$  [Chang et al., 2019, Zhou et al., 2022]. However, verification is often slow, and becomes a significant bottleneck in the learning process.
Our key idea is to combine the benefits of these two ideas with a fast gradient-based heuristic approach for generating counterexamples. In particular, our proposed approach for training Lyapunov control functions and associated control policies (Algorithm 1; DITL) involves five parts:
1. heuristic counterexample generation (lines 10-12),
2. choosing a collection  $S$  of inputs to update  $\theta$  and  $\beta$  in each training (gradient update) iteration (lines 9-13),
3. the design of the loss function  $\mathcal{L}$
4. initialization of policy  $\pi_{\beta}$  (line 3), and
5. warm starting the training (line 4).
We describe each of these next.
Algorithm 1 DITL Lyapunov learning algorithm.
1: Input: Dynamical system model  $f(x,u)$  and target valid region  $\mathcal{R}(\gamma)$   
2: Output: Lyapunov function  $V_{\theta}$  and control policy  $\pi_{\beta}$   
3:  $\pi_{\beta} = \text{Initialize}()$   
4:  $V_{\theta} = \text{PreTrainLyapunov}(N_0, \pi_{\beta})$   
5:  $B = \text{InitializeBuffer}()$   
6:  $W \gets \emptyset$   
7: while True do  
8: for  $N$  iterations do  
9:  $\hat{S} = \text{Sample}(r,B) // \text{sample } r$  points from  $B$   
10:  $T = \text{Sample}(q, \mathcal{R}(\gamma)) // \text{sample } q$  points from  $\mathcal{R}(\gamma)$   
11:  $T' = \text{PGD}(T)$   
12:  $T'' = \text{FilterCounterExamples}(T')$   
13:  $S = \hat{S} \cup T'' \cup W$   
14:  $B = B \cup T''$   
15:  $(\theta, \beta) \gets (\theta, \beta) - \mu \sum_{i \in S} \nabla_{\theta, \beta} \mathcal{L}(x_i; V_{\theta}, \pi_{\beta})$   
16: end for  
17: (success,  $\hat{W}$ ) = Verify  $(V_{\theta}, \pi_{\beta})$   
18: if success then  
19: return  $V_{\theta}, \pi_{\beta}$   
20: else  
21:  $W \gets W \cup \hat{W}$   
22: end if  
23: end while  
24: returnFAILED // Timeout
Heuristic Counterexample Generation As discussed in Section 3, verification in general involves two optimization problems:  $\min_{x\in \mathcal{R}(\gamma)}V_{\theta}(x) - V_{\theta}(f(x,\pi_{\beta}(x)))$  and  $\min_{x\in \mathcal{R}(\gamma)}V_{\theta}(x)$ . We propose a projected gradient descent (PGD) approach for approximating solutions to either of these problems. PGD proceeds as follows, using the second minimization problem as an illustration; the process is identical in the first case. Beginning with a starting point  $x_0\in \mathcal{R}(\gamma)$ , we iteratively update  $x_{k + 1}$ :
$$
x _ {k + 1} = \Pi \left\{x _ {k} + \alpha_ {k} \operatorname {s g n} \left(\nabla V _ {\theta} (x _ {k})\right) \right\},
$$
where  $\Pi\{\}$  projects the gradient update step onto the domain  $\mathcal{R}(\gamma)$  by clipping each dimension of  $x_{k}$ , and  $\alpha_{k}$  is the PGD learning rate. Note that as  $f$  is typically differentiable, we can take gradients directly through it when we apply PGD for the first verification problem above.
We run this procedure for  $K$  iterations, and return the result (which may or may not be an actual counterexample). Let  $T' = \mathrm{PGD}(T)$  denote running PGD for a set of  $T$  starting points for both of the optimization problems above, resulting in a corresponding set  $T'$  of counterexamples.
Selecting Inputs for Gradient Updates A natural idea for selecting samples  $S$  is to simply add counterexamples identified in each iteration of the Lyapunov learning process. However, as  $S$  then grows without bound, this progressively slows down the learning process. In addition, it is important for  $S$  to contain a relatively broad sample of the valid region to ensure that counterexamples we generate along the way do not cause undesired distortion of  $V_{\theta}$  and  $\pi_{\beta}$  in subregions not well represented in  $S$ . Finally, we need to retain the memory of previously generated counterexamples, as
these are often particularly challenging parts of the input domain for learning. We therefore construct  $S$  as follows.
We create an input buffer  $B$ , and initialize it with a random sample of inputs from  $\mathcal{R}(\gamma)$ . Let  $W$  denote a set of counterexamples that we wish to remember through the entire training process, initialized to be empty. In our implementation,  $W$  consists of all counterexamples generated by the sound verification tools described in Section 3.
At the beginning of each training update iteration, we randomly sample a fixed-size subset  $\hat{S}$  of the buffer  $B$ . Next, we take a collection  $T$  of random samples from  $\mathcal{R}$  to initialize PGD, and generate  $T' = \mathrm{PGD}(T)$ . We then filter out all the counterexamples from  $T'$ , retaining only those with either  $V_{\theta}(x) \leq 0$  or  $V_{\theta}(x) - V_{\theta}(f(x, \pi_{\beta}(x))) \leq \epsilon$ , where  $\epsilon$  is a non-negative hyperparameter; this yields a set  $T''$ . We then use  $S = \hat{S} \cup T'' \cup W$  for the gradient update in the current iteration. Finally, we update the buffer  $B$  by adding to it  $T''$ . This process is repeated for  $N$  iterations.
After  $N$  iterations, we check to see if we have successfully learned a stable policy and a Lyapunov control function by using the tools from Section 3 to verify  $V_{\theta}$  and  $\pi_{\beta}$ . If verification succeeds, training is complete. Otherwise, we use the counterexamples  $\hat{W}$  generated by the MILP solver to update  $W = W \cup \hat{W}$ , and repeat the learning process above.
Loss Function Design Next, we design of the loss function  $\mathcal{L}(x;V_{\theta},\pi_{\beta})$  for a given input  $x$ . A key ingredient in this loss function is a term that incentivizes the learning process to satisfy condition (b) of  $\epsilon$ -stability:  $\mathcal{L}_1(x;V_\theta ,\pi_\beta) = \mathrm{ReLU}(V_\theta (f(x,\pi_\beta (x))) - V_\theta (x) + \eta)$ , where the parameter  $\eta \geq 0$  determines how aggressively we try to satisfy this condition during learning.
There are several options for learning  $V_{\theta}$  satisfying conditions (a) and (c) of  $\epsilon$ -stability. The simplest is to set all bias terms in the neural network to zero, which immediately satisfies  $V_{\theta}(0) = 0$ . An effective way to deal with condition (c) is to maximize the lower bound on  $V_{\theta}(x)$ . To this end, we propose to make use of the following loss term:  $\mathcal{L}_2(x;V_\theta ,\pi_\beta) = \mathrm{ReLU}(-V_\theta^{LB})$ , where  $V_{\theta}^{LB}$  is the lower bound on the Lyapunov function over the target domain  $\mathcal{R}(\gamma)$ . We use use  $\alpha, \beta$ -CROWN [Xu et al., 2020] to obtain this lower bound.
The downside of setting bias terms to zero is that we lose many learnable parameters, reducing flexibility of the neural network. If we consider a general neural network, on the other hand, it is no longer the case that  $V_{\theta}(x) = 0$  by construction. However, it is straightforward to ensure this by defining the final Lyapunov function as  $\tilde{V}_{\theta}(x) = V_{\theta}(x) - V_{\theta}(0)$ . Now, satisfying condition (c) amounts to satisfying the condition that  $V_{\theta}(x) \geq V_{\theta}(0)$ , which we accomplish via the following pair of loss terms:  $\mathcal{L}_3(x;V_\theta ,\pi_\beta) = \mathrm{ReLU}(V_\theta (0) - V_\theta (x) + \mu \min (\| x\| _2,\nu))$ , where  $\mu$  and  $\nu$  are hyperparameters of the term  $\mu \min (\| x\| _2,\nu)$  which effectively penalizes  $V_{\theta}(x)$  for being too large, and  $\mathcal{L}_4(x;V_\theta ,\pi_\beta) = \| V_\theta (0)\| _2^2$ .
The general loss function is then a weighted sum of these loss terms,
$$
\mathcal {L} (x; V _ {\theta}, \pi_ {\beta}) = \mathcal {L} _ {1} (x; V _ {\theta}, \pi_ {\beta}) + c _ {2} \mathcal {L} _ {2} (x; V _ {\theta}, \pi_ {\beta}) + c _ {3} \mathcal {L} _ {3} (x; V _ {\theta}, \pi_ {\beta}) + c _ {4} \mathcal {L} _ {4} (x; V _ {\theta}, \pi_ {\beta}).
$$
When we set bias terms to zero, we would set  $c_{3} = c_{4} = 0$ ; otherwise, we set  $c_{2} = 0$ .
Initialization We consider two approaches for initializing the policy  $\pi_{\beta}$ . The first is to linearize the dynamics  $f$  around the origin, and use a policy computing based on a linear quadratic regulator (LQR) to obtain a simple linear policy  $\pi_{\beta}$ . The second approach is to use deep reinforcement, such as PPO [Liu et al., 2019], where rewards correspond to stability (e.g., reward is the negative value of the  $l_{2}$  distance of state to origin). To initialize  $V_{\theta}$ , we fix  $\pi_{\beta}$  after its initialization, and follow our learning procedure above using solely heuristic counterexample generation to pre-train  $V_{\theta}$ .
The next result now follows by construction.
Theorem 4.1. If Algorithm 1 returns  $V_{\theta}, \pi_{\beta}$ , they are guaranteed to satisfy  $\epsilon$ -stability conditions.
Benchmark Domains Our evaluation of the proposed DITL approach uses four benchmark control domains: inverted pendulum, path tracking, cartpole, and drone planar vertical takeoff and landing (PVTOL). Details about these domains are provided in the Supplement.
Baselines We compare the proposed approach to four baselines: linear quadratic regulator (LQR), sum-of-squares (SOS), neural Lyapunov control (NLC) [Chang et al., 2019], and neural Lyapunov control for unknown systems (UNL) [Zhou et al., 2022]. The first two, LQR and SOS, are the more traditional approaches for computing Lyapunov functions when the dynamics are either linear (LQR) or polynomial (SOS) [Tedrake, 2009]. LQR solutions (when the system can be stabilized) are obtained through matrix multiplication, while SOS entails solving a semidefinite program (we solve it using YALMIP with MOSEK solver in MATLAB 2022b). The next two baselines, NLC and UNL, are recent approaches for learning Lyapunov functions in continuous-time systems (no approach for learning provably stable control using neural network representations exists for discrete time systems). Both NLC and UNL yield provably stable control for general non-linear dynamical systems, and are thus the most competitive baselines to date. In addition to these baselines, we consider an ablation in which PGD-based counterexample generation for our approach is entirely replaced by the sound MILP-based method during learning (we refer to it as DITL-MILP).
Efficacy Metrics We compare approaches in terms of three efficacy metrics. The first is (serial) runtime (that is, if no parallelism is used), which we measure as wall clock time when only a single task is running on a machine. For inverted pendulum and path tracking, all comparisons were performed on a machine with AMD Ryzen 9 5900X 12-Core Processor and Linux Ubuntu 20.04.5 LTS OS. All cartpole and PVTOL experiments were run on a machine with a Xeon Gold 6150 CPU (64-bit 18-core x86), Rocky Linux 8.6. UNL and RL training for Path Tracking are the only two cases that make use of GPUs, and was run on NVIDIA GeForce RTX 3090. The second metric was the size of the valid region, measured using  $\ell_2$  norm for NLC and UNL, and  $\ell_{\infty}$  norm (which dominates  $\ell_2$ ) for LQR, SOS, and our approach. The third metric is the region of attraction (ROA). Whenever verification fails, we set ROA to 0. Finally, we compare all methods whose results are stochastic (NLC, UNL, and ours) in terms of success rate.
Verification Details For LQR, SOS, NLC, and UNL, we used dReal as the verification tool, as done by Chang et al. [2019] and Zhou et al. [2022]. For DITL verification we used CPLEX version 22.1.0.
□
Our Approach
#
NLC
SOS
#
LQR
Valid Region
Figure 1: ROA plot of inverted pendulum (left) and path tracking (right). We select the best result for each method.
Inverted Pendulum For the inverted pendulum domain, we initialize the control policy using the LQR solution (see the Supplement for details). We train  $V_{\theta}$  with non-zero bias terms. We set  $\epsilon = 0.1 (< 0.007\%$  of the valid region), and approximate ROA using a grid of 2000 cells along each coordinate using the level set certified with MILP (4). All runtime is capped at 600 seconds, except the UNL baseline, which we cap at 16 minutes, as it tends to be considerably slower than other methods. Our results are presented in Table 1. While LQR and SOS are fastest, our approach (DITL) is the next fastest, taking on average  $\sim 8$  seconds, with NLC and UNL considerably slower. However, DITL yields an ROA  $a factor > 4$  larger than the nearest baseline (LQR), and  $100\%$  success rate. Finally, the DITL-MILP ablation is two orders of magnitude slower (runtime  $> 300$  seconds) and less effective (average ROA  $= 42$ ,  $80\%$  success rate) than DITL. We visualize maximum ROA produced by all methods in Figure 1 (left).
**Path Tracking** In path tracking, we initialize our approach using both the RL and LQR solutions, drawing a direct comparison between the two (see the Supplement for details). We set  $\epsilon = 0.1$ . The running time of RL is  $\sim 155$  seconds. The results are provided in Table 2. We can observe that both RL- and LQR-initialized variants of our approach outperform all prior art, with RL exhibiting  $a$
Table 1: Inverted Pendulum
Valid Region	Runtime (s)	ROA	Max ROA	Success Rate
NLC (free)	||x||2≤ 6.0	28 ± 29	11 ± 4.6	22	100%
NLC (max torque 6.0)	||x||2≤ 6.0	519 ± 184	13 ± 27	66	20%
UNL (max torque 6.0)	||x||2≤ 4.0	821 ± 227	1 ± 2	7	30%
LQR	||x||∞≤ 5.8	< 1	14	14	success
SOS	||x||∞≤ 1.7	< 1	6	6	success
DITL	||x||∞≤ 12	8.1 ± 4.7	61 ± 31	123	100%
Table 2: Path Tracking
Valid Region	Runtime (s)	ROA	Max ROA	Success Rate
NLC	||x||2≤1.0	109±81	0.5±0.2	0.76	100%
NLC	||x||2≤1.5	151±238	1.4±0.9	2.8	80%
UNL	||x||2≤0.8	925±110	0.1±0.2	0.56	10%
LQR	||x||∞≤0.7	<1	1.02	1.02	success
SOS	||x||∞≤0.8	<1	1.8	1.8	success
DITL (LQR)	||x||∞≤3.0	9.8±4	8±3	12.5	100%
DITL (RL)	||x||∞≤3.0	14±11	9±3.5	16	100%
factor of 5 advantage over the next best (SOS, in this case) in terms of ROA, and nearly a factor of 6 advantage in terms of maximum achieved ROA (NLC is the next best in this case). Moreover, our approach again has a  $100\%$  success rate. Our runtime is an order of magnitude lower than NLC or UNL. Overall, the RL-initialized variant slightly outperforms LQR initialization. The DITL-MILP ablation again performs far worse than DITL: running time is several orders of magnitude slower (at  $>550$  seconds), with low efficacy (ROA is 1.1, success rate  $10\%$ ). We visualize comparison of maximum ROA produced by all methods in Figure 1 (right).
Cartpole For cartpole, we used LQR for initialization, and set bias terms of  $V_{\theta}$  to zero. We set  $\epsilon = 0.1$  (0.01% of the valid region area) and the running time limit to 2 hours for all approaches. None of the baselines successfully attained a provably stable control policy and associated Lyapunov function for this problem. Failure was either because we could find counterexamples within the target valid region, or because verification exceeded the time limit. DITL found a valid region of  $\| x \|_{\infty} \leq 1.0$ , in  $\leq 1.6$  hours with a 100% success rate, and average ROA of 0.021 ± 0.012.
PVTOL The PVTOL setup was similar to cartpole. We set  $\epsilon = 0.1$  (0.0001% of the valid region area), and maximum running time to 24 hours. None of the baselines successfully identified a provably stable control policy. In contrast, DITL found one within  $13 \pm 6$  hours on average, yielding a 100% success rate. We identified a valid region of  $\|x\|_{\infty} \leq 1.0$ , and ROA of  $0.011 \pm 0.008$ .
We presented a novel algorithmic framework for learning Lyapunov control functions and policies for discrete-time nonlinear dynamical systems. Our approach combines mixed-integer linear programming verification tool with a training procedure that leverages gradient-based approximate verification. This combination enables a significant improvement in effectiveness compared to prior art: our experiments demonstrate that our approach yields several factors larger regions of attraction in inverted pendulum and path tracking, and ours is the only approach that successfully finds stable policies in cartpole and PVTOL.
This research was partially supported by the NSF (grants CNS-1941670, ECCS-2020289, IIS-1905558, IIS-2214141, and CNS-2231257), AFOSR (grant FA9550-22-1-0054), ARO (grant W911NF-19-1-0241), and NVIDIA.