SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junlin Wu
St. Louis, MO 63130
junlin.wu@wustl.edu
Electrical & Systems Engineering
andrewclark@wustl.edu
Yiannis Kantaros
ioannisk@wustl.edu
Yevgeniy Vorobeychik
yvorobeychik@wustl.edu
While ensuring stability for linear systems is well understood, it remains a major challenge for nonlinear systems. A general approach in such cases is to compute a combination of a Lyapunov function and an associated control policy. However, finding Lyapunov functions for general nonlinear systems is a challenging task. To address this challenge, several methods have been proposed that represent Lyapunov functions using neural networks. However, such approaches either focus on continuous-time systems, or highly restricted classes of nonlinear dynamics. We propose the first approach for learning neural Lyapunov control in a broad class of discrete-time systems. Three key ingredients enable us to effectively learn provably stable control policies. The first is a novel mixed-integer linear programming approach for verifying the discrete-time Lyapunov stability conditions, leveraging the particular structure of these conditions. The second is a novel approach for computing verified sublevel sets. The third is a heuristic gradient-based method for quickly finding counterexamples to significantly speed up Lyapunov function learning. Our experiments on four standard benchmarks demonstrate that our approach significantly outperforms state-of-the-art baselines. For example, on the path tracking benchmark, we outperform recent neural Lyapunov control baselines by an order of magnitude in both running time and the size of the region of attraction, and on two of the four benchmarks (cartpole and PVTOL), ours is the first automated approach to return a provably stable controller. Our code is available at: https://github.com/jlwu002/nlc_discrete.
Stability analysis for dynamical systems aims to show that the system state will return to an equilibrium under small perturbations. Designing stable control in nonlinear systems commonly relies on constructing Lyapunov functions that can certify the stability of equilibrium points and estimate their region of attraction. However, finding Lyapunov functions for arbitrary nonlinear systems is a challenging task that requires substantial expertise and manual effort [Khalil, 2015, Lavaei and Bridgeman, 2023]. To address this challenge, recent progress has been made in learning Lyapunov functions in continuous-time nonlinear dynamics [Abate et al., 2020, Chang et al., 2019, Zhou et al., 2022]. However, few approaches exist for doing so in discrete-time systems [Dai et al., 2021], and none for general Lipschitz-continuous dynamics. Since modern learning-based controllers often
take non-negligible time for computation, the granularity of control is effectively discrete-time, and developing approaches for stabilizing such controllers is a major open challenge.
We propose a novel method to learn Lyapunov functions and stabilizing controllers, represented as neural networks (NNs) with ReLU activation functions, for discrete-time nonlinear systems. The proposed framework broadly consists of a learner, which uses a gradient-based method for updating the parameters of the Lyapunov function and policy, and a verifier, which produces counterexamples (if any exist) to the stability conditions that are added as training data for the learner. The use of a verifier in the learning loop is critical to enable the proposed approach to return a provably stable policy. However, no prior approaches enable sound verification of neural Lyapunov stability for general discrete-time dynamical systems. The closest is Dai et al. [2021], who assume that dynamics are represented by a neural network, an assumption that rarely holds in real systems. On the other hand, approaches for continuous-time systems [Chang et al., 2019, Zhou et al., 2022] have limited efficacy in discrete-time environments, as our experiments demonstrate. To address this gap, we develop a novel verification tool that checks if the candidate NN Lyapunov function satisfies the Lyapunov conditions by solving Mixed Interger Linear Programs (MILPs). This approach, which takes advantage of the structure of discrete-time Lyapunov stability conditions, can soundly verify a broad class of dynamical system models.
However, using a sound verification tool in the learning loop makes it a significant bottleneck, severely limiting scalability. We address this problem by also developing a highly effective gradient-based technique for identifying counterexamples, resorting to the full MILP-based verifier only as a last resort. The full learning process thereby iterates between learning and verification steps, and returns only when the sound verifier is able to prove that the Lyapunov function and policy satisfy the stability conditions.
The final technical challenge stems from the difficulty of verifying stability near the origin [Chang et al., 2019], typically addressed heuristically by either adding a fixed tolerance to a stability condition [Dai et al., 2021], or excluding a small area around the origin from verification [Chang et al., 2019, Zhou et al., 2022]. We address it by adapting Lyapunov stability theory to ensure convergence to a small region near the origin, thereby achieving the first (to our knowledge) sound approach for computing a stable neural controller that explicitly accounts for such approximations near the origin.
We evaluate the proposed approach in comparison to state-of-the-art baselines on four standard nonlinear control benchmarks. On the two simpler domains (inverted pendulum and path following), our approach outperforms the state-of-the-art continuous-time neural Lyapunov control approaches by at least several factors and up to an order of magnitude in both running time and the size of the region of attraction. On the two more complex domains—cartpole and PVTOL—ours is the first automated approach that returns a provably stable controller. Moreover, our ablation experiments demonstrate that both the MILP-based verifier and heuristic counterexample generation technique we propose are critical to the success of our approach.
- A novel MILP-based approach for verifying a broad class of discrete-time controlled nonlinear systems.
- A novel approach for learning provably verified stabilizing controllers for a broad class of discrete-time nonlinear systems which combines our MILP-based verifier with a heuristic gradient-based approximate counterexample generation technique.
- A novel formalization of approximate stability in which the controller provably converges to a small ball near the origin in finite time.
- Extensive experiments on four standard benchmarks demonstrate that by leveraging the special structure of Lyapunov stability conditions for discrete-time system, our approach significantly outperforms prior art.
Related Work Much prior work on learning Lyapunov functions focused on continuous-time systems [Abate et al., 2020, Ravanbakhsh and Sankaranarayanan, 2019, Chang et al., 2019, Zhou et al., 2022]. Common approaches have assumed that dynamics are linear [Donti et al., 2021, Tedrake, 2009] or polynomial [Ravanbakhsh and Sankaranarayanan, 2019]. Recently, Chang et al. [2019], Rego and de Araujo [2022], and Zhou et al. [2022] proposed learning Lyapunov functions represented as neural networks while restricting policies to be linear. These were designed for continuous-time dynamics, and are not effective in discrete-time settings, as we show in the experiments. Chen et al.
[2021a] learns convex Lyapunov functions for discrete-time hybrid systems. Their approach requires hybrid systems to admit a mixed-integer linear programming formulation, essentially restricting it to piecewise-affine systems, and does not learn stabilizing controllers for these. Our approach considers a much broader class of dynamical system models, and learns provably stable controllers and Lyapunov functions, allowing both to be represented as neural networks. Kolter and Manek [2019] learn stable nonlinear dynamics represented as neural networks, but do not provide stability guarantees with respect to the true underlying system. In addition, several approaches have been proposed that either provide only probabilistic stability guarantees [Berkenkamp et al., 2017, Richards et al., 2018], or do not guarantee stability [Choi et al., 2020, Han et al., 2020]. Several recent approaches propose methods for verifying stability in dynamical discrete-time systems. Chen et al. [2021b] compute an approximate region of attraction of dynamical systems with neural network dynamics, but assume that Lyapunov functions are quadratic. Dai et al. [2021] consider the problem of verifying Lyapunov conditions in discrete-time systems, as well as learning provably stable policies. However, they assume that dynamics are represented by neural networks with (leaky) ReLU activation functions. Our verification approach, in contrast, is for arbitrary Lipschitz continuous dynamics.
We consider a discrete-time nonlinear dynamical system
$$
x _ {t + 1} = f \left(x _ {t}, u _ {t}\right), \tag {1}
$$
where  $x_{t} \in \mathcal{X}$  is state in a domain  $\mathcal{X}$  and  $u_{t}$  the control input at time  $t$ . We assume that  $f$  is Lipschitz continuous with Lipschitz constant  $L_{f}$ . This class of dynamical systems includes the vast majority of (non-hybrid) dynamical system models in prior literature. We assume that  $x = 0$  is an equilibrium point for the system, that is,  $f(0, u_0) = 0$  for some  $u_0$ . Let  $\pi(x)$  denote a control policy, with  $u_{t} = \pi(x_{t})$ . For example, in autonomous vehicle path tracking,  $x$  can measure path tracking error and  $u$  the steering angle. In the conventional setup, the goal is to learn a control policy  $\pi$  such that the dynamical system in Equation (1) converges to the equilibrium  $x = 0$ , a condition referred to as stability. To this end, we can leverage the Lyapunov stability framework [Tedrake, 2009]. Specifically, the goal is to identify a Lyapunov function  $V(x)$  and controller  $\pi$  that satisfies the following conditions over a subset of the domain  $\mathcal{R} \subseteq \mathcal{X}$ : 1)  $V(0) = 0$ ; 2)  $V(x) > 0$ ,  $\forall x \in \mathcal{R} \setminus \{0\}$ ; and 3)  $V(f(x, \pi(x))) - V(x) < 0, \forall x \in \mathcal{R}$ . These conditions imply that the system is (locally) stable in the Lyapunov sense [Bof et al., 2018, Tedrake, 2009].
In practice, due to numerical challenges in verifying stability conditions near the origin, it is common to verify slight relaxations of the Lyapunov conditions. These typically fall into two categories: 1) Lyapunov conditions are only checked for  $\| x\| _p\geq \epsilon$  [Chang et al., 2019, Zhou et al., 2022] (our main baselines, and the only other methods for learning neural Lyapunov control for general nonlinear dynamics), and/or 2) a small tolerance  $\delta >0$  is added in verification, allowing small violations of Lyapunov conditions near the origin [Dai et al., 2021].
Our goal now is to formally investigate the implications of numerical approximations of this kind. We next define a form of stability which entails finite-time convergence to  $\mathcal{B}(0,\epsilon) = \{x\mid \| x\|_{\infty} < \epsilon \}$ .
Definition 2.1 ( $\epsilon$ -stability). We call pair of  $(V, \pi)$ $\epsilon$ -stable within a region  $\mathcal{R}$  when the following conditions are satisfied: (a)  $V(0) = 0$ ; (b) there exists  $\zeta > 0$  such that  $V(f(x, \pi(x))) - V(x) < -\zeta$  for all  $x \in \mathcal{R} \setminus \mathcal{B}(0, \epsilon)$ ; and (c)  $V(x) > 0$  for all  $x \in \mathcal{R} \setminus \mathcal{B}(0, \epsilon)$ .
A key ingredient to achieving stability is to identify a region of attraction (ROA), within which we are guaranteed to converge to the origin from any starting point. In the context of  $\epsilon$ -stability, our goal is to converge to a small ball near the origin, rather than the origin; let  $\epsilon$ -ROA be the set of initial inputs that has this property. In order to enable us to identify an  $\epsilon$ -ROA, we introduce a notion of an invariant sublevel set. Specifically, we refer to a set  $\mathcal{D}(\mathcal{R}, \rho) = \{x \in \mathcal{R} | V(x) \leq \rho\}$  with the property that  $x \in \mathcal{D}(\mathcal{R}, \rho) \Rightarrow f(x, \pi(x)) \in \mathcal{R}$  as a  $\mathcal{R}$ -invariant sublevel set. In other words,  $\mathcal{D}(\mathcal{R}, \rho)$  is a sublevel set which is additionally forward invariant with respect to  $\mathcal{R}$ . We assume that  $\mathcal{B}(0, \epsilon) \subset \mathcal{D}(\mathcal{R}, \rho)$ .
Next, we formally prove that  $\epsilon$ -stability combined with a sublevel set  $\mathcal{D}(\mathcal{R}, \rho)$  entails convergence in three senses: 1) that we reach an  $\epsilon$ -ball around the origin in finite time, 2) that we reach it infinitely often, and 3) we converge to an arbitrarily small ball around the origin. We refer to the first of these senses as reachability. What we show is that for  $\epsilon$  sufficiently small, reachability implies convergence in all three senses. For this, a key additional assumption is that  $V$  and  $\pi$  are Lipschitz continuous, with Lipschitz constants  $L_v$  and  $L_{\pi}$ , respectively (e.g., in ReLU neural networks).
Theorem 2.2. Suppose  $V$  and  $\pi$  are  $\epsilon$ -stable on a compact  $\mathcal{R}$ ,  $\mathcal{D}(\mathcal{R},\rho)$  is an  $\mathcal{R}$ -invariant set, and  $\exists c_{1}$  such that  $\| \pi (0) - u_0\|_{\infty}\leq c_1\epsilon$ . Then if  $x_0\in \mathcal{D}(\mathcal{R},\rho)\setminus \mathcal{B}(0,\epsilon)$ :
(i) there exists a finite  $K$  such that  $x_{K}\in \mathcal{B}(0,\epsilon)$
(ii)  $\exists c_{2}$  such that if  $c_{2}\epsilon < \rho$  and  $\mathcal{B}(0,c_2\epsilon)\subset \mathcal{R}$ , then there exists a finite  $K$  such that  $\forall k\geq K$ ,  $x_{k}\in \mathcal{D}(\mathcal{R},c_{2}\epsilon)$  and the sequence  $\{x_{k}\}_{k\geq 0}$  reaches  $\mathcal{B}(0,\epsilon)$  infinitely often, and furthermore
(iii) for any  $\eta > 0$  such that  $\mathcal{B}(0, \eta) \subset \mathcal{R}$ ,  $\exists \epsilon$  and finite  $K$  such that  $\| \pi(0) - u_0 \|_{\infty} \leq c_1 \epsilon \Rightarrow \| x_k \|_{\infty} \leq \eta \forall k \geq K$ .
Proof. We prove (i) by contradiction. Suppose that  $\| x_{k}\|_{\infty} > \epsilon \forall k\in \{0,\dots ,\lceil V(x_{0}) / \zeta \rceil \}$ . Then, when  $k = \lceil V(x_0) / \zeta \rceil$ , condition (b) of  $\epsilon$ -stability and  $\mathcal{R}$ -invariance of  $\mathcal{D}(\mathcal{R},\rho)$  implies that  $V(x_{k}) < V(x_{0}) - \zeta k < 0$ , contradicting condition (c) of  $\epsilon$ -stability.
To prove (ii), fix the finite  $K$  from (i), and let  $k \geq K$  be such that  $x_{k} \in \mathcal{B}(0, \epsilon)$ . By Lipschitz continuity of  $V$  and the fact that  $V(0) = 0$  and  $V(x) \geq 0$  (conditions (a) and (c) of  $\epsilon$ -stability),  $V(x_{k+1}) \leq L_{v} \| x_{k+1}\|$ , where  $\| \cdot \|$  is the  $\ell_{\infty}$  norm here and below. Moreover, since  $x_{k+1} = f(x_{k}, \pi(x_{k}))$ ,  $f(0, u_{0}) = 0$  (stability of the origin), and by Lipschitz continuity of  $f$  and  $\pi$ ,
$$
\left\| x _ {k + 1} \right\| \leq L _ {f} \left\| \left(x _ {k}, \pi \left(x _ {k}\right) - u _ {0}\right) \right\| \leq L _ {f} \max  \left\{\left\| x _ {k} \right\|, \left\| \pi \left(x _ {k}\right) - u _ {0} \right\| \right\}.
$$
By Lipschitz continuity of  $\pi$ , triangle inequality, and the condition that  $\| \pi(0) - u_0 \| \leq c_1 \epsilon$ , we have  $\| \pi(x_k) - u_0 \| \leq L_\pi \| x_k \| + c_1 \epsilon \leq (L_\pi + c_1) \epsilon$ . Let  $c_2 = \max \{L_v, 1\} L_f \max \{1, L_\pi + c_1\}$ . Then  $\| x_{k+1} \| \leq c_2 \epsilon$  and  $V(x_{k+1}) \leq c_2 \epsilon$ . Thus, if  $c_2 \epsilon < \rho$  and  $\mathcal{B}(0, c_2 \epsilon) \subset \mathcal{R}$ , and  $\mathcal{D}(\mathcal{R}, \rho)$  is  $\mathcal{R}$ -invariant, then  $x_{k+1} \in \mathcal{D}(\mathcal{R}, c_2 \epsilon)$  which is  $\mathcal{R}$ -invariant. Additionally, either  $x_{k+1} \in \mathcal{B}(0, \epsilon)$ , and by the argument above  $x_{k+2} \in \mathcal{D}(\mathcal{R}, c_2 \epsilon)$ , or  $x_{k+1} \in \mathcal{D}(\mathcal{R}, c_2 \epsilon) \setminus \mathcal{B}(0, \epsilon)$ , and by  $\epsilon$ -stability  $x_{k+2} \in \mathcal{D}(\mathcal{R}, c_2 \epsilon)$ . Thus, by induction, we have that for all  $k \geq K$ ,  $x_k \in \mathcal{D}(\mathcal{R}, c_2 \epsilon)$ . Finally, since for all  $k \geq K$ ,  $x_k \in \mathcal{D}(\mathcal{R}, c_2 \epsilon)$ , if  $x_k \notin \mathcal{B}(0, \epsilon)$ , by (i) it must reach  $\mathcal{B}(0, \epsilon)$  in finite time. Consequently, there is an infinite subsequence  $\{x_{k'}\}$  of  $\{x_k\}_{k \geq 0}$  such that all  $x_{k'} \in \mathcal{B}(0, \epsilon)$ , that is, the sequence  $\{x_k\}_{k \geq 0}$  reaches  $\mathcal{B}(0, \epsilon)$  infinitely often.
We prove part (iii) by contradiction. Fix  $\eta > 0$  and define  $S = \{x \in \mathcal{R} : \| x\| > \eta\}$ . Suppose that  $\forall \epsilon > 0$  there exists  $x \in S$  such that  $V(x) \leq c_2\epsilon$  where  $c_2$  is as in (ii). Then for any (infinite) sequence of  $\{\epsilon_t\}$  we have  $\{x_t\}$  such that  $V(x_t) \leq c_2\epsilon_t$ , where  $x_t \in S$ . Now, consider a set  $\bar{S} = \{x \in \mathcal{R} : \| x\| \geq \eta\}$ . Since  $\bar{S}$  is compact and  $\{x_t\}$  is an infinite sequence, there is an infinite subsequence  $\{x_{t_k}\}$  of  $\{x_t\}$  such that  $\lim_{k \to \infty} x_{t_k} = x^*$  and  $x^* \in \bar{S}$ . Since  $V$  is continuous, we have  $\lim_{k \to \infty} V(x_{t_k}) = V(x^*)$ . Now, we choose  $\{\epsilon_t\}$  such that  $\lim_{t \to \infty} \epsilon_t = 0$ . This means that  $\lim_{k \to \infty} V(x_{t_k}) = V(x^*) \leq 0$ , and since  $x^* \in \bar{S}$ , this contradicts condition (c) of  $\epsilon$ -stability. Since by (ii), there exists a finite  $K$ ,  $V(x_k) \leq c_2\epsilon$ ,  $\forall k \geq K$ , we have  $\forall k \geq K$ ,  $\| x_k\| \leq \eta$ .
The crucial implication of Theorem 2.2 is that as long as we choose  $\epsilon >0$  sufficiently small, verifying that  $V$  and  $\pi$  are  $\epsilon$ -stable together with identifying an  $\mathcal{R}$ -invariant set  $\mathcal{D}(\mathcal{R},\rho)$  suffices for convergence arbitrarily close to the origin. One caveat is that we need to ensure that  $\pi (0)$  is sufficiently close to  $u_{0}$  for any  $\epsilon$  we choose. In most domains, this can be easily achieved: for example, if  $u_{0} = 0$  (as is the case in many settings, including three of our four experimental domains), we can use a representation for  $\pi$  with no bias terms, so that  $\pi (0) = 0 = u_0$  by construction. In other cases, we can simply check this condition after learning.
Another caveat is that we need to define  $\mathcal{R}$  to enable us to practically achieve these properties. To this end, we define  $\mathcal{R}$  parametrically as  $\mathcal{R}(\gamma) = \{x\in \mathcal{X}\mid \| x\|_{\infty}\leq \gamma \}$ . Additionally, we introduce the following useful notation. Define  $\mathcal{R}(\epsilon ,\gamma) = \{x\in \mathcal{X}\mid \epsilon \leq \| x\|_{\infty}\leq \gamma \}$ . Note that  $\mathcal{R}(0,\infty) = \mathcal{X}$ ,  $\mathcal{R}(\epsilon ,\infty) = \{x\in \mathcal{X}\mid \| x\|_{\infty}\geq \epsilon \}$  and  $\mathcal{R}(0,\gamma) = \mathcal{R}(\gamma)$ . Thus, for  $\gamma$  sufficiently large compared to  $\epsilon$ , conditions such as  $\mathcal{B}(0,\epsilon)\subset \mathcal{D}(\mathcal{R},\rho)$ ,  $\mathcal{B}(0,\eta)\subset \mathcal{R}$ , and  $\mathcal{B}(0,c_2\epsilon)\subset \mathcal{R}$  will be easy to satisfy. Following conventional naming, we denote  $\mathcal{R}(\epsilon ,\gamma)$  as  $\epsilon$ -valid region, i.e., the region that satisfies conditions (a)-(c) of  $\epsilon$ -stability, and refer to a function  $V$  satisfying these conditions as an  $\epsilon$ -Lyapunov function.
We assume that the  $\epsilon$ -Lyapunov function as well as the policy can be represented in a parametric function class, such as by a deep neural network. Formally, we denote the parametric  $\epsilon$ -Lyapunov function by  $V_{\theta}(x)$  and the policy by  $\pi_{\beta}(x)$ , where  $\theta$  and  $\beta$  are their respective parameters. Let  $\mathcal{D}(\gamma, \rho) \equiv \mathcal{D}(\mathcal{R}(\gamma), \rho)$ . Our goal is to learn  $V_{\theta}$  and  $\pi_{\beta}$  such that we maximize the size of the  $\mathcal{R}(\gamma)$ -invariant sublevel set  $\mathcal{D}(\gamma, \rho)$  such that  $V_{\theta}$  and  $\pi_{\beta}$  are provably  $\epsilon$ -stable on  $\mathcal{R}(\gamma)$ . Armed with Theorem 2.2, we refer to this set simply as ROA below for simplicity and for consistency with prior work, e.g., [Chang et al., 2019, Zhou et al., 2022], noting all the caveats discussed above.
Define  $\mathcal{P}(\gamma)$  as the set  $(\theta, \beta)$  for which the conditions (a)-(c) of  $\epsilon$ -stability are satisfied over the domain  $\mathcal{R}(\gamma)$ . Our main challenge below is to find  $(\theta, \beta) \in \mathcal{P}(\gamma)$  for a given  $\gamma$ . That, in turn, entails solving the key subproblem of verifying these conditions for given  $V_{\theta}$  and  $\pi_{\beta}$ .
Next, in Section 3 we address the verification problem, and in Section 4 we describe our approach for jointly learning  $V_{\theta}$  and  $\pi_{\beta}$  that can be verified to satisfy the  $\epsilon$ -stability conditions for a given  $\mathcal{R}(\gamma)$ .
3 Verifying Stability Conditions
Prior work on learning  $\epsilon$ -Lyapunov functions for continuous-time nonlinear control problems has leveraged off-the-shelf SMT solvers, such as dReal [Gao et al., 2013]. However, these solvers scale poorly in our setting (see Supplement C for details). In this section, we propose a novel approach for verifying the  $\epsilon$ -Lyapunov conditions for arbitrary Lipschitz continuous dynamics using mixed-integer linear programming, through obtaining piecewise-linear bounds on the dynamics. We assume  $V_{\theta}$  and  $\pi_{\beta}$  are  $K$ - and  $N$ -layer neural networks, respectively, with ReLU activations.
We begin with the problem of verifying condition (c) of  $\epsilon$ -stability, which we represent as a feasibility problem: to find if there is any point  $\tilde{x} \in \mathcal{R}(\epsilon, \gamma)$  such that  $V(\tilde{x}) \leq 0$ . We can formulate it as the following MILP:
$$
z _ {K} \leq 0 \tag {2a}
$$
$$
z _ {l + 1} = g _ {\theta_ {l}} \left(z _ {l}\right), \quad 0 \leq l \leq K - 1 \tag {2b}
$$
$$
\epsilon \leq \| x \| _ {\infty} \leq \gamma , \quad z _ {0} = x, \tag {2c}
$$
where  $l$  refers to a layer in the neural network  $V_{\theta}$ ,  $z_{K} = V_{\theta}(x)$ , and the associated functions  $g_{\theta_l}(z_l)$  are either  $W_{l}z_{l} + b_{l}$  for a linear layer (with  $\theta_l = (W_l,b_l)$ ) or  $\max \{z_l,0\}$  for a ReLU activation. Any feasible solution  $x^{*}$  is then a counterexample, and if the problem is infeasible, the condition is satisfied. ReLU activations  $g(z)$  can be linearized by introducing an integer variable  $a\in \{0,1\}$ , and replacing the  $z^{\prime} = g(z)$  terms with constraints  $z^{\prime}\leq Ua$  and  $z^{\prime}\leq -L(1 - a)$ , where  $L$  and  $U$  are specified so that  $L\le z\le U$  (we deal with identifying  $L$  and  $U$  below).
Next, we cast verification of condition  $(b)$  of  $\epsilon$ -stability, which involves the nonlinear control dynamics  $f$ , as the following feasibility problem:
$$
\bar {z} _ {K} - z _ {K} \geq - \zeta \tag {3a}
$$
$$
y _ {i + 1} = h _ {\beta_ {i}} \left(y _ {i}\right), \quad 0 \leq i \leq N - 1 \tag {3b}
$$
$$
\bar {z} _ {l + 1} = g _ {\theta_ {l}} \left(\bar {z} _ {l}\right), \quad 0 \leq l \leq K - 1 \tag {3c}
$$
$$
\bar {z} _ {0} = f (x, y _ {N}) \tag {3d}
$$
$$
y _ {0} = x, \quad \text {c o n s t r a i n t s (2 b) - (2 c)}, \tag {3e}
$$
where  $h_{\beta_i}()$  are functions computed at layers  $i$  of  $\pi_{\beta}$ ,  $z_K = V_\theta(x)$ , and  $\bar{z}_K = V_\theta(f(x, \pi_\beta(x)))$ .