SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Spencer M Richards, Felix Berkenkamp, and Andreas Krause. The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, pages 466-476. PMLR, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques E Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction-based regularization. The International Journal of Robotics Research, 40(10-11):1123-1150, 2021.
Jarrod M. Snider. Automatic steering methods for autonomous automobile path tracking. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RITR-09-08, 2009.
Russ Tedrake. Underactuated robotics: Learning, planning, and control for efficient and agile machines course notes for mit 6.832. Working draft edition, 3:4, 2009.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 33:1129-1141, 2020.
Ruikun Zhou, Thanin Quartz, Hans De Sterck, and Jun Liu. Neural lyapunov control of unknown nonlinear systems with stability guarantees. In Neural Information Processing Systems, 2022.
A Further Details about Experiments
A.1 Dynamics
Since these are described as continuous-time domains, we set the time between state updates to be  $0.05s$  in all cases for our discretized versions of these systems.
Inverted Pendulum We use the following standard dynamics model for inverted pendulum [Chang et al., 2019, Richards et al., 2018, Zhou et al., 2022]:
$$
\ddot {\theta} = \frac {m g \ell \sin (\theta) + u - b \dot {\theta}}{m \ell^ {2}},
$$
with gravity constant  $g = 9.81$ , ball weight  $m = 0.15$ , friction  $b = 0.1$ , and pendulum length  $l = 0.5$ . We set the maximum torque allowed to be  $6Nm$ , meaning  $u \in [-6.0, 6.0]$ , and it is applied to all experiments except for the NLC (free) case.
Path Tracking Our path tracking dynamics follows Snider [2009] and Chang et al. [2019]:
$$
\dot {s} = \frac {v \cos (\theta_ {e})}{1 - \dot {e} _ {r a} \kappa (s)}, \quad \dot {e} _ {r a} = v \sin (\theta_ {e}), \dot {\theta} _ {e} = \frac {v \tan (\delta)}{L} - \frac {v \kappa (s) \cos (\theta_ {e})}{1 - \dot {e} _ {r a} \kappa (s)},
$$
where the control policy determines the steering angle  $\delta$ .  $e_{ra}$  is the distance error and  $\theta_e$  is the angle error. We can simplify the dynamics by defining  $u = \tan (\delta)$ , so that non-linearity of dynamics is only in terms of state variables. Since in practice  $|\delta |\leq 40^{\circ}$ , we set  $u\in [\tan (-40^{\circ}),\tan (40^{\circ})]$ ; this is applied to all experiments. We set speed  $v = 2.0$ , while the track is a circle with radius 10.0 (and, thus,  $\kappa = 0.1$ ) and  $L = 1.0$ .
For RL in path tracking, the reward function is  $-\frac{1}{10} (e_{ra}^2 + \theta_e^2)^{1/2}$ , and we use PPO [Schulman et al., 2017], training over 50K steps.
Cartpole For cartpole we use the dynamics from Tedrake [2009]. The original dynamics is
$$
\ddot {x} = \frac {1}{m _ {c} + m _ {p} \sin^ {2} \theta} \left[ f _ {x} + m _ {p} \sin \theta \left(l \dot {\theta} ^ {2} + g \cos \theta\right) \right] \tag {6}
$$
$$
\ddot {\theta} = \frac {1}{l (m _ {c} + m _ {p} \sin^ {2} \theta)} \Big [ - f _ {x} \cos \theta - m _ {p} l \dot {\theta} ^ {2} \cos \theta \sin \theta - (m _ {c} + m _ {p}) g \sin \theta \Big ]. \tag {7}
$$
We change the variable and rename  $\pi -\theta$  as  $\theta$ , so that in our setting,  $\theta$  represents the angle between the pole and the upward horizontal direction, the pole angle is positive if it is to the right. The purpose of this transformation is that the equilibrium point is the origin.
Using the second order chain rule
$$
\frac {\mathrm {d} ^ {2} y}{\mathrm {d} z ^ {2}} = \frac {\mathrm {d} ^ {2} y}{\mathrm {d} u ^ {2}} \left(\frac {\mathrm {d} u}{\mathrm {d} z}\right) ^ {2} + \frac {\mathrm {d} y}{\mathrm {d} u} \frac {\mathrm {d} ^ {2} u}{\mathrm {d} z ^ {2}}
$$
let  $y = \pi -\theta ,z = t,u = \theta$  , we have the new dynamics as
$$
\ddot {x} = \frac {1}{m _ {c} + m _ {p} \sin^ {2} \theta} \left[ f _ {x} + m _ {p} \sin \theta \left(l \dot {\theta} ^ {2} - g \cos \theta\right) \right] \tag {8}
$$
$$
\ddot {\theta} = \frac {1}{l (m _ {c} + m _ {p} \sin^ {2} \theta)} \Big [ - f _ {x} \cos \theta - m _ {p} l \dot {\theta} ^ {2} \cos \theta \sin \theta + (m _ {c} + m _ {p}) g \sin \theta \Big ]. \qquad (9)
$$
Here  $m_c = 1.0$  is the weight of the cart,  $m_p = 0.1$  is the weight of the pole,  $x$  is the horizontal position of the cart,  $l = 1.0$  is the length of the pole,  $g = 9.81$  is the gravity constant and  $f_x$  is the controller (force applied to the cart). Furthermore, the maximum force allowed to the cart is set to be  $30N$ , meaning  $f_x \in [-30, 30]$ ; this is applied to all experiments.
PVTOL For PVTOL, our dynamics follows Singh et al. [2021]. Specifically, the system has the following state representation:  $x = \left(p_{x}, p_{z}, v_{x}, v_{z}, \phi, \dot{\phi}\right)$ .  $(p_{x}, p_{z})$  and  $(v_{x}, v_{z})$  are the 2D position and velocity, respectively, and  $(\phi, \dot{\phi})$  are the roll and angular rate. Control  $u \in \mathbb{R}_{>0}^{2}$  corresponds to the controlled motor thrusts. The dynamics  $f(x, u)$  are described by
$$
\dot {x} (t) = \left[ \begin{array}{c} v _ {x} \cos \phi - v _ {z} \sin \phi \\ v _ {x} \sin \phi + v _ {z} \cos \phi \\ \dot {\phi} \\ v _ {z} \dot {\phi} - g \sin \phi \\ - v _ {x} \dot {\phi} - g \cos \phi \\ 0 \end{array} \right] + \left[ \begin{array}{c c} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ (1 / m) & (1 / m) \\ l / J & (- l / J) \end{array} \right] u
$$
where  $g = 9.8$  is the acceleration due to gravity,  $m = 4.0$  is the mass,  $l = 0.25$  is the moment-arm of the thrusters, and  $J = 0.0475$  is the moment of inertia about the roll axis. The maximum force allowed to both  $u_{1}$  and  $u_{2}$  is set to be  $39.2N$ , which is  $mg$ , meaning  $u_{1}, u_{2} \in [0, 39.2]$ ; this is applied to all experiments.
A.2 Computation of the LQR Solution
Given the system  $\frac{dx}{dt} = Ax + Bu$ , the objective function of the LQR is to compute the optimal controller  $u = -Kx$  that minimizes the quadratic cost  $\int_0^\infty (x'Qx + u'Ru)dt$ , where  $R$  and  $Q$  are identity matrices. The details of the system linearization as well as the LQR solution are described below.
Inverted Pendulum We linearize the system as
$$
A = \left[ \begin{array}{c c} 0 & 1 \\ \frac {g}{l} & - \frac {b}{m l ^ {2}} \end{array} \right], B = \left[ \begin{array}{c} 0 \\ \frac {1}{m l ^ {2}} \end{array} \right]
$$
The final LQR solution is  $u = -1.97725234\theta - 0.97624064\dot{\theta}$ .
Path Tracking We linearize the system as
$$
A = \left[ \begin{array}{c c} 0 & 2 \\ 0 & - 0. 0 4 \end{array} \right], B = \left[ \begin{array}{c} 0 \\ 1 \end{array} \right].
$$
The final LQR solution is  $u' = 2u + 0.1 = -e_{ra} - 2.19642572\theta_e$ .
Cartpole We linearize the system as
$$
A = \left[ \begin{array}{c c c c} 0 & 1 & 0 & 0 \\ 0 & 0 & - 0. 9 8 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 0. 7 8 & 0 \end{array} \right], B = \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ - 1 \end{array} \right].
$$
The final LQR solution is  $u = x + 2.4109461\dot{x} + 34.36203947\theta + 10.70094483\dot{\theta}$ .
PVTOL We linearize the system as
$$
A = \left[ \begin{array}{l l l l l l} 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & - g & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \end{array} \right], B = \left[ \begin{array}{l l} 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & 0 \\ \frac {1}{m} & \frac {1}{m} \\ \frac {r}{J} & - \frac {r}{J} \end{array} \right]
$$
The final LQR solution is  $u_{1} = 0.70710678x - 0.70710678y - 5.03954871\theta + 1.10781077\dot{x} - 1.82439774\dot{y} - 1.20727555\dot{\theta}$ , and  $u_{2} = -0.70710678x - 0.70710678y + 5.03954871\theta - 1.10781077\dot{x} - 1.82439774\dot{y} + 1.20727555\dot{\theta}$ .
A.3 PPO Training for Path Tracking
We use the implementation in Stable-Baselines [Raffin, 2020] for PPO training. The hyperparameters of are fine-tuned using their auto-tuning script with a budget of 1000 trials with a maximum of 50000 steps. The final hyperparameters are in Table 3.
Table 3: Hyperparameter for PPO (Path Tracking)
Parameter	Value
policy	MlpPolicy
n_timesteps	!!float 100000
batch_size	32
n_steps	64
gamma	0.95
learning_rate	0.000208815
ent coef	1.90E-06
clip_range	0.1
n_epochs	10
gae_lambda	0.99
max_grad_norm	0.8
vf coef	0.550970466
activation_fn	nn.ReLU
log_std_init	-0.338380542
ortho_init	False
net_arch	[8, 8]
vf	[8, 8]
sde_sample_freq	128
A.4 Benchmark Models
When relevant, benchmark models (as well as our approach) are run for 10 seeds (seed 0 to 9). Mean  $\pm$  standard deviation are reported in Table 1 and Table 2. For SOS benchmark, we use polynomials of degree  $\leq 6$  for all environments.
NLC and UNL For Cartpole and PVTOL, we train against diameter 1.0 under the  $l_{2}$  norm for NLC and UNL. The main reason for the failure of training is that the dReal verifier becomes incredibly show after some iterations. For example, for seed 0, NLC only finished 167 certifications within 2 hours limit for Cartpole, and 394 certifications within 24 hours limit for PVTOL.
LQR and SOS For Cartpole environment, we first certify against target region  $0.1 \leq ||x||_{\infty} \leq 0.16$  for LQR benchmark where dReal returns the counterexample; we later attempt to certify against  $0.1 \leq ||x||_{\infty} \leq 0.15$ , dReal did not return any result within the 2-hour limit. For SOS benchmark, dReal verifier did not return any result within the 2 hours limit against target region  $0.1 \leq ||x||_{\infty} \leq 0.5$ . For PVTOL environment, LQR benchmark returns the counterexample against target region  $0.1 \leq ||x||_{\infty} \leq 0.5$  after 10 hours, we then attempt to certify against  $0.1 \leq ||x||_{\infty} \leq 0.4$  and fail to return any result within the remaining 14 hours time limit. For SOS benchmark, dReal verifier did not return any result within the 24 hours limit against target region  $0.1 \leq ||x||_{\infty} \leq 0.5$ .
A.5 Bounding the Functions
There are three types of bounding in our paper: 1) bound the dynamics  $f(x,u)$  with linear upper/ lower bound functions for verification; 2) bound the dynamics with constants for the calculation of  $M$  for ReLU; 3) bound the dynamics with constants for the calculation of ROA. Note that the bounding for 2) and 3) are similar, except that for 2) it is calculated for each subgrid, and for 3) it is calculated for the entire valid region.
Inverted Pendulum Split the region: we split the region over the domain of  $\theta$ , please refer to our code for the details of the splitting.
(1) Bound for verification: since the only non-linear part of the function is  $\sin (\theta)$ , we use the upper/ lower bounds in  $\alpha ,\beta$  -CROWN. (2) Bound for  $M$  ..  $|\ddot{\theta} |\leq \frac{g}{l} +\frac{|u|_{\mathrm{max}}}{ml^2} +\frac{b|\dot{\theta}|_{\mathrm{max}}}{ml^2}$  where  $|u|_{\mathrm{max}}$  is calculated using IBP bound. (3) Bound for ROA:  $||f(x,u)||_{\infty} = \gamma +\max (\gamma ,\frac{mgl + |u|_{\mathrm{max}} + b\gamma}{ml^2})dt,$  where  $|u|_{\mathrm{max}}$  is the maximum force allowed.
Path Tracking Split the region: we split the region over the domain of  $\theta_{e}$ , please refer to our code for the details of the splitting.
(1) Bound for verification: for  $\dot{e}_{ra}$  since the only nonlinear function is  $\sin (\theta)$ , we use the bound in  $\alpha$ ,  $\beta$ -CROWN; for  $\dot{\theta}_e$  we bound the term  $\frac{v\kappa(s)\cos(\theta_e)}{1 - \dot{e}_{ra}\kappa(s)}$ , which is concave in interval  $[- \pi / 2, \pi / 2]$ , and convex otherwise. (2) Bound for  $M$ :  $|\dot{e}_{ra}| \leq v$ ,  $|\dot{\theta}_e| \leq \frac{v|u|_{\max}}{L} + \frac{v}{R - v}$ , where  $|u|_{\max}$  is calculated using IBP bound. (3) Bound for ROA:  $||f(x,u)||_{\infty} = \gamma + \max(v, \frac{v|u|_{\max}}{L} + \frac{v}{R - v})dt$ , where  $|u|_{\max}$  is the maximum force allowed.
Cartpole Split the region: we split the region over the domain of  $\theta$  and  $\dot{\theta}$  with interval 0.1 when either  $\theta$  or  $\dot{\theta}$  is greater than 0.1. In case when both  $\theta$  and  $\dot{\theta}$  are smaller than 0.1, we split  $\theta$ ,  $\dot{\theta}$  into intervals of 0.05. Please refer to our code for the details of the splitting.
(1) Bound for verification: we split the dynamics into six functions and bound them separately for both one variable and multiple variables:  $f_{1} = \frac{1}{m_{c} + m_{p}\sin^{2}\theta}, f_{2} = \frac{-\cos\theta}{l(m_{c} + m_{p}\sin^{2}\theta)}, f_{3} = \frac{-gm_{p}\sin\theta\cos\theta}{m_{c} + m_{p}\sin^{2}\theta},$ $f_{4} = \frac{(m_{c} + m_{p})g\sin\theta}{l(m_{c} + m_{p}\sin^{2}\theta)}, f_{5} = \frac{m_{p}l\sin\theta\dot{\theta}^{2}}{m_{c} + m_{p}\sin^{2}\theta}, f_{6} = \frac{-m_{p}l\dot{\theta}^{2}\cos\theta\sin\theta}{l(m_{c} + m_{p}\sin^{2}\theta)}$ . (2) Bound for  $M$ :  $|\ddot{x}| \leq |u|_{\max} + m_p l|\dot{\theta}|_{\max}^2 + \frac{m_p g}{2}, |\ddot{\theta}| \leq \frac{1}{l}(|u|_{\max} + \frac{m_p l|\dot{\theta}|_{\max}^2}{2} + (m_c + m_p)g)$  where  $|u|_{\mathrm{max}}$  is calculated using IBP bound. (3) Bound for ROA:  $||f(x,u)||_{\infty} = \gamma + \max(\gamma, |u|_{\max} + m_p l\gamma^2 + \frac{m_p g}{2}, \frac{1}{l}(|u|_{\max} + \frac{m_p l\gamma^2}{2} + (m_c + m_p)g))dt$ , where  $|u|_{\mathrm{max}}$  is the maximum force allowed.
PVTOL Split the region: we split the region over the domain of  $\theta$  with interval 0.25 and split the region over the domain of  $\dot{x},\dot{y},\dot{\theta}$  with interval 0.5. Furthermore, in the MILP we add a constraint to ensure the  $l_{\infty}$  norm of state value is no less than 0.1. Please refer to our code for the details of the splitting.
(1) Bound for verification: we split the dynamics into three functions and bound them separately for multi-variables:  $f_{1} = x\cos y$ ,  $f_{2} = x\sin y$ ,  $f_{3} = xy$ . (2) Bound for  $M$ : we bound each term of the dynamics with the min/max values using the monotonic property. (3) Bound for ROA:  $||f(x,u)||_{\infty} = \gamma + \max(\gamma, \gamma^2 + g + \frac{2|u|_{\max}}{m}, 2\gamma, \frac{2l|u|_{\max}}{J})dt$ , where  $|u|_{\max}$  is the maximum force allowed.
A.6 Speeding up Verification
When there is a limit on parallel resources, so that verification must be done serially for the subgrids  $k$ , we can reduce the number of grids to verify in each step as follows. The first time a verifier is called, we perform it over all subgrids; generally, only a subset will return a counterexample. Subsequently, we only verify these subgrids until none returns a counterexample, and only then attempt full verification.
A.7 ROA Calculation
For Inverted Pendulum and Path Tracking, we approximate ROA using a grid of 2000 cells along each coordinate. For Cartpole, we use a grid of 150 cells along each coordinate, and for PVTOL, we use a grid of 50 cells along each coordinate.
For our baselines where the systems are continuous, the set levels are  $\{x\in \mathcal{R}\mid V(x) < \rho \}$  where  $\rho = \min_{x\in \partial \mathcal{R}}V(x)$ . Note that for discrete-time systems, having  $\rho = \min_{x\in \partial \mathcal{R}}V(x)$  is not sufficient. For example, assume  $\mathcal{R}$  is the valid region, any point outside of  $\mathcal{R}$  satisfy  $V(f(x,u)) > V(x)$ , and  $\forall x\in \partial \mathcal{R},V(x) = \rho$ . Assume we have  $x\in \mathcal{R}\setminus \partial \mathcal{R}$  and  $f(x,u) = y,y\notin \mathcal{R}$  (this is possible since
the system is discrete). Since  $x$  is in the valid region, we have  $V(y) < V(x) < \rho$ . However, since  $y$  is no longer within the valid region, assume  $V(f(y,u)) > \rho > V(y)$ , the system will not converge to equilibrium point starting from point  $y$ . This means  $\{x \in \mathcal{R} \mid V(x) < \rho\}$  is not a subset of ROA.
B Computing Linear Bounds on Lipschitz-Continuous Dynamics
Theorem B.1. Given function  $g(x): \mathbb{R}^n \mapsto \mathbb{R}$  where the Lipschitz constant is  $\lambda$ , we have  $g(x) \leq g(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) + \lambda \cdot \sum_{i} (x_i - x_{i,l})$  and  $g(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) - \lambda \cdot \sum_{i} (x_i - x_{i,l}) \leq g(x)$ , where  $x_i \in [x_{i,l}, x_{i,u}]$ ,  $\forall i \in [n]$ .
Proof. For any  $i \in [n]$ , given  $x_i \in [x_{i,l}, x_{i,u}]$ , for arbitrary fixed values along dimension  $[n] \setminus \{j\}$  (we denote them as  $x_{-i}$ ), we have  $|g(x_i, x_{-i}) - g(x_{i,l}, x_{-i})| \leq \lambda |x_i - x_{i,l}|$ .
we show that  $g(x_{i},x_{-i})\leq g(x_{i,l},x_{-i}) + \lambda (x_{i} - x_{i,l})$
- if  $g(x_{i},x_{-i}) \geq g(x_{i,l},x_{-i})$ , then  $g(x_{i},x_{-i}) - g(x_{i,l},x_{-i}) \leq \lambda (x_{i} - x_{i,l})$ , meaning  $g(x_{i},x_{-i}) \leq g(x_{i,l},x_{-i}) + \lambda (x_{i} - x_{i,l})$ .
- if  $g(x_{i},x_{-i}) < g(x_{i,l},x_{-i})$ , since  $\lambda (x_{i} - x_{i,l})\geq 0$ , we have  $g(x_{i},x_{-i})\leq g(x_{i,l},x_{-i}) + \lambda (x_{i} - x_{i,l})$ .
we show that  $g(x_{i,l},x_{-i}) - \lambda (x_i - x_{i,l})\leq g(x_i,x_{-i})$
- if  $g(x_{i},x_{-i}) \geq g(x_{i,l},x_{-i})$ , then since  $-\lambda (x_{i} - x_{i,l}) \leq 0$ , we have  $g(x_{i,l},x_{-i}) - \lambda (x_{i} - x_{i,l}) \leq g(x_{i},x_{-i})$ .
- if  $g(x_{i},x_{-i}) < g(x_{i,l},x_{-i})$ , then  $g(x_{i,l},x_{-i}) - g(x_{i},x_{-i}) \leq \lambda (x_{i} - x_{i,l})$ , meaning  $g(x_{i,l},x_{-i}) - \lambda (x_{i} - x_{i,l}) \leq g(x_{i},x_{-i})$ .
By applying the above result along all  $n$  dimensions recursively, we have the upper bound as  $g(x) \leq g(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) + \sum_{i} \lambda(x_i - x_{i,l})$ , and the lower bound as  $g(x_{1,l}, x_{2,l}, \ldots, x_{n,l}) - \sum_{i} \lambda(x_i - x_{i,l}) \leq g(x)$ .
In this section, we conduct an ablation study on the MILP solver. Table 4 are the results for DITLDReal, in which we use our framework, but replace all the MILP components with dReal. As we can see, using MILP is an essential ingredient in the success of the proposed approach:
Table 4: Ablation study on MILP solver.
Environment	Runtime	ROA	Max ROA	Success Rate
Inverted Pendulum (DITL-dReal)	6.0 ± 1.7 (s)	57 ± 24	75	100%
Inverted Pendulum (DITL, main paper)	8.1 ± 4.7(s)	61 ± 31	123	100%
Path Tracking (RL, DITL-dReal)	600 ± 0 (s)	0 ± 0	0	0%
Path Tracking (RL, DITL, main paper)	14 ± 11 (s)	9 ± 3.5	16	100%
Path Tracking (LQR, DITL-dReal)	420.9 ± 182 (s)	4 ± 4	11	60%
Path Tracking (LQR, DITL, main paper)	9.8 ± 4 (s)	8 ± 3	12.5	100%
Cartpole (DITL-dReal)	>2 (hours)	N/A	N/A	0%
Cartpole (DITL, main paper)	0.9 ± 0.3 (hours)	0.021 ± 0.012	0.045	100%
PVTOL (DITL-dReal)	>24 (hours)	N/A	N/A	0%
PVTOL (DITL, main paper)	13 ± 6 (hours)	0.011 ± 0.008	0.028	100%