SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The objective can be represented as minimizing a linear function of the output of a residual neural network with a skip connection from the input to the output layer, subject to constraints on the input. Now,  $f(x)$  appears in the objective, similar to the standard forward verification problem with an augmented network architecture. Similarly, optimizing an intermediate bound is equivalent to a skip connection from layer  $i$  to the output (Appendix E). These connections allow us to implement our method using standard verification tools [Brix et al., 2023, Xu et al., 2021, 2020].
When  $f$  is a feedforward ReLU network, the lower bound described in this section is precisely the same as Theorem 1 (since both are solving the dual of the same linear program). This shows that the introduction of  $\gamma$  constitutes our generalization to the bound propagation framework.
Selection of  $c$  The dependence of the optimization on  $c$  is not a major limitation. The bounds of all intermediate neurons are optimized only with respect to the input and output constraints, not the hyperplanes used to describe the input half-space. Thus, their optimized bounds can be shared across the optimization of all input hyperplanes. For applications such as adversarial robustness, the box constraint of two hyperplanes per dimension could be chosen to scalably bound the input. The selection of appropriate  $c$  depends on the application and is not our main focus.
3.3 Branch and Bound
Though more rounds of iterative tightening will lower the gap in  $S_{\mathrm{over}} \supseteq S_{\mathrm{LP}}$ , our current formulation still faces two sources of looseness: the gap in  $S_{\mathrm{LP}} \supseteq S_{\mathrm{MILP}}$  and the gap in  $S_{\mathrm{MILP}} \supseteq f^{-1}(S_{\mathrm{out}})$ . To overcome both of these issues, we can make use of branching [Bunel et al., 2020]. While there are several possibilities here, we focus on input branching, which gave the biggest empirical gains in our experiments described in Section 4. More concretely, we divide the input domain  $\mathcal{X} = [l^{(0)}, \boldsymbol{u}^{(0)}]$  into several regions by choosing a coordinate  $i$  and computing
$$
\mathcal {X} _ {a} = \mathcal {X} \cap \left\{\boldsymbol {x}: \boldsymbol {x} _ {i} \geq s _ {i} \right\}, \mathcal {X} _ {b} = \mathcal {X} \cap \left\{\boldsymbol {x}: \boldsymbol {x} _ {i} \leq s _ {i} \right\}
$$
so that  $\mathcal{X} = \mathcal{X}_a\cup \mathcal{X}_b$ . Doing this recursively, we obtain several regions and can compute an overapproximation of  $f^{-1}(S_{\mathrm{out}})$  when the input is in each of those regions, and take a union of the resulting sets. The finer the input domain of our over-approximation, the tighter the approximation.
We evaluate INVPROP on three different benchmarks. Across benchmarks, we find orders of magnitude improvement over prior work, and our methods work even for relatively large networks (167k neurons) and high dimensionality inputs (8112 dimensions). All implementation details are described in Appendix F and the utilized hardware is described in Appendix G.
(a) ReBreach-LP
(b) INVPROP
Figure 3: Double Integrator. Each green region represents the preimage of the system one to ten steps back from the obstacle (red box). Each preimage is approximated 1 million samples. The blue bounding boxes represent over-approximations, and INVPROP tightly over-approximates preimages even ten steps back.
Table 1: Double Integrator Results. INVPROP vs SOTA (from Figure 3). We are significantly tighter and faster than ReBreach-LP method even without input branching.
METHOD	REBREACH-LP	INVPROP
INPUT BRANCHING	YES	NO
APPROX RATIO ↓	4021.47	1.46
TIME (SEC)	42.86 ± 0.04	17.89 ± 0.03
4.1 Backward Reachability Analysis for Neural Feedback Loops
Double Integrator After combining the state-feedback control policy  $\pi$  of the benchmark with the state transition function from Section 2.3, (see Section F.2), we get a three layer MLP with 12, 7, and 2 neurons, which is typical for applications in this setting. We model multiple time steps via composing copies of this function. Moreover, we leverage intermediate bounds for time step  $t$  to bound the  $t + 1$  time step transition. We measure the tightness of an over-approximation using its Approximation Ratio, defined as  $\frac{\mathrm{vol}(S_{\mathrm{over}})}{\mathrm{vol}(f^{-1}(S_{\mathrm{out}}))}$ . Both of these volumes are heuristically estimated with 1 million samples of the input space. While INVPROP allows the optimization of arbitrary input cutting planes, we use 40 planes with slopes of equally distributed angles.
We find that INVPROP is significantly tighter and faster than ReBreach-LP, the SOTA available method for this problem [Rober et al., 2022a,b].<sup>2</sup> As evident in Figure 3, ReBreach-LP suffers from increasingly weak bounds as  $t$  increases, whereas our approach is able to compute tight bounds for all  $t$ . Different to their implementation, we are able to leverage the output constraint to improve the intermediate neuron bounds. Furthermore, we can iteratively tighten these bounds with respect to each other. Even though we optimize many more quantities, our efficient bound propagation allows us to solve the problem faster overall. We provide a visualization of the improvement of the approximation ratio score and intermediate bounds over time in Figure 2. Both our increased tightness and speed are quantified in Table 1.
6D Quadrotor We consider over-approximating the preimage of the linearized quadrotor discussed in Figure 15 of Rober et al. [2022b]. This quadrotor has a 6-dimensional state space with 3 dimensions dedicated to position and is given by the dynamics
$$
\boldsymbol {x} _ {t + 1} = \left[ \begin{array}{c c c c c c} 1 & 0 & 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 0 & 1 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \end{array} \right] \boldsymbol {x} _ {t} + \left[ \begin{array}{c c c} 0. 5 & 0 & 0 \\ 0 & 0. 5 & 0 \\ 0 & 0 & 0. 5 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right] \pi (\boldsymbol {x} _ {t})
$$
The policy, after performing the same encoding as the double integrator and clipping policy outputs, forms a five layer MLP with 26, 26, 9, 9, and 3 neurons. For these dynamics, we want to prove that the initial state range of  $[-5.25, -4.75] \times [-.25, .25] \times [2.25, 2.75] \times [0.95, 0.99] \times [-0.01, 0.01] \times [-0.01, 0.01]$  (black box in Figure 4) will never collide with the obstacle at  $[-1, 1] \times [-1, 1] \times [1.5, 3.5] \times [-1, 1] \times [-1, 1] \times [-1, 1]$  (red box). We try to find the tightest bounding box in the 3 position dimensions, reflecting a total of 6 input cutting planes (one for each direction in each dimension). To benchmark against prior work, we run ReBreach-LP with no output partitioning and
For example, the 10 time step transition function can be represented as a 21 layer MLP (after fusing consecutive linear layers in the composition).
2We do not compare with Everett et al. [2022] which improves upon ReBreach-LP, as their implementation faces numerical instability when applied for ten timesteps. However, preliminary experiments show that we also outperform their bounds.
(a) ReBreach-LP no partitions
Figure 4: 6D Quadrotor. We visualize the three dimensions of the 6D quadrotor that specify physical position. We consider the initial state range of the black box and simulate a few one-step trajectories, indicated by the orange lines. Our obstacle is the red box. For the three images, the blue box represents the over-approximation computed by (a) ReBreach-LP with no partitioning, (b) ReBreach-LP with 15625 partitions, and (c) INVPROP, respectively. A tighter blue box is better.
(b) ReBreach-LP 15625 partitions
(c) INVPROP
Table 2: Quadrotor Results. We compare the tightness of the over-approximations and run time (mean and std taken over 5 runs).
METHOD	REBREACH-LP	REBREACH-LP	INVPROP
PARTITIONS	N/A	15625	N/A
APPROX VOL ↓	64	0.064	0.0000249
TIME (SEC)	11.8 ± 2.2	662.2 ± 31.4	213.8 ± 11.1
ReBreach-LP with 15625 output partitions. We run our algorithm with no branching. In Figure 4, we compare these three strategies, and we find that our over-approximation of the pre-image (in blue) is smaller than the state-of-the-art over-approximations by 257 times while being 3.29 times faster. Therefore, our algorithm scales well to higher dimensional control examples.
4.2 Robustness Verification
Similar to us, Yang et al. [2021b] encode the implicit output constraint of the local robustness verification query to tighten the bounds on neurons. However, they need an LP solver, which does not scale to large problems. The state-of-the-art verification toolkit  $\alpha, \beta$ -CROWN [Zhang et al., 2018a, Xu et al., 2021, Wang et al., 2021, Zhang et al., 2022] is able to scale to large networks, but does not currently utilize the implicit output constraint. We demonstrate the benefit of encoding the output constraint by extending  $\alpha, \beta$ -CROWN and comparing the performance on the benchmark used by Yang et al. [2021b] as well as benchmarks from the VNN-COMP 2023 [Brix et al., 2023].
MNIST Yang et al. [2021b] provides four networks with ReLU activation functions and dense linear layers of 2/6/6/9 layers of 100/100/200/200 neurons, each trained on MNIST. For each network, 50 inputs were tested for local robustness. For the complete benchmark definition, we refer to Yang et al. [2021b]. The DeepSRGR results are taken from Yang et al. [2021b], they do not report a timeout for their experiments. Both  $\alpha, \beta$ -CROWN and  $\alpha, \beta$ -CROWN+INVPROP were run with a per-input timeout of 5 minutes. Except for the input bounds, all bounds of all layers of each network are tightened using output constraints. We report the results in Table 3. Notably, we can verify more instances than the SOTA tool  $\alpha, \beta$ -CROWN, in sometimes less than one fifth the average runtime. We include a more detailed comparison between the results for  $\alpha, \beta$ -CROWN and  $\alpha, \beta$ -CROWN+INVPROP in Appendix H.
VNN-COMP '23: YOLO In 2023, the VNN-COMP contained YOLO as an unscored benchmark for object detection. It is a modified version of YOLOv2 [Redmon and Farhadi, 2016] with a network of 167k neurons and 5 residual layers, and is available at [Zhong, 2023, Brix, 2023]. The network processes  $3 \times 52 \times 52$  images and uses convolutions, average pooling and ReLU activations. This benchmark is well suited for INVPROP, as the definition of the adversarial examples is a conjunction of constraints over the output. Therefore, a strong output constraint can be used to tighten the bounds of intermediate layers. The benchmark consists of 464 instances, of which 72 were randomly selected for the VNN-COMP. For our comparison, we remove those 348 instances that  $\alpha, \beta$ -CROWN verifies as robust without tightening the bounds of any intermediate layer. Those instances are verified within
Figure 5: YOLO Results. Runtime comparison of  $\alpha, \beta$ -CROWN and  $\alpha, \beta$ -CROWN+INVPROP on the YOLO benchmark (167k neurons and 5 residual layers). For the comparison, only those test instances were used that could not immediately be verified by  $\alpha, \beta$ -CROWN without any iterative tightening of intermediate layer bounds.  $\alpha, \beta$ -CROWN+INVPROP can verify more properties and is faster for almost all instances.
Table 3: MNIST Results. Robustness verification with DeepSRGR (output constraints via LP solver),  $\alpha ,\beta$  CROWN (GPU support) and  $\alpha ,\beta$  -CROWN+INVPROP (output constraints with GPU support)
NETWORK	DEEPRGR	α,β-CROWN	α,β-CROWN + INVPROP
VERIF.	AVG. TIME [S]	VERIF.	AVG. TIME [S]	VERIF.	AVG. TIME [S]
FFN4 (3x100)	35	781	45	7.4	45	6.4
FFN5 (6x100)	31	1689	38	6.7	39	8.7
FFN6 (6x200)	31	6178	29	5.4	30	1.0
FFN7 (9x200)	36	8960	37	3.2	40	1.9
less than four seconds each by both methods, with no room for improvement. We compare the performance of  $\alpha, \beta$ -CROWN and  $\alpha, \beta$ -CROWN + INVPROP on the remaining 116 instances in Figure 5.  $\alpha, \beta$ -CROWN can verify 48 instances, all other instances reach the timeout of five minutes. After extending  $\alpha, \beta$ -CROWN with INVPROP on the last two intermediate layers, almost all instances can be solved faster, and 6 previously timed out instances become verifiable.
4.3 OOD Detection
Consider the calibrated OOD detector presented as discussed in Section 2.3, encoded by a four layer MLP with 200, 200, 3, and 2 neurons. We over-approximate the set of inputs which induce a sufficiently high in-distribution (ID) confidence (measured by  $\max \{y_0, y_1\} > y_2$ ) using 40 hyperplanes of equal slope, pictured in green in Figure 6. This set is non-convex, making the convex hull a poor over-approximation. With 4 input space branches, we get a much tighter over-approximation, as shown in the right plot. We compare the performance of our approach with and without branching over the input space with the MILP baseline (see Table 4). This demonstrates a simple proof-of-concept for how INVPROP can be used for verifying some calibration properties.
Formally Verified Neural Networks. There has been a large body of work on the formal verification of neural networks, tackling the problem from the angles of Interval Bound Propagation [Gowal et al., 2018, Gehr et al., 2018], Convex Relaxations [Wong and Kolter, 2018, Shiqi et al., 2018, Salman et al., 2019, Dvijotham et al., 2018], Abstract Interpretation [Singh et al., 2018], LP [Ehlers, 2017], SDP [Raghunathan et al., 2018], SMT [Katz et al., 2017], and MILP [Tjeng et al., 2017]. However, most of this work is for forward verification (i.e., bounding the NN output given a
(a) No branching
Figure 6: OOD Detection. Green: Empirical ID inputs (1 million samples). Blue: border of the verified preimage. Branching is union of preimages for each of 4 quadrants.
(b) Branching
Table 4: OOD Results. Comparison of methods for over-approximating the preimage of the OOD region (Figure 6). A lower approximation ratio (APPROX RAT.) is better.
METHOD	INPUT BRANCH.	APPROX RATIO	TIME (SEC)
MILP	NO	1.47	1562.26
INVPROP	NO	4.39	3.77 ± 0.02
INVPROP	YES	1.14	12.02 ± 0.06
set of inputs). Our work strictly generalizes bound propagation as presented in Xu et al. 2021 since setting  $\gamma = 0$  in Theorem 1 recovers its results.
Formal Analysis of the Input Domain Prior work has studied trying to determine the inverse of a neural network [Kindermann and Linden, 1990, Saad and Wunsch, 2007], though these methods are prohibitively slow. Zhang et al. [2018b] used an LP solver to compute changes to a given input that would lead to a change in the classification result while adhering to additional criteria. Yang et al. [2021b] encode the overapproximated network in an LP problem to tighten bounds by incorporating the output constraint. They require an LP solver and we compare our performance against theirs in Section 4.2. Zhong et al. [2023], Wu et al. [2022] similarly tighten intermediate bounds by using an LP solver. Dimitrov et al. [2022] compute input regions that only consist of adversarial examples (i.e., adhere to a given output constraint) that are maximized using an LP solver. Finally, a concurrent work Zhang et al. [2023] targets under- and over-approximating the preimage via LiRPA-based forward verification [Xu et al., 2020] with input-space and ReLU-space partitioning; since our work can be viewed as a generalization of LiPRA, their work may benefit from our results of tightening the intermediate layer bounds using output constraints during bound propagation.
Formal Verification for Neural Feedback Loops. Our control application was motivated by the growing body of work on backward reachability analysis and over-approximating states that result in a target set [Everett et al., 2021]. The original method solely utilized input constraints for deriving intermediate bounds. The later development of Everett et al. 2022 improved upon this by optimizing  $l^{(0)}$  and  $\pmb{u}^{(0)}$  with respect to output bounds but their implementation faces numerical instability when applied for many iterations. We also support the partitioning over the input space introduced by Rober et al. 2022a. The work of Vincent and Schwager 2021 explores utilizing an LP with complete neuron branching to verify control safety, which can be viewed as a domain-specific implementation of our MILP formulation of the inverse verification problem.
Certified OOD detection. There is a wide variety of OOD detection systems [Yang et al., 2021a, Salehi et al., 2022]. Commonly, they are evaluated empirically based on known OOD data [Wang et al., 2022, Liang et al., 2018, Chen et al., 2021]. Therefore, they fail to provide verified guarantees for their effectiveness. In fact, many OOD detection systems are susceptible to adversarial attacks [Sehwag et al., 2019, Chen et al., 2022]. Meinke et al. [2021] show how to verify robustness to adversarial examples around given input samples. Berrada et al. [2021] develop a general framework for certifying properties of output distributions of neural networks given constraints on the input distribution. However, this work is still constrained to verifying properties of the outputs given constraints (albeit probabilistic) on the inputs, and INVPROP is able to certify arbitrary regions of the input space that lead to confident predictions.
We present the challenge of over-approximating neural network preimages and provide an efficient algorithm to solve it. By doing so, we demonstrate strong performance on multiple application areas. We believe there is a large scope for future investigation and new applications of INVPROP.
Limitations For higher-dimensional instances, our method would best work for box overapproximations (2d hyperplanes in  $d$  dimensions) and would struggle at more complicated shapes such as spheres. We expect more complex problems will benefit from a domain-specific strategy. Moreover, iteratively refining all of the intermediate bounds utilizing INVPROP faces a quadratic dependence on network depth, as opposed to a linear dependence in traditional forward verification. To mitigate this, output constraints could only be applied to layers close to the output, as we used for robustness. Finally, our branching strategy might not scale to higher dimensions, though this trade-off is well-studied [Bunel et al., 2020, Wang et al., 2021, Palma et al., 2023].
Potential Negative Social Impact Our work improves reliable ML through facilitating systems that provably align with practitioner expectations and we expect INVPROP to have positive societal impact. We acknowledge that our improvements may be repurposed as model attacks, though we believe the positive use cases of our technique greatly outweigh current speculation of misuse.
Acknowledgements We thank Michael Everett and Nicholas Rober for helpful discussion and feedback on the paper. Huan Zhang acknowledges the support of the Schmidt Futures AI2050 Early Career Fellowship.
Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. Interpreting neural network judgments via minimal, stable, and symbolic corrections. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018b. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/300891a62162b960cf02ce3827bb363c-Paper.pdf.
Xiyue Zhang, Benjie Wang, and Marta Kwiatkowska. Provable preimage under-approximation for neural networks. arXiv preprint arXiv:2305.03686, 2023.
Xiangru Zhong. Yolo-benchmark. https://github.com/xiangruzh/Yolo-Benchmark, 2023.
Yuyi Zhong, Quang-Trung Ta, and Siau-Cheng Khoo. Arena: Enhancing abstract refinement for neural network verification. In Cezara Dragoi, Michael Emmi, and Jingbo Wang, editors, Verification, Model Checking, and Abstract Interpretation, pages 366-388, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-24950-1.
A OOD Detection
Figure 7: To improve our model's ability to detect data outside the training distribution (of green and red points), we randomly sample points that are far from every training point. In the right contour, we see that the model's confidence of in-distribution increases as we approach the centers of the distribution, demonstrating our model is more calibrated.
For the OOD benchmark, we train a neural network on two clusters of data. Standard training would lead to an uncalibrated predictor, we add in OOD data through sampling the input space and rejecting inputs that are too close to any training data point. Training a classifier on this dataset induces the contour on the right, which is a calibrated classifier we can certify properties over. For our experiments, we consider a high-confidence output as  $90\%$  confidence of the data being in-distribution.
B Example
Example 1. Consider the toy neural network in Figure 8 under  $\mathcal{X} = [-2,2]$  and  $S_{\mathrm{out}} = \{y:1\leq y\leq$ $1.02\}$ . Suppose we wanted to find the tightest  $[l_0^{(1)},u_0^{(1)}]$  (bounds on  $x_0^{(1)})$ . If we only enforce the constraints preceding the ReLU (standard bound propagation), we get  $[-2,2]$ . If we only enforce the constraints following the ReLU, we get  $(-\infty ,1.02]$ . However, when we utilize all of the constraints, we find that the true intermediate bounds are [0, 0.01], which is over 100 times tighter than the intersection of the two previous methods (all derivations provided below). Therefore, new techniques are necessary for optimizing intermediate bounds in this setting.
B.1 Deriving Intermediate Bounds for Example B
**Preceding Constraints.** The linear layer gives us  $\boldsymbol{x}_0^{(1)} = \hat{\boldsymbol{x}}_0^{(0)}$ . Since the only constraint we have is  $\hat{\boldsymbol{x}}_0^{(0)} \in [-2, 2]$ , the tightest intermediate bounds we can derive are  $[-2, 2]$ .
Following Constraints. Since the output of a ReLU is non-negative, we know that  $\hat{\pmb{x}}_0^{(1)},\hat{\pmb{x}}_1^{(1)}\geq 0$  Since  $\pmb {x}_0^{(2)} = \hat{\pmb{x}}_0^{(1)} + \hat{\pmb{x}}_1^{(1)}$  and the output constraint enforces  $\pmb {x}_0^{(2)}\leq 1.02$  , we derive that  $\hat{\pmb{x}}_0^{(1)}\leq 1.02$  If we set  $\hat{\pmb{x}}_0^{(1)} = 1.02 - \hat{\pmb{x}}_1^{(1)}$  , we see that  $\hat{\pmb{x}}_0^{(1)}$  can take the entire interval [0, 1.02]. Since the only constraint we have on  $\pmb {x}_0^{(1)}$  is that ReLU  $(\hat{\pmb{x}}_0^{(1)}) = \pmb{x}^{(1)}$  , our desired interval is the preimage of [0, 1.02]. This means that  $\pmb {x}_0^{(1)}$  can take any value in the interval  $(- \infty ,1.02]$
All Constraints. We first note that by the first linear layer, we have  $\boldsymbol{x}_1^{(1)} = \boldsymbol{x}_0^{(1)} + 1$ . Therefore, if  $\boldsymbol{x}_0^{(1)}$  is less than 0, then  $\boldsymbol{x}_1^{(1)}$  is less than 1, which means  $\mathsf{ReLU}(\boldsymbol{x}_0^{(1)}) + \mathsf{ReLU}(\boldsymbol{x}_1^{(1)})$  is less than 1, which contradicts the output constraint. If  $\boldsymbol{x}_0^{(1)}$  is always non-negative, then we have that the output is equivalent to  $\mathsf{ReLU}(\boldsymbol{x}_0^{(1)}) + \mathsf{ReLU}(\boldsymbol{x}_0^{(1)} + 1) = 2\boldsymbol{x}_0^{(1)} + 1$ . Therefore, the output constraint implies  $\boldsymbol{x}_0^{(1)} \leq 0.01$ . Any  $\boldsymbol{x}_0^{(1)} \in [0, 0.01]$  is achievable by setting the input to the desired value.
Figure 8: A simple neural network from  $\mathbb{R}$  to  $\mathbb{R}$ . Every node is the sum of its incoming nodes unless the edge is labeled ReLU.
C Optimizing Intermediate Bounds
We present a generalized theorem which provides a lower bound for any linear combination of  $\hat{\pmb{x}}^{(0)}$  and  $\pmb{x}^{(i)}$ . We prove this theorem in Appendix D. We note that by selecting the coefficients of these variables (named  $\pmb{c}^{(i)}$  for  $i \in \{0,1,\dots,L-1\}$ ), we recover the two following functionalities which we name  $g_{c}(\alpha,\gamma)$  and  $g_{i,j}^{b}(\alpha,\gamma)$  (respectively).