SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.3 (Soundness). If the CHECK sub-procedure is sound, then the value  $\mathbf{x}^{\mathbf{A}}$  returned by Algorithm 1 is a robust explanation - this satisfies Equation (1) of Definition 2.1.
Theorem 3.4 (Optimality). If the CHECK sub-procedure is sound and complete, then the robust explanation  $\mathbf{x}^{\mathbf{A}}$  returned by Algorithm 1 is optimal - this satisfies Equation (2) of Definition 2.1.
Intuitively, optimality holds because if it is not possible for an  $\epsilon$ -perturbation on some feature  $\chi^i$  in explanation  $\mathbf{x}^{\mathbf{A}}$  to change the prediction, then it will be added to the irrelevant features  $\mathbf{x}^{\mathbf{B}}$  when feature  $\chi^i$  is considered during the execution of Algorithm 1.
Proposition 3.5 (Complexity). Given a  $d$ -dimensional input  $\mathbf{x}$  and a network  $f$ , the complexity of computing an optimal robust explanation using the VERIX algorithm is  $O(d \cdot P(f))$ , where  $P(f)$  is the cost of checking a specification (of the form in Equation (3)) over  $f$ .
An optimal explanation can be achieved from one traversal of input features as we are computing the local optima. If  $f$  is piecewise-linear, checking a specification over  $f$  is NP-complete [30]. We remark that such complexity analysis is closely related to that of the deletion-based method [8]; here we particularly focus on how a network  $f$  and a  $d$ -dimensional input  $\mathbf{x}$  would affect the complexity of computing an optimal robust explanation.
3.3 Feature-level sensitivity traversal
While Example 3.1 used a simple sequential order for illustration purpose, we introduce a heuristic based on feature-level sensitivity, inspired by the occlusion method [64], to produce actual traversals.
Definition 3.6 (Feature-Level Sensitivity). Given an input  $\mathbf{x} = \langle \chi^1, \dots, \chi^d \rangle$  and a network  $f$ , the feature-level sensitivity (in classification for a label  $c$  or in regression for a single quantity) for a feature  $\chi^i$  with respect to a transformation  $\mathcal{T}$  is
$$
\text {s e n s i t i v i t y} \left(\chi^ {i}\right) = f _ {(c)} (\mathbf {x}) - f _ {(c)} \left(\mathbf {x} ^ {\prime}\right), \tag {5}
$$
where  $\mathbf{x}'$  is  $\mathbf{x}$  with  $\chi^i$  replaced by  $\mathcal{T}(\chi^i)$ .
Typical transformations include deletion  $(\mathcal{T}(\chi) = 0)$  and reversal  $(\mathcal{T}(\chi) = \overline{\chi} - \chi)$ , where  $\overline{\chi}$  is the upper bound for feature  $\chi$ . Intuitively, we measure how sensitive (in terms of an increase or decrease) a model's confidence is to each individual feature. Given sensitivity values with respect to some transformation, we rank the feature indices into a traversal order from least to most sensitive.
We have implemented the VERIX algorithm in Python, using the Marabou neural network verification tool [31] to implement the CHECK sub-procedure of Algorithm 1 (Line 10). The VERIX code is available at https://github.com/NeuralNetworkVerification/VeriX. We trained fully-connected and convolutional networks on the MNIST [34], GTSRB [47], and TaxiNet [29] datasets for classification and regression tasks. Model specifications are in Appendix D. Experiments were performed on a workstation equipped with AMD Ryzen 75700G CPUs running Fedora 37. We set a time limit of 300 seconds for each CHECK call.
(a) "keep right, 50 mph, road work, no passing"
Figure 4: Optimal robust explanations (green) from VERIX on GTSRB (left) and MNIST (right) images. (b) Pixels in yellow are those in the explanation to rule out different counterfactuals.
(b) Handwritten digit “1”, not “8”, not “5”, not “2”
4.1 Example explanations for image recognition benchmarks
Figure 4 shows examples of VERIX explanations for GTSRB and MNIST images. The convolutional model trained on GTSRB and fully-connected model on MNIST are in Appendix D, Tables 8 and 6. Aligning with our intuition, VERIX can distinguish the traffic signs (no matter a circle, a triangle, or a square in Figure 6a) from their surroundings well; the explanations focus on the actual contents within the signs, e.g., the right arrow denoting "keep right" and the number 50 as in "50 mph". Interestingly, for traffic signs consisting of irregular dark shapes on a white background such as "road work" and "no passing", VERIX discovers that the white background contains the essential features. We observe that MNIST explanations for the fully-connected model are in general more scattered around the background because the network relies on the non-existence of white pixels to rule out different counterfactuals (Figure 4b shows which pixels in the explanation have associated counterfactuals with predictions of "8", "5", and "2", respectively), whereas GTSRB explanations for the convolutional model can safely disregard the surrounding pixels outside the traffic signs.
4.2 Visualization of the effect of varying perturbation magnitude  $\epsilon$
A key parameter of VERIX is the perturbation magnitude  $\epsilon$ . When  $\epsilon$  is varied, the irrelevant features change accordingly. Figure 5 visualizes this, showing how the irrelevant features change when  $\epsilon$  is tightened from  $100\%$  to  $10\%$  and further to  $5\%$ . As  $\epsilon$  decreases, more pixels become irrelevant. Intuitively, the VERIX explanation helps reveal how the network classifies this image as "0". The deep blue pixels are those that are irrelevant with  $\epsilon = 100\%$ . Light blue pixels are more sensitive, allowing perturbations of only  $10\%$ . The light yellow pixels represent  $5\%$ , and bright yellow are pixels that cannot even be perturbed  $5\%$  without changing the prediction. The resulting pattern is roughly consistent with our intuition, as the shape of the "0" can be seen embedded in the explanation.
(a) MNIST "0" (b)  $\epsilon: 1 \to 0.1$  (c)  $\epsilon: 1 \to 0.05$
Figure 5: Expansion of the irrelevant pixels when perturbation magnitude  $\epsilon$  decreases from  $100\%$  to  $10\%$  and further to  $5\%$  (from deep blue to light yellow). Each brighter color denotes the pixels added when moving to the next smaller  $\epsilon$ , e.g.,  $100\%$ ,  $90\%$ ,  $80\%$  and so on.
We remark that determining a suitable magnitude  $\epsilon$  is non-trivial because if  $\epsilon$  is too loose, explanations may be too conservative, allowing very few pixels to change. On the other hand, if  $\epsilon$  is too small, nearly the whole set of pixels could become irrelevant. For instance, in Figure 5, if we set  $\epsilon$  to  $1\%$  then all pixels become irrelevant – the classifier's prediction is robust to  $1\%$  -perturbations. The "color map" we propose makes it possible to visualize not only the explanation but also how it varies with  $\epsilon$ . The user then has the freedom to pick a specific  $\epsilon$  depending on their application.
4.3 Sensitivity vs. random traversal to generate explanations
To show the advantage of the feature-level sensitivity traversal, Figure 6 compares VERIX explanations using sensitivity-based and random traversals. Sensitivity, as shown in the heatmaps of Figures 6a and 6b, prioritizes pixels that have more influence on the network's decision, whereas a random ranking is simply a shuffling of all the pixels. We mention that, to compute the sensitivity, we used pixel deletion  $\mathcal{T}(\chi) = 0$  for GTSRB and reversal  $\mathcal{T}(\chi) = \overline{\chi} -\chi$  for MNIST (deleting background pixels of MNIST images may have little effect as they often have zero values). We observe that the sensitivity traversal generates much more sensible explanations. In Figure 6c, we
(a) GTSRB "priority road", sensitivity, explanations.
(b) MNIST "0", sensitivity, explanations.
(c) Sensitivity vs. random traversals in explanation size (number of pixels).
Figure 6: Comparing VERIX explanations from sensitivity and random traversals. (a)(b) 1st column: the original image; 2nd column: sensitivity heatmap; 3rd and 4th columns: explanations from sensitivity (green) and random (red) traversals. (c) Each blue triangle is a unique explanation size from sensitivity ranking, and each set of circles shows explanation sizes from 100 random rankings.
(a) Original, VERIX, Anchors, segmentation
Figure 7: Comparing VERIX (green) to Anchors (red) on two different versions of a "keep right" traffic sign from the GTSRB dataset, one with strong light in the background and one without.
(b) Original, VERIX, Anchors, misclassified as "yield"
compare explanation sizes for the first 10 images (to avoid potential selection bias) of the MNIST test set. For each image, we show 100 explanations from random traversal compared to the deterministic explanation from sensitivity traversal. We observe that the latter is almost always smaller, often significantly so, suggesting that sensitivity-based traversals are a reasonable heuristic for attempting to approach globally optimal explanations.
4.4 VERIX vs. existing approaches
We compare VERIX with Anchors [43]. Figure 7 shows both approaches applied to two different "keep right" traffic signs. Anchors performs image segmentation and selects a set of the segments as the explanation, making its explanations heavily dependent on the quality of the segmentation. For instance, distractions such as strong light in the background may compromise the segments (Figure 7a, last column) thus resulting in less-than-ideal explanations, e.g., the top right region of the anchor (red) is outside the actual traffic sign. Instead, VERIX utilizes the model to compute the sensitivity traversal, often leading to more reasonable explanations. Anchors is also not designed to provide formal guarantees. In fact, replacing the background of an anchor explanation – used by the original paper [43] to justify "almost" guarantee – can change the classification. For example, the last column of Figure 7b is classified as "yield" with confidence  $99.92\%$ . We conducted a quantitative evaluation on the robustness of explanations, as shown in Table 2 under "robust wrt # perturbations". When quantifying the robustness of Anchors, we generate 100 explanations for 100 MNIST and 100 GTSRB images separately, and impose 10, 100, 500, 1000 perturbations on each explanation by overlapping it with other images in the same dataset. If the prediction remains unchanged, then the explanation is robust against these perturbations. We notice that the robustness of Anchors decreases quickly when the number of perturbations increases, and when imposing 1000 perturbations per image, only 37 MNIST and 15 GTSRB explanations out of 100 are robust. This was not done for LIME, since it would be unfair to LIME as it does not have robustness as a primary goal. In contrast, VERIX provides provable robustness guarantees against any  $\epsilon$ -perturbations in the input space.
Two key metrics for evaluating the quality of an explanation are the size and the generation time. Table 2 shows that overall, VERIX produces much smaller explanations than Anchors and LIME but
Table 2: VERIX vs. existing approaches, showing average explanation size (number of pixels), generation time (seconds), and an empirical evaluation of the robustness of the produced explanations against perturbations. In VERIX,  $\epsilon$  is set to  $5\%$  for MNIST and  $0.5\%$  for GTSRB.
MNIST	GTSRB
size	time	robust wrt # perturbations	size	time	robust wrt # perturbations
VERIX	sensitivity	180.6	174.77	100%	357.0	853.91	100%
random	294.2	157.47	100%	383.5	814.18	100%
Anchors	494.9	13.46	10	100	500	1000	557.7	26.15	10	100
85%	52%	40%	37%	72%	23%
LIME	432.8	9.03	-	452.9	7.85	-
(a) 3.94 (3.91) m
(b) 5.09 (4.97) m
Figure 8: VERIX applied to the TaxiNet dataset - each column includes a sampled camera view (top), its VERIX explanation (bottom), and the cross-track estimate of the form "actual (estimate) meters".
(c) 3.32 (3.38) m
(d) 1.75 (1.55) m
(e) 1.58 (1.60) m
(f) 3.88 (4.07) m
takes much longer (our limitation) to perform the computation necessary to ensure formal guarantees – this thus provides a trade-off between time and quality. Table 2 also shows that sensitivity traversal produces significantly smaller sizes than its random counterpart with only a modest overhead in time.
4.5 Deployment in vision-based autonomous aircraft taxiing
We also applied VERIX to the real-world safety-critical aircraft taxiing scenario [29] shown in Figure 9. The vision-based autonomous taxiing system needs to make sure the aircraft stays on the taxiway utilizing only pictures taken from the camera on the right wing. The task is to evaluate the cross-track position of the aircraft so that a controller can adjust its position accordingly. To achieve this, a regression model is used that takes a picture as input and produces an estimate of the current position. A preprocessing step crops out the sky and aircraft nose, keeping the crucial taxiway region (in the red box). This is then downsampled into a gray-scale image of size  $27 \times 54$  pixels. We label each image with its corresponding lateral distance to the runway centerline together with the taxiway
Figure 9: An autonomous aircraft taxiing scenario [29]. Pictures taken from the camera fixed on the right wing are cropped (red box) and downsampled.
heading angle. We trained a fully-connected regression network on this dataset, referred to as the TaxiNet model (Appendix D.3, Table 10), to predict the aircraft's cross-track distance.
Figure 8 exhibits VERIX applied to the TaxiNet dataset, including a variety of taxiway images with different heading angles and number of lanes. For each taxiway, we show its VERIX explanation accompanied by the cross-track estimate. We observe that the model is capable of detecting the more remote line – its contour is clearly marked in green. Meanwhile, the model is mainly focused on the centerline (especially in Figures 8b, 8d, 8e, and 8f), which makes sense as it needs to measure how far the aircraft has deviated from the center. Interestingly, while we intuitively might assume that the model would focus on the white lanes and discard the rest, VERIX shows that the bottom middle region is also crucial to the explanation (e.g., as shown in Figures 8a and 8c). This is because the model must take into account the presence and absence of the centerline. This is in fact in consistent with our observations about the black background in MNIST images (Figure 1). We used  $\epsilon = 5\%$  for these explanations, which suggests that for modest perturbations (e.g., brightness change due to different weather conditions) the predicted cross-track estimate will remain within an acceptable discrepancy, and taxiing will not be compromised.
Table 3: Complete vs. incomplete verification, showing average explanation size (number of pixels) and generation time (seconds). As in Table 2,  $\epsilon$  is set to  $5\%$  for MNIST and  $0.5\%$  for GTSRB.
MNIST	GTSRB
complete	incomplete	complete	incomplete
size	time	size	time	size	time	size	time
VERIX	sensitivity	180.6	174.77	330.8	106.11	357.0	853.91	385.2	858.39
random	294.2	157.47	467.4	100.19	383.5	814.18	400.2	816.38
4.6 Using sound but incomplete analysis as the CHECK sub-procedure
We also evaluate using sound but incomplete analysis, such as DeepPoly [46], as the CHECK subprocedure of Algorithm 1. In Table 3, we report the quantitative comparison.
We observe that, in general, explanations are larger when using incomplete analysis, but have shorter generation times. In other words, there is a trade-off between complete and incomplete analyses with respect to explanation size and generation time. This makes sense as the DeepPoly analysis deploys bound propagation without complete search, so when the verifier returns Unknown on a certain feature (Line 10 of Algorithm 1), it is unknown whether perturbing this feature (together with the current irrelevant set  $\mathbf{B}$ ) can or cannot change the model's prediction. Therefore, such features are put into the explanation set  $\mathbf{A}$ , as only  $\epsilon$ -robust features are put into the irrelevant set  $\mathbf{B}$ . This results in larger explanations. We re-emphasize that such explanations are no longer locally minimal, but they are still sound as  $\epsilon$ -perturbations on the irrelevant features will definitely not alter the prediction. Another observation is that on MNIST, the discrepancy between explanations (size and time) from complete and incomplete verifiers is more pronounced than the discrepancy on GTSRB. This is because here we set  $\epsilon$  to  $5\%$  for MNIST and to  $0.5\%$  for GTSRB. So when generating explanations on MNIST, complete search is often required for a precise answer due to the larger  $\epsilon$  value, whereas on GTSRB, bound propagation is often enough due to the small  $\epsilon$ , and thus complete search is not always needed.
Due to space limitations, additional analyses of runtime performance and scalability for both complete and incomplete verifiers are included in Appendix B.
Earlier work on formal explanations [40] has the following limitations. First, in terms of scalability, they can only handle simple machine learning models such as naive Bayes classifiers [39], random forests [28, 6], decision trees [27], and boosted trees [24, 25]. In particular, [26] addresses networks but with very simple structure (e.g., one hidden layer of 15 or 20 neurons to distinguish two MNIST digits). In contrast, VERIX works with models applicable to real-world safety-critical scenarios. Second, the size of explanations can be unnecessarily large. As a workaround, approximate explanations [52, 53] are proposed as a generalization to provide probabilistic (thus compromised) guarantees of prediction invariance. VERIX, by using feature-level sensitivity ranking, produces reasonably-sized explanations with rigorous guarantees. Third, these formal explanations allow any possible input in feature space, which is not necessary or realistic. For this, [33] brings in bounded perturbations for NLP models but their  $k$ -NN box closure essentially chooses a finite set of the  $k$  closest tokens for each word so the perturbation space is intrinsically discrete. In other words, their method will not scale to our high-dimensional image inputs with infinite perturbations. One the other hand, those model-agnostic feature selection methods such as Anchors [43] and LIME [42] can work with large networks, however, they cannot provide strong guarantees like our approach can. A brief survey of work in the general area of neural network verification appears in Appendix C.
We have presented the VERIX framework for computing optimal robust explanations and counterfactuals along the decision boundary. Our approach provides provable guarantees against infinite and continuous perturbations. A possible future direction is generalizing to other crucial properties such as fairness. Recall the diagnosis model in Section 1; our approach can discover potential bias (if it exists) by including skin tones ("Fitzpatrick I-II, III-IV, V-VI" [19]) in the produced explanation; then, a dermatologist could better interpret whether the model is producing a fair and unbiased result.
This work was supported in part by NSF grant 2211505, the Stanford Center for AI Safety, and the Stanford Center for Automated Reasoning.
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
[2] Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. Optimization and abstraction: a synergistic approach for analyzing neural network robustness. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 731-744, 2019.
[3] Stanley Bak, Changliu Liu, Taylor T. Johnson, Christopher Brix, and Mark Müller. The 3rd international verification of neural networks competition (vnn-comp), 2022. https://sites.google.com/view/vnn2022.
[4] Shahaf Bassan and Guy Katz. Towards formal approximated minimal explanations of neural networks. arXiv preprint arXiv:2210.13915, 2022.
[5] Elena Botoeva, Panagiotis Kouvaros, Jan Kronqvist, Alessio Lomuscio, and Ruth Misener. Efficient verification of relu-based neural networks via dependency analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3291-3299, 2020.
[6] Ryma Boumazouza, Fahima Cheikh-Alili, Bertrand Mazure, and Karim Tabia. Asteryx: A model-agnostic sat-based approach for symbolic and score-based explanations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 120–129, 2021.
[7] Rudy Bunel, Ilker Turkaslan, Philip HS Torr, M Pawan Kumar, Jingyue Lu, and Pushmeet Kohli. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research, 21:1-39, 2020.
[8] John W Chinneck and Erik W Dravnieks. Locating minimal infeasible constraint sets in linear programs. ORSA Journal on Computing, 3(2):157-168, 1991.
[9] François Chollet et al. Keras. https://keras.io, 2015.
[10] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Advances in Neural Information Processing Systems, 35:18157-18167, 2022.
[11] Adnan Darwiche and Auguste Hirth. On the reasons behind decisions. In Proceedings of the 24th European Conference on Artificial Intelligence, 2020.
[12] Sumanth Dathathri, Sicun Gao, and Richard M Murray. Inverse abstraction of neural networks using symbolic interpolation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3437-3444, 2019.
[13] Alessandro De Palma, Harkirat S Behl, Rudy Bunel, Philip Torr, and M Pawan Kumar. Scaling the convex barrier with active sets. In International Conference on Learning Representations, 2021.
[60] Haoze Wu, Aleksandar Zeljic, Guy Katz, and Clark Barrett. Efficient neural network analysis with sum-of-infeasibilities. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 143-163. Springer, 2022.
[61] Min Wu and Marta Kwiatkowska. Robustness guarantees for deep neural networks on videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 311-320, 2020.
[62] Min Wu, Matthew Wicker, Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. A game-based approximate verification of deep neural networks with provable guarantees. Theoretical Computer Science, 807:298-329, 2020.
[63] Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. Artificial intelligence in healthcare. Nature biomedical engineering, 2(10):719-731, 2018.
[64] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
[65] Tom Zelazny, Haoze Wu, Clark Barrett, and Guy Katz. On optimizing back-substitution methods for neural network verification. In Formal Methods in Computer Aided Design (FMCAD), pages 17-26. IEEE, 2022.
[66] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 4944–4953, 2018.
[67] Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. Interpreting neural network judgments via minimal, stable, and symbolic corrections. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 4879-4890, 2018.
A Proofs for Section 3.2