SlowGuess/NeurIPS_2023_OCR huggingface dataset:
VERIX: Towards Verified Explainability of Deep Neural Networks
minwu@cs.stanford.edu
Haoze Wu
haozewu@cs.stanford.edu
We present VERIX (VERIFIED explainability), a system for producing optimal robust explanations and generating counterfactuals along decision boundaries of machine learning models. We build such explanations and counterfactuals iteratively using constraint solving techniques and a heuristic based on feature-level sensitivity ranking. We evaluate our method on image recognition benchmarks and a real-world scenario of autonomous aircraft taxiing.
Broad deployment of artificial intelligence (AI) systems in safety-critical domains, such as autonomous driving [18] and healthcare [63], necessitates the development of approaches for trustworthy AI. One key ingredient for trustworthiness is explainability: the ability for an AI system to communicate the reasons for its behavior in terms that humans can understand.
Early work on explainable AI includes well-known model-agnostic explainers which produce explanations that remain valid for nearby inputs in feature space. In particular, LIME [42] and SHAP [38] learn simple, and thus interpretable, models locally around a given input. Following LIME, work on Anchors [43] attempts to identify a subset of such input explanations that are (almost) sufficient to ensure the corresponding output value. Such approaches can produce explanations efficiently, however, they do not provide any formal guarantees and are thus inappropriate for use in high-risk scenarios. For instance, in healthcare, if a diagnosis model used by a dermatologist has an explanation claiming that it depends only on a patient's skin lesions (such as " PLAque", "Pustule", and "ulcer" in [10]), yet in actuality, patients with similar such lesions but different skin tones ("Fitzpatrick I-II, III-IV, V-VI" [19]) receive dissimilar diagnoses, then the explanation is not only wrong, but may actually mask bias in the model. Another drawback of model-agnostic approaches is that they often depend on access to training data, which may not always be available (perhaps due to privacy concerns). And even if available, distribution shift can compromise the results.
Recent efforts towards formal explainable AI [40] aim to compute rigorously defined explanations that can guarantee soundness, in the sense that fixing certain input features is sufficient to ensure the invariance of a model's prediction. However, their work only considers unbounded perturbations, which may be too course-grained to be useful (for other limitations, see Section 5). To mitigate those drawbacks, [33] bring in two types of bounded perturbations,  $\epsilon$ -ball and  $k$ -NN box closure, and show how to compute optimal robust explanations with respect to these perturbations for natural
(a) Original "2"
(b) Segmentation
(c) VERIX
Figure 1: Intuition for our VERIX approach: (a) An MNIST handwritten “2”; (b) Segmentation of “2” into 3 partitions; (c) Our VERIX explanation (green pixels) of “2”; (d)(e)(f) Masking white pixels or whitening black pixels may turn “2” into possible counterfactuals.
(d) “2” into “7”
(e) “2” into “0”
(f) “2” into “3”
language processing (NLP) models.  $k$ -NN box closure essentially chooses a finite set of the  $k$  closest tokens for each word in a text sample, so the perturbation space is intrinsically discrete; on the other hand,  $\epsilon$ -ball perturbations provide a way to handle a continuous word embedding (though the authors of [33] suggest that these may be more cumbersome in NLP applications and focus on  $k$ -NN box perturbations in their experimental results).
In this paper, we present VERIX (VERIFIED explainability), a tool for producing optimal robust explanations and generating counterfactuals along decision boundaries of deep neural networks. Our contributions can be summarized as follows.
- We utilize constraint solving techniques to compute robust and optimal explanations with provable guarantees against infinite and continuous perturbations in the input space.
- We bring a feature-level sensitivity traversal into the framework to efficiently approximate global optima, which improves scalability for high-dimensional inputs and large models.
- We note for the first time the relationship between our explanations and counterfactuals, and show how to compute such counterfactuals automatically at no additional cost.
- We provide an extensive evaluation on a variety of perception models, including a safety-critical real-world autonomous aircraft taxiing application.
We start by providing intuition for our VERIX approach by analyzing an example explanation in Figure 1. This explanation is generated for a fully-connected model trained on the MNIST dataset. Model-agnostic explainers such as Anchors [43] rely on partitioning an image into a disjoint set of segments and then selecting the most prominent segment(s). Figure 1b shows "2" divided into 3 parts using k-means clustering [37]. Based on this segmentation, the purple and yellow parts would be chosen for the explanation, suggesting that the model largely relies on these segments to make its decision. This also matches our intuition, as a human would immediately identify these pixels as containing information and disregard the background. However, does this mean it is enough to focus on the salient features when explaining a classifier's prediction? Not necessarily. VERIX's explanation is highlighted in green in Figure 1c. It demonstrates that whatever is prominent is important but what is absent in the background also matters. We observe that VERIX not only marks those white pixels forming the silhouette of "2" but also includes some background pixels that might affect the prediction if changed. For instance, neglecting the bottom white pixels may lead to a misclassification as a "7"; meanwhile, the classifier also needs to check if the pixels along the left and in the middle are not white to make sure it is not "0" or "3". While Figures 1d, 1e, and 1f are simply illustrative to provide intuition about why different parts of the explanation may be present, we remark that explanations from VERIX are produced automatically and deterministically.
2 VERIX: Verified eXplainability
Let  $f$  be a neural network and  $\mathbf{x}$  a  $d$ -dimensional input vector of features  $\langle \chi^1, \dots, \chi^d \rangle$ . We use  $\Theta(\mathbf{x})$ , or simply  $\Theta$ , when the context is clear, to denote its set of feature indices  $\{1, \dots, d\}$ . We write  $\mathbf{x}^{\mathbf{A}}$  where  $\mathbf{A} \subseteq \Theta(\mathbf{x})$  to denote only those features indexed by indices in  $\mathbf{A}$ . We denote model prediction as  $f(\mathbf{x}) = c$ , where  $c$  is a single quantity in regression or a label among others ( $c \in C$ ) in classification. For the latter, we use  $f_c(\mathbf{x})$  to denote the confidence value (pre- or post- softmax) of classifying as  $c$ , i.e.,  $f(\mathbf{x}) = \arg \max_{c} f_c(\mathbf{x})$ . Depending on different application domains,  $\mathbf{x}$  can be an image consisting of  $d$  pixels as in our case or a text comprising  $d$  words as in NLP [33]. In this paper, we focus on perception models. This has the additional benefit that explanations in this context are self-illustrative and thus easier to understand.
2.1 Optimal robust explanations
Existing work such as abductive explanations [26], prime implicants [45], and sufficient reasons [11] define a formal explanation as a minimal subset of input features that are responsible for a model's decision, in the sense that any possible perturbations on the rest features will never change prediction. Building on this, [33] introduces bounded perturbations and computes "distance-restricted" such explanations for NLP models. Our definition closely follows [33] except: (1) rather than minimizing an arbitrary cost function, we consider the uniform case (i.e., the same cost is assigned to each feature) so as to compute the smallest number of features; (2) we focus on continuous  $\epsilon$ -ball perturbations not discrete  $k$ -NN box closure; (3) we allow for bounded variation (parameterized by  $\delta$ ) in the output to accommodate both classification (set  $\delta$  to 0) and regression ( $\delta$  could be some pre-defined hyper-parameter quantifying the allowable output change) whilst [33] focuses on sentiment analysis (i.e., binary classification).
Definition 2.1 (Optimal Robust Explanation). Given a neural network  $f$ , an input  $\mathbf{x}$ , a manipulation magnitude  $\epsilon$ , and a discrepancy  $\delta$ , a robust explanation with respect to norm  $p \in \{1, 2, \infty\}$  is a set of input features  $\mathbf{x}^{\mathbf{A}}$  such that if  $\mathbf{B} = \Theta(\mathbf{x}) \setminus \mathbf{A}$ , then
$$
\forall \mathbf {x} ^ {\mathbf {B} ^ {\prime}}. \left\| \mathbf {x} ^ {\mathbf {B}} - \mathbf {x} ^ {\mathbf {B} ^ {\prime}} \right\| _ {p} \leq \epsilon \Rightarrow | f (\mathbf {x}) - f \left(\mathbf {x} ^ {\prime}\right) | \leq \delta , \tag {1}
$$
where  $\mathbf{x}^{\mathbf{B}'}$  is some perturbation on features  $\mathbf{x}^{\mathbf{B}}$  and  $\mathbf{x}'$  is the input variant combining  $\mathbf{x}^{\mathbf{A}}$  and  $\mathbf{x}^{\mathbf{B}'}$ . In particular, we say that the robust explanation  $\mathbf{x}^{\mathbf{A}}$  is optimal if
$$
\forall \chi \in \mathbf {x} ^ {\mathbf {A}}. \exists \mathbf {x} ^ {\mathbf {B} ^ {\prime}}, \chi^ {\prime}. \| (\mathbf {x} ^ {\mathbf {B}} \oplus \chi) - \left(\mathbf {x} ^ {\mathbf {B} ^ {\prime}} \oplus \chi^ {\prime}\right) \| _ {p} \leq \epsilon \wedge | f (\mathbf {x}) - f \left(\mathbf {x} ^ {\prime}\right) | > \delta , \tag {2}
$$
where  $\chi^{\prime}$  is some perturbation of  $\chi$  and  $\oplus$  denotes concatenation of two features.
We refer to  $\mathbf{x}^{\mathrm{B}}$  as the irrelevant features. Intuitively, perturbations bounded by  $\epsilon$  imposed upon the irrelevant features  $\mathbf{x}^{\mathrm{B}}$  will never change prediction, as shown by the small blue "+" variants in Figure 2. Moreover, each feature  $\chi$  (and their combinations) in the optimal explanation  $\mathbf{x}^{\mathrm{A}}$  can be perturbed, together with the irrelevant features, to go beyond the decision boundary, i.e., the orange "+" variants. We mention two special cases: (1) if  $\mathbf{x}$  is  $\epsilon$ -robust, then all features are irrelevant, i.e.,  $\mathbf{A} = \emptyset$ , meaning there is no valid explanation as any  $\epsilon$ -perturbation does not affect the prediction at all (in other words, a larger  $\epsilon$  is required to get a meaningful explanation); (2) if perturbing any feature in input  $\mathbf{x}$  can change the prediction, then  $\mathbf{A} = \Theta(\mathbf{x})$ , meaning the entire input is an explanation.
We remark that our definition of optimality is local in that it computes a minimal subset of features. An interesting problem would be to find a globally optimal explanation, i.e., the smallest (fewest features) among all possible local optima, also known as the cardinality-minimal explanation [26]. Approaches (such as those based on minimum hitting sets [33, 4]) for computing such global optima are often too computationally diffi
cult to converge for large models and high-dimensional inputs as in our case. Therefore, we propose a tractable heuristic (Section 3.3) that approximates the ideal and works fairly well in practice.
Figure 2: Graphical illustration of VERIX. Each gray square denotes the original input  $\mathbf{x}$  (big blue  $^{+^{\prime \prime}}$ ) or a variant (smaller  $^{+^{\prime \prime}}$ ). Variants (blue  $^{+^{\prime \prime}}$ ) that do not change the explanation A (green circle) are guaranteed to lie on the same side of the decision boundary. Counterfactuals (orange  $^{+^{\prime \prime}}$ ) with perturbed explanations A' are classified differently.
2.2 Counterfactuals along decision boundary
While there are infinitely many variants in the input space, we are particularly interested in those that lie along the decision boundary of a model. Figure 2 shows several pairs of variants (blue and orange “+”) connected by red dotted lines. Each pair has the property that the blue variant has the same prediction as the original input  $\mathbf{x}$ , whereas the orange variant, obtained by further perturbing one
Table 1: Evolving index set  $\mathbf{A}$  for explanation  $\mathbf{x^A}$  and index set  $\mathbf{B}$  for irrelevant features  $\mathbf{x^B}$  along reasoning result (True / False) of the CHECK sub-procedure when processing Example 3.1.
feature	A	B	B'	reasoner	irrelevant features xB	explanation xA
x1	∅	∅	B ∪ {1}	True	{x1}	-
x2	∅	{1}	B ∪ {2}	True	{x1, x2}	-
x3	∅	{1,2}	B ∪ {3}	True	{x1, x2, x3}	-
x4	∅	{1,2,3}	B ∪ {4}	False	{x1, x2, x3}	{x4}
x5	{4}	{1,2,3}	B ∪ {5}	False	{x1, x2, x3}	{x4, x5}
x6	{4,5}	{1,2,3}	B ∪ {6}	True	{x1, x2, x3, x6}	{x4, x5}
x7	{4,5}	{1,2,3,6}	B ∪ {7}	True	{x1, x2, x3, x6, x7}	{x4, x5}
x8	{4,5}	{1,2,3,6,7}	B ∪ {8}	False	{x1, x2, x3, x6, x7}	{x4, x5, x8}
x9	{4,5,8}	{1,2,3,6,7}	B ∪ {9}	True	{x1, x2, x3, x6, x7, x9}	{x4, x5, x8}
single feature  $\chi$  in the optimal explanation  $\mathbf{x}^{\mathbf{A}}$  (together with the irrelevant features, i.e.,  $\mathbf{x}^{\mathbf{B}'} \oplus \boldsymbol{\chi}'$  in Equation (2), produces a different prediction. We note that these orange variants are essentially counterfactual explanations [51]: each is a concrete example of a nearby point in the input space illustrating one way to change the model prediction. We emphasize that our focus in this paper is to compute explanations of the form in Definition 2.1, but it is noteworthy that these counterfactuals are generated automatically and at no additional cost during the computation. In fact, we end up with a distinct counterfactual for each feature in our explanation, as we will see below (see [20, 50] for a comprehensive review of counterfactual explanations and their uses).
3 Computing VERIX explanations by constraint solving
Before presenting the VERIX algorithm (Algorithm 1) in detail, we first illustrate it via a simple example.
Example 3.1 (VERIX Computation). Suppose  $\mathbf{x}$  is an input with 9 features  $\langle \chi^1,\dots ,\chi^9\rangle$  as in Figure 3, and we have classification network  $f$ , a perturbation magnitude  $\epsilon$ , and are using  $p = \infty$ . The outer loop of the algorithm traverses the input features. For simplicity, assume the order of the traversal is from  $\chi^1$  to  $\chi^9$ . Both the explanation index set  $\mathbf{A}$  and the irrelevant set  $\mathbf{B}$  are initialized to  $\emptyset$ . At each iteration, VERIX decides whether to add the index  $i$  to  $\mathbf{A}$  or  $\mathbf{B}$ . The evolution of the index sets is shown in Table 1. Concretely, when  $i = 1$ , VERIX formulates a pre-condition which specifies that  $\chi^1$  can be perturbed by  $\epsilon$  while the other features remain unchanged. An automated reasoner is then invoked to check whether the pre-condition logically implies the post-condition (in this case,  $f(\mathbf{x}) = f(\hat{\mathbf{x}})$ , meaning the prediction is the same after perturbation). Suppose the reasoner returns True; then, no  $\epsilon$ -perturbation on  $\chi^1$  can alter the prediction. Following Equation (1) of Definition 2.1, we thus add  $\chi^1$  to the irrelevant features  $\mathbf{x}^{\mathrm{B}}$ . Figure 3, top left, shows a visualization of this. VERIX next moves on to  $\chi^2$ . This time the
precondition allows  $\epsilon$ -perturbations on both  $\chi^1$  and  $\chi^2$  while keeping the other features unchanged. The post-condition remains the same. Suppose the reasoner returns True again - we then add  $\chi^2$  to  $\mathbf{x}^{\mathbf{B}}$  (Figure 3, top middle). Following similar steps, we add  $\chi^3$  to  $\mathbf{x}^{\mathbf{B}}$  (Figure 3, top right). When it comes to  $\chi^4$ , we allow  $\epsilon$ -perturbations for  $\langle \chi^1, \chi^2, \chi^3, \chi^4 \rangle$  while the other features are fixed. Suppose this time the reasoner returns False - there exists a counterexample (this counterexample is the counterfactual for feature  $\chi^4$ ) that violates  $f(\mathbf{x}) = f(\hat{\mathbf{x}})$ , i.e., the prediction can be different. Then, according to Equation (2) of Definition 2.1, we add  $\chi^4$  to the optimal explanation  $\mathbf{x}^{\mathbf{A}}$  (shown as green in Figure 3, middle left). The computation continues until all the input features are visited. Eventually, we have  $\mathbf{x}^{\mathbf{A}} = \langle \chi^4, \chi^5, \chi^8 \rangle$  (Figure 3, bottom right), which means that, if the features in the explanation are fixed, the model's prediction is invariant to any possible  $\epsilon$ -perturbation on the other features. Additionally, for each of the features in  $\mathbf{x}^{\mathbf{A}}$ , we have a counterfactual that demonstrates how that feature can be altered (together with irrelevant features) to change the prediction.
Figure 3: Computing a VERIX explanation by constraint solving for a simple input  $\mathbf{x} = \langle \chi^1,\dots ,\chi^9\rangle$  .Green is the optimal robust explanation  $\mathbf{x^{A}}$  gray denotes irrelevant features  $\mathbf{x}^{\mathrm{B}}$
3.1 Building optimal robust explanations and counterfactuals iteratively
We now formally describe our VERiX methodology, which exploits an automated reasoning engine for neural network verification as a black-box sub-procedure. We assume the reasoner takes as inputs a network  $f$  and a specification
$$
\phi_ {i n} (\hat {\mathbf {x}}) \Rightarrow \phi_ {o u t} (\hat {c}) \tag {3}
$$
where  $\hat{\mathbf{x}}$  are variables representing the network inputs and  $\hat{c}$  are expressions representing the network outputs.  $\phi_{in}(\hat{\mathbf{x}})$  and  $\phi_{out}(\hat{c})$  are formulas. We use  $\hat{\chi}^i$  to denote the variable corresponding to the  $i^{th}$  feature. The reasoner checks whether a specification holds on a network.
Inspired by the deletion-based method [8], we propose Algorithm 1, in which the VERIX procedure takes as input a network  $f$  and an input  $\mathbf{x} = \langle \chi^1,\dots ,\chi^d\rangle$  It outputs an optimal explanation  $\mathbf{x}^{\mathrm{A}}$  with respect to perturbation magnitude  $\epsilon$  distance metric  $p$  and discrepancy  $\delta$  .It also outputs a set of counterfactuals, one for each feature in  $\mathbf{x}^{\mathrm{A}}$  . The procedure maintains three sets,A,B,and C, throughout: A comprises feature indices forming the explanation; B includes feature indices that can be excluded from the explanation; and C is the set of counterfactuals. Recall that  $\mathbf{x}^{\mathrm{B}}$  denotes the irrelevant features (i.e.,perturbing  $\mathbf{x}^{\mathrm{B}}$  while leaving  $\mathbf{x}^{\mathrm{A}}$  unchanged never changes the prediction). To start with, these sets are initialized to  $\varnothing$  (Line 2), and the prediction for input  $\mathbf{x}$  is recorded as  $c_{r}$  for which we remark that  $c$  may or may not be an accurate prediction according to the ground truth - VERIX generates an explanation regardless. Overall, the procedure examines every feature
$\chi^i$  in  $\mathbf{x}$  according to TRAVERSALORDER (Line 5) to determine whether  $i$  can be added to  $\mathbf{B}$  or must belong to  $\mathbf{A}$ . The traversal order can significantly affect the size and shape of the explanation. We propose a heuristic for computing a traversal order that aims to produce small explanations in Section 3.3 (in Example 3.1, a sequential order is used for ease of explanation). For each  $i$ , we compute  $\phi$ , a formula that encodes two conditions: (i) the current  $\chi^i$  and  $\mathbf{x}^{\mathbf{B}}$  are allowed to be perturbed by at most  $\epsilon$  (Line 8); and (ii) the rest of the features are fixed (Line 9). The property that we check is that  $\phi$  implies  $|\hat{c} - c| \leq \delta$  (Line 10), denoting prediction invariance.
An automated reasoning sub-procedure CHECK is deployed to examine whether on network  $f$  the specification  $\phi \Rightarrow |\hat{c} - c| \leq \delta$  holds (Line 10), i.e., whether perturbing the current  $\chi^i$  and irrelevant features while fixing the rest ensures a consistent prediction. It returns (True,  $m$ ) if this is the case (where  $m$  is arbitrary) and (False,  $m$ ) if not, where  $m$  is a concrete input falsifying the formula. In practice, this CHECK can be instantiated with an off-the-shelf neural network verification tool [46, 41, 31, 56, 21]. If HOLD is True,  $i$  is added to the irrelevant set B (Line 11). Otherwise,  $i$  is added to the explanation index set A (Line 12), which conceptually indicates that  $\chi^i$  contributes to the explanation of the prediction (since feature indices in B have already been proven to not affect prediction). In other words, an  $\epsilon$ -perturbation that includes the irrelevant features as well as the current  $\chi^i$  can breach the decision boundary of  $f$ . This fact is represented by the counterexample  $m$ , which represents the counterfactual for  $\chi^i$  and is added to the set C of counterfactuals. The procedure continues until all feature indices in  $x$  are traversed and placed into one of the two disjoint sets A and B. At the end,  $x^A$  is returned as the optimal explanation and C is returned as the set of counterfactuals.
Algorithm 1 VERIX (VERIFIED explainability)
Input: neural network  $f$  and input  $\mathbf{x} = \langle \chi^1,\dots ,\chi^d\rangle$
Parameter:  $\epsilon$ -perturbation, norm  $p$ , and discrepancy  $\delta$
Output: optimal robust explanation  $\mathbf{x}^{\mathbf{A}}$ , counterfactuals C for each  $\chi \in \mathbf{x}^{\mathbf{A}}$
1: function VERIX(f, x)
2:  $\mathbf{A},\mathbf{B},\mathbf{C}\mapsto \emptyset ,\emptyset ,\emptyset$
3:  $c \mapsto f(\mathbf{x})$
4:  $\hat{c} \mapsto f(\hat{\mathbf{x}})$
5:  $\pi \mapsto$  TRAVERSALORDER(x)
6: for  $i$  in  $\pi$  do
7:  $\mathbf{B}^{+}\mapsto \mathbf{B}\cup \{i\}$
8:  $\phi \mapsto (\left\| \hat{\chi}^{\mathbf{B}^{+}} - \chi^{\mathbf{B}^{+}}\right\|_{p}\leq \epsilon)$
9:  $\phi \mapsto \phi \wedge (\hat{\chi}^{\Theta \setminus \mathbf{B}^{+}} = \chi^{\Theta \setminus \mathbf{B}^{+}})$
0:  $(\mathrm{HOLD},m)\mapsto \mathrm{CHECK}(f,\phi \Rightarrow |\hat{c} -c|\leq \delta)$
1: if HOLL then  $\mathbf{B} \mapsto \mathbf{B}^+$
2: else  $\mathbf{A} \mapsto \mathbf{A} \cup \{i\}$ ;  $\mathbf{C} \mapsto \mathbf{C} \cup \{m\}$
3: return  $(\mathbf{x}^{\mathrm{A}},\mathbf{C})$
<sup>1</sup>Properties of counterfactuals such as actionability [20] can be addressed by using an appropriate  $\epsilon$  to ensure that the counterexamples returned by verifiers will always be from the input data distribution.
3.2 Soundness and optimality of explanations
To ensure the VERIX procedure returns a robust explanation, we require that CHECK is sound, i.e., the solver returns True only if the specification actually holds. For the robust explanation to be optimal, CHECK also needs to be complete, i.e., the solver always returns True if the specification holds. We can incorporate various existing reasoners as the CHECK sub-routine. We note that an incomplete reasoner (the solver may return Unknown) does not undermine the soundness of our approach, though it does affect optimality (the produced explanations may be larger than necessary).
Lemma 3.2. If the CHECK sub-procedure is sound, then, at the end of each for-loop iteration (Lines 7-12) in Algorithm 1, the irrelevant set of indices  $\mathbf{B}$  satisfies
$$
\left(\left\| \hat {\chi} ^ {\mathbf {B}} - \chi^ {\mathbf {B}} \right\| _ {p} \leq \epsilon\right) \wedge \left(\hat {\chi} ^ {\Theta \backslash \mathbf {B}} = \chi^ {\Theta \backslash \mathbf {B}}\right) \Rightarrow | \hat {c} - c | \leq \delta . \tag {4}
$$
Intuitively, any  $\epsilon$ -perturbation imposed upon all irrelevant features when fixing the others will always keep the prediction consistent, i.e., the infinite number of input variants (indicated with a small blue “+” in Figure 2) will always remain within the decision boundary. We include rigorous proofs for Lemma 3.2 and Theorems 3.3 and 3.4 in Appendix A. Soundness directly follows from Lemma 3.2.