SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anton Xue Rajeev Alur Eric Wong
Department of Computer and Information Science
{antonxue, alur, exwong}@seas.upenn.edu
Explanation methods for machine learning models tend not to provide any formal guarantees and may not reflect the underlying decision-making process. In this work, we analyze stability as a property for reliable feature attribution methods. We prove that relaxed variants of stability are guaranteed if the model is sufficiently Lipschitz with respect to the masking of features. We develop a smoothing method called Multiplicative Smoothing (MuS) to achieve such a model. We show that MuS overcomes the theoretical limitations of standard smoothing techniques and can be integrated with any classifier and feature attribution method. We evaluate MuS on vision and language models with various feature attribution methods, such as LIME and SHAP, and demonstrate that MuS endows feature attributions with non-trivial stability guarantees.
Modern machine learning models are incredibly powerful at challenging prediction tasks but notoriously black-box in their decision-making. One can therefore achieve impressive performance without fully understanding why. In settings like medical diagnosis [1, 2] and legal analysis [3, 4], where accurate and well-justified decisions are important, such power without proof is insufficient. In order to fully wield the power of such models while ensuring reliability and trust, a user needs accurate and insightful explanations of model behavior.
One popular family of explanation methods is feature attributions [5, 6, 7, 8]. Given a model and input, a feature attribution method generates a score for each input feature that denotes its importance to the overall prediction. For instance, consider Figure 1, in which the Vision Transformer [9] classifier predicts the full image (left) as "Goldfish". We then use a feature attribution method like SHAP [7] to score each feature and select the top-25%, for which the masked image (middle) is consistently predicted as "Goldfish". However, adding a single patch of features (right) alters the prediction confidence so much that it now yields "Axolotl". This suggests that the explanation is brittle [10], as small changes easily cause it to induce some other class. In this paper, we study how to overcome such behavior by analyzing the stability of an explanation: we consider an explanation to be stable if, once the explanatory features are included, the addition of more features does not change the prediction.
Stability implies that the selected features are enough to explain the prediction [12, 13, 14] and that this selection maintains strong explanatory power even in the presence of additional information [10, 15]. Similar properties are studied in literature and identified as useful for interpretability [16], and we emphasize that our main focus is on analyzing and achieving provable guarantees. Stability guarantees, in particular, are useful as they allow one to predict how model behavior varies with the explanation. Given a stable explanation, one can include more features, e.g., adding context, while
Goldfish  $(98.72\%)$  Cockroach  $(0.0218\%)$
Goldfish (95.86%) Axolotl (1.62%)
Axolotl  $(55.14\%)$  Goldfish  $(30.24\%)$
Pneumonia  $(25.50\%)$
Pneumonia  $(29.56\%)$
Pneumonia  $(50.05\%)$
Figure 1: (Top) Classification by Vision Transformer [9]. (Bottom) Pneumonia detection from an X-ray image by DenseNet-Res224 from TorchXRayVision [11]. Both are  $224 \times 224$  pixel images whose attributions are derived from SHAP [7] with top-25% selection. A single  $28 \times 28$  pixel patch of difference between the two attributions (marked green) significantly affects prediction confidence.
maintaining confidence in the consistency of the underlying explanatory power. Crucially, we observe that such guarantees only make sense when jointly considering the model and explanation method: the explanation method necessarily depends on the model to yield an explanation, and stability is then evaluated with respect to the model.
Thus far, existing works on feature attributions with formal guarantees face computational tractability and explanatory utility challenges. While some methods take an axiomatic approach [8, 17], others use metrics that appear reasonable but may not reliably reflect useful model behavior, a common and known limitation [18]. Such explanations have been criticized as a plausible guess at best, and completely misleading [19] at worst.
In this paper, we study how to construct explainable models with provable stability guarantees. We jointly consider the classification model and explanation method and present a formalization for studying such properties that we call explainable models. We focus on binary feature attributions [20] wherein each feature is either marked as explanatory (1) or not explanatory (0). We present a method to solve this problem, inspired by techniques from adversarial robustness, particularly randomized smoothing [21, 22]. Our method can take any off-the-shelf classifier and feature attribution method to efficiently yield an explainable model that satisfies provable stability guarantees. In summary, our contributions are as follows:
- We formalize stability as a key property for binary feature attributions and study this in the framework of explainable models. We prove that relaxed variants of stability are guaranteed if the model is sufficiently Lipschitz with respect to the masking of features.
- To achieve the sufficient Lipschitz condition, we develop a smoothing method called Multiplicative Smoothing (MuS). We show that MuS achieves strong smoothness conditions, overcomes key theoretical and practical limitations of standard smoothing techniques, and can be integrated with any classifier and feature attribution method.
- We evaluate MuS on vision and language models along with different feature attribution methods. We demonstrate that MuS-smoothed explainable models achieve strong stability guarantees at a small cost to accuracy.
We observe that formal guarantees for explanations must take into account both the model and explanation method, and for this, we present in Section 2.1 a pairing that we call explainable models. This formulation allows us to describe the desired stability properties in Section 2.2. We show in Section 2.3 that a classifier with sufficient Lipschitz smoothness with respect to feature masking allows us to yield provable guarantees of stability. Finally, in Section 2.4, we show how to adapt existing feature attribution methods into our explainable model framework.
2.1 Explainable Models
We first present explainable models as a formalism for rigorously studying explanations. Let  $\mathcal{X} = \mathbb{R}^n$  be the space of inputs, a classifier  $f: \mathcal{X} \to [0,1]^m$  maps inputs  $x \in \mathcal{X}$  to  $m$  class probabilities that sum to 1, where the class of  $f(x) \in [0,1]^m$  is taken to be the largest coordinate. Similarly, an explanation method  $\varphi: \mathcal{X} \to \{0,1\}^n$  maps an input  $x \in \mathcal{X}$  to an explanation  $\varphi(x) \in \{0,1\}^n$  that indicates which features are considered explanatory for the prediction  $f(x)$ . In particular, we may pick and adapt  $\varphi$  from among a selection of existing feature attribution methods like LIME [6], SHAP [7], and many others [5, 8, 23, 24, 25], wherein  $\varphi$  may be thought of as a top- $k$  feature selector. Note that the selection of input features necessarily depends on the explanation method executing or analyzing the model, and so it makes sense to jointly study the model and explanation method: given a classifier  $f$  and explanation method  $\varphi$ , we call the pairing  $\langle f, \varphi \rangle$  an explainable model. Given some  $x \in \mathcal{X}$ , the explainable model  $\langle f, \varphi \rangle$  maps  $x$  to both a prediction and explanation. We show this in Figure 2, where  $\langle f, \varphi \rangle(x) \in [0,1]^m \times \{0,1\}^n$  pairs the class probabilities and the feature attribution.
Figure 2: An explainable model  $\langle f, \varphi \rangle$  outputs both a classification and a feature attribution. The feature attribution is a binary-valued mask (white 1, black 0) that can be applied over the original input. Here  $f$  is Vision Transformer [9] and  $\varphi$  is SHAP [7] with top-25% feature selection.
For an input  $x \in \mathcal{X}$ , we will evaluate the quality of the binary feature attribution  $\varphi(x)$  through its masking on  $x$ . That is, we will study the behavior of  $f$  on the masked input  $x \odot \varphi(x) \in \mathcal{X}$ , where  $\odot$  is the element-wise vector product. To do this, we define a notion of prediction equivalence: for two  $x, x' \in \mathcal{X}$ , we write  $f(x) \cong f(x')$  to mean that  $f(x)$  and  $f(x')$  yield the same class. This allows us to formalize the intuition that an explanation  $\varphi(x)$  should recover the prediction of  $x$  under  $f$ .
Definition 2.1. The explainable model  $\langle f, \varphi \rangle$  is consistent at  $x$  if  $f(x) \cong f(x \odot \varphi(x))$ .
Evaluating  $f$  on  $x \odot \varphi(x)$  this way lets us apply the model as-is and therefore avoids the challenge of constructing a surrogate model that is accurate to the original [26]. Moreover, this approach is reasonable, especially in domains like vision — where one intuitively expects that a masked image retaining only the important features should induce the intended prediction. Indeed, architectures like Vision Transformer [9] can maintain high accuracy with only a fraction of the image present [27].
Particularly, we would like for  $\langle f,\varphi \rangle$  to generate explanations that are stable and concise (i.e. sparse). The former is our central guarantee and is ensured through smoothing. The latter implies that  $\varphi (x)$  has few ones entries, and is a desirable property since a good explanation should not contain too much redundant information. However, sparsity is a more difficult property to enforce, as this is contingent on the model having high accuracy with respect to heavily masked inputs. For sparsity, we present a simple heuristic in Section 2.4 and evaluate its effectiveness in Section 4.
2.2 Stability Properties of Explainable Models
Given an explainable model  $\langle f, \varphi \rangle$  and some  $x \in \mathcal{X}$ , stability means that the prediction does not change even if one adds more explanatory features to  $\varphi(x)$ . For instance, the model-explanation pair in Figure 1 is not stable, as the inclusion of a single feature group (patch) changes the prediction. To formalize this notion of stability, we first introduce a partial ordering: for  $\alpha, \alpha' \in \{0, 1\}^n$ , we write  $\alpha \succeq \alpha'$  iff  $\alpha_i \geq \alpha_i'$  for all  $i = 1, \ldots, n$ . That is,  $\alpha \succeq \alpha'$  iff  $\alpha$  includes all the features selected by  $\alpha'$ .
Definition 2.2. The explainable model  $\langle f, \varphi \rangle$  is stable at  $x$  if  $f(x \odot \alpha) \cong f(x \odot \varphi(x))$  for all  $\alpha \succeq \varphi(x)$ .
Note that the constant explanation  $\varphi(x) = 1$ , the vector of ones, makes  $\langle f, \varphi \rangle$  trivially stable at every  $x \in \mathcal{X}$ , though this is not a concise explanation. Additionally, stability at  $x$  implies consistency at  $x$ .
Unfortunately, stability is a difficult property to enforce in general, as it requires that  $f$  satisfy a monotone-like behavior with respect to feature inclusion — which is especially challenging for complex models like neural networks. Checking stability without additional assumptions on  $f$  is also hard: if  $k = \| \varphi(x) \|_1$  is the number of ones in  $\varphi(x)$ , then there are  $2^{n-k}$  possible  $\alpha \succeq \varphi(x)$  to check. This large space of possible  $\alpha \succeq \varphi(x)$  motivates us to examine instead relaxations of stability. We introduce lower and upper relaxations of stability below.
Definition 2.3. The explainable model  $\langle f, \varphi \rangle$  is incrementally stable at  $x$  with radius  $r$  if  $f(x \odot \alpha) \cong f(x \odot \varphi(x))$  for all  $\alpha \succeq \varphi(x)$  where  $\| \alpha - \varphi(x) \|_1 \leq r$ .
Incremental stability is the lower relaxation since it considers the case where the mask  $\alpha$  has only a few features more than  $\varphi(x)$ . For instance, if one can probably add up to  $r$  features to a masked  $x \odot \varphi(x)$  without altering the prediction, then  $\langle f, \varphi \rangle$  would be incrementally stable at  $x$  with radius  $r$ . We next introduce the upper relaxation that we call decremental stability.
Definition 2.4. The explainable model  $\langle f, \varphi \rangle$  is decrementally stable at  $x$  with radius  $r$  if  $f(x \odot \alpha) \cong f(x \odot \varphi(x))$  for all  $\alpha \succeq \varphi(x)$  where  $\|1 - \alpha\|_1 \leq r$ .
Decremental stability is a subtractive form of stability, in contrast to the additive nature of incremental stability. Particularly, decremental stability considers the case where  $\alpha$  has much more features than  $\varphi(x)$ . If one can provably remove up to  $r$  non-explanatory features from the full  $x$  without altering the prediction, then  $\langle f, \varphi \rangle$  is decrementally stable at  $x$  with radius  $r$ . Note also that decremental stability necessarily entails consistency of  $\langle f, \varphi \rangle$ , but for simplicity of definitions, we do not enforce this for incremental stability. Furthermore, observe that for a sufficiently large radius of  $r = \lceil (n - \|\varphi(x)\|_1)/2 \rceil$ , incremental and decremental stability together imply stability.
Remark 2.5. Similar notions to the above have been proposed in the literature, and we refer to [16] for an extensive survey. In particular, for [16], consistency is akin to preservation, and stability is similar to continuity, except we are concerned with adding features. In this regard, incremental stability is most similar to incremental addition and decremental stability to incremental deletion.
2.3 Lipschitz Smoothness Entails Stability Guarantees
If  $f: \mathcal{X} \to [0,1]^m$  is Lipschitz with respect to the masking of features, then we can guarantee relaxed stability properties for the explainable model  $\langle f, \varphi \rangle$ . In particular, we require for all  $x \in \mathcal{X}$  that  $f(x \odot \alpha)$  is Lipschitz with respect to the mask  $\alpha \in \{0,1\}^n$ . This then allows us to establish our main result (Theorem 3.3), which we preview below in Remark 2.6.
Remark 2.6 (Sketch of main result). Consider an explainable model  $\langle f, \varphi \rangle$  where for all  $x \in \mathcal{X}$  the function  $g(x, \alpha) = f(x \odot \alpha)$  is  $\lambda$ -Lipschitz in  $\alpha \in \{0,1\}^n$  with respect to the  $\ell^1$  norm. Then at any  $x$ , the radius of incremental stability  $r_{\mathrm{inc}}$  and radius of decremental stability  $r_{\mathrm{dec}}$  are respectively:
$$
r _ {\mathrm {i n c}} = \frac {g _ {A} (x , \varphi (x)) - g _ {B} (x , \varphi (x))}{2 \lambda}, \qquad r _ {\mathrm {d e c}} = \frac {g _ {A} (x , \mathbf {1}) - g _ {B} (x , \mathbf {1})}{2 \lambda},
$$
where  $g_{A} - g_{B}$  is called the confidence gap, with  $g_{A}, g_{B}$  the top-two class probabilities:
$$
k ^ {\star} = \underset {1 \leq k \leq m} {\operatorname {a r g m a x}} g _ {k} (x, \alpha), \quad g _ {A} (x, \alpha) = g _ {k ^ {\star}} (x, \alpha), \quad g _ {B} (x, \alpha) = \underset {i \neq k ^ {\star}} {\max } g _ {i} (x, \alpha). \tag {1}
$$
Observe that Lipschitz smoothness is, in fact, a stronger assumption than necessary, as besides  $\alpha \succeq \varphi(x)$ , it also imposes guarantees on  $\alpha \preceq \varphi(x)$ . Nevertheless, Lipschitz smoothness is one of the few classes of properties that can be guaranteed and analyzed at scale on arbitrary models [22, 28].
2.4 Adapting Existing Feature Attribution Methods
Most existing feature attribution methods assign a real-valued score to feature importance, rather than a binary value. We therefore need to convert this to a binary-valued method for use with a
stable explainable model. Let  $\psi : \mathcal{X} \to \mathbb{R}^n$  be such a continuous-valued method like LIME [6] or SHAP [7], and fix some desired incremental stability radius  $r_{\mathrm{inc}}$  and decremental stability radius  $r_{\mathrm{dec}}$ . Given some  $x \in \mathcal{X}$  a simple construction for binary  $\varphi(x) \in \{0,1\}^n$  is described next.
Remark 2.7 (Iterative construction of  $\varphi(x)$ ). Consider any  $x \in \mathcal{X}$  and let  $\rho$  be an index ordering on  $\psi(x)$  from high-to-low (i.e. most probable class first). Initialize  $\alpha = 0$ , and for each  $i \in \rho$ : assign  $\alpha_i \gets 1$  then check whether  $\langle f, \varphi : x \mapsto \alpha \rangle$  is now consistent, incrementally stable with radius  $r_{\mathrm{inc}}$ , and decrementally stable with radius  $r_{\mathrm{dec}}$ . If so, terminate with  $\varphi(x) = \alpha$ , and continue otherwise.
Note that the above method of constructing  $\varphi(x)$  does not impose sparsity guarantees in the way that we may guarantee stability through Lipschitz smoothness. Instead, the ordering from a continuous-valued  $\psi(x)$  serves as a greedy heuristic for constructing  $\varphi(x)$ . We show in Section 4 that some feature attributions (e.g., SHAP [7]) tend to yield sparser selections on average compared to others (e.g., Vanilla Gradient Saliency [5]).
3 Multiplicative Smoothing for Lipschitz Constants
In this section, we present our main technical contribution of Multiplicative Smoothing (MuS). The goal is to transform an arbitrary base classifier  $h: \mathcal{X} \to [0,1]^m$  into a smoothed classifier  $f: \mathcal{X} \to [0,1]^m$  that is Lipschitz with respect to the masking of features. This then allows one to couple  $f$  with an explanation method  $\varphi$  in order to form an explainable model  $\langle f, \varphi \rangle$  with provable stability guarantees.
We give an overview of our MuS in Section 3.1, where we illustrate that a principal motivation for its development is because standard smoothing techniques may violate a property that we call masking equivalence. We present the Lipschitz constant of the smoothed classifier  $f$  in Section 3.2 and show how this is used to certify stability. Finally, we give an efficient computation of MuS in Section 3.3, allowing us to exactly evaluate  $f$  at a low sample complexity.
Figure 3: Evaluating  $f(x)$  is done in three stages. (Stage 1) Generate  $N$  samples of binary masks  $s^{(1)},\ldots ,s^{(N)}\in \{0,1\} ^n$ , where each coordinate is Bernoulli with parameter  $\lambda$  (here  $\lambda = 1 / 4$ ). (Stage 2) Apply each mask on the input to yield  $x\odot s^{(i)}$  for  $i = 1,\dots ,N$ . (Stage 3) Average over  $h(x\odot s^{(i)})$  to compute  $f(x)$ , and note that the predicted class is given by a weighted average.
3.1 Technical Overview of MuS
Our key insight is that randomly dropping (i.e., zeroing) features will attain the desired smoothness. In particular, we uniformly drop features with probability  $1 - \lambda$  by sampling binary masks  $s \in \{0, 1\}^n$  from some distribution  $\mathcal{D}$  where each coordinate is distributed as  $\operatorname*{Pr}[s_i = 1] = \lambda$ . Then define  $f$  as:
$$
f (x) = \underset {s \sim \mathcal {D}} {\mathbb {E}} h (x \odot s), \quad \text {s u c h t h a t} s _ {i} \sim \mathcal {B} (\lambda) \text {f o r} i = 1, \dots , n \tag {2}
$$
where  $\mathcal{B}(\lambda)$  is the Beronulli distribution with parameter  $\lambda \in [0,1]$ . We give an overview of evaluating  $f(x)$  in Figure 3. Importantly, our main results on smoothness (Theorem 3.2) and stability (Theorem 3.3) hold provided each coordinate of  $\mathcal{D}$  is marginally Bernoulli with parameter  $\lambda$ , and so we avoid fixing a particular choice for now. However, it will be easy to intuit the exposition with  $\mathcal{D} = \mathcal{B}^n(\lambda)$ , the coordinate-wise i.i.d. Bernoulli distribution with parameter  $\lambda$ .
We can equivalently parametrize  $f$  using the mapping  $g(x, \alpha) = f(x \odot \alpha)$ , where it follows that:
$$
g (x, \alpha) = \underset {s \sim \mathcal {D}} {\mathbb {E}} h (x \odot \tilde {\alpha}), \quad \tilde {\alpha} = \alpha \odot s. \tag {3}
$$
Note that one could have alternatively first defined  $g$  and then  $f$  due to the identity  $g(x, \mathbf{1}) = f(x)$ . We require that the relationship between  $f$  and  $g$  follows an identity that we call masking equivalence:
$$
g (x \odot \alpha , \mathbf {1}) = f (x \odot \alpha) = g (x, \alpha), \quad \text {f o r a l l} x \in \mathcal {X} \text {a n d} \alpha \in \{0, 1 \} ^ {n}. \tag {4}
$$
This follows by the definition of  $g$ , and the relevance to stability is this: if masking equivalence holds, then we can rewrite stability properties involving  $f$  in terms of  $g$ 's second parameter as follows:
$$
f (x \odot \alpha) = g (x, \alpha) \cong g (x, \varphi (x)) = f (x \odot \varphi (x)), \quad \text {f o r a l l} \alpha \succeq \varphi (x), \quad (\text {c . f . D e f i n i t i o n 2 . 2})
$$
where incremental and decremental stability may be analogously defined. This translation is useful, as we will prove that  $g$  is  $\lambda$ -Lipschitz in its second parameter (Theorem 3.2), which then allows us to establish the desired stability properties (Theorem 3.3).
Since many choices are valid, we have not given an explicit construction for  $\mathcal{D}$ . Rather, so long as each coordinate of  $s\sim \mathcal{D}$  obeys  $s_i\sim \mathcal{B}(\lambda)$  then the Lipschitz properties for  $g$  follow. The implication here is that although simple distributions like  $\mathcal{B}^n (\lambda)$  suffice for  $\mathcal{D}$ , they may not be sample efficient. We show in Section 3.3 how to exploit a structured statistical dependence in order to reduce the sample complexity of computing MuS.
Importantly, we are motivated to develop MuS because standard smoothing techniques, namely additive smoothing [21, 22], may fail to satisfy masking equivalence. Additive smoothing is by far the most popular smoothing technique and differs from our scheme (3) in how noise is applied, where let  $\mathcal{D}_{\mathrm{add}}$  and  $\mathcal{D}_{\mathrm{mult}}$  be any two distributions on  $\mathbb{R}^n$ :
$$
g (x, \alpha) = \underset {s \sim \mathcal {D}} {\mathbb {E}} h (x \odot \tilde {\alpha}), \qquad \tilde {\alpha} = \left\{ \begin{array}{l l} \alpha + s, & s \sim \mathcal {D} _ {\mathrm {a d d}}, \\ \alpha \odot s, & s \sim \mathcal {D} _ {\mathrm {m u l t}}, \end{array} \right. \quad \text {a d d i t i v e n o i s e}
$$
Particularly, additive smoothing has counterexamples for masking equivalence.
Proposition 3.1. There exists  $h: \mathcal{X} \to [0,1]$  and distribution  $\mathcal{D}$ , where for
$$
g ^ {+} (x, \alpha) = \underset {s \sim \mathcal {D}} {\mathbb {E}} h (x \odot \tilde {\alpha}), \qquad \tilde {\alpha} = \alpha + s,
$$
we have  $g^{+}(x,\alpha)\neq g^{+}(x\odot \alpha ,1)$  for some  $x\in \mathcal{X}$  and  $\alpha \in \{0,1\} ^n$
Proof. Observe that it suffices to have  $h, x, \alpha$  such that  $h(x \odot (\alpha + s)) > h((x \odot \alpha) \odot (1 + s))$  for a non-empty set of  $s \in \mathbb{R}^n$ . Let  $\mathcal{D}$  be a distribution on these  $s$ , then:
$$
g ^ {+} (x, \alpha) = \underset {s \sim \mathcal {D}} {\mathbb {E}} h (x \odot (\alpha + s)) > \underset {s \sim \mathcal {D}} {\mathbb {E}} h ((x \odot \alpha) \odot (\mathbf {1} + s)) = g ^ {+} (x \odot \alpha , \mathbf {1})
$$
Intuitively, this occurs because additive smoothing primarily applies noise by perturbing feature values, rather than completely masking them. As such, there might be "information leakage" when non-explanatory bits of  $\alpha$  are changed into non-zero values. This then causes each sample of  $h(x\odot \tilde{\alpha})$  within  $g(x,\alpha)$  to observe more features of  $x$  than it would have been able to otherwise.
3.2 Certifying Stability Properties with Lipschitz Classifiers