SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chris Lin*
University of Washington  
clin25@cs.washington.edu
Ian Covert*†
Stanford University icovert@stanford.edu
To explain predictions made by complex machine learning models, many feature attribution methods have been developed that assign importance scores to input features. Some recent work challenges the robustness of these methods by showing that they are sensitive to input and model perturbations, while other work addresses this issue by proposing robust attribution methods. However, previous work on attribution robustness has focused primarily on gradient-based feature attributions, whereas the robustness of removal-based attribution methods is not currently well understood. To bridge this gap, we theoretically characterize the robustness properties of removal-based feature attributions. Specifically, we provide a unified analysis of such methods and derive upper bounds for the difference between intact and perturbed attributions, under settings of both input and model perturbations. Our empirical results on synthetic and real-world data validate our theoretical results and demonstrate their practical implications, including the ability to increase attribution robustness by improving the model's Lipschitz regularity.
In recent years, machine learning has shown great promise in a variety of real-world applications. An obstacle to its widespread deployment is the lack of transparency, an issue that has prompted a wave of research on interpretable or explainable machine learning [45, 51, 53, 56, 60, 67]. One popular way of explaining a machine learning model is feature attribution, which assigns importance scores to input features of the model [45, 51, 53, 56, 60]. Many feature attribution methods can be categorized as either gradient-based methods that compute gradients of model predictions with respect to input features [6], or removal-based methods that remove features to quantify each feature's influence [17]. While substantial progress has been made and current tools are widely used for model debugging and scientific discovery [10, 18, 37, 50], some important concerns remain. Among them is unclear robustness properties: feature attributions are vulnerable to adversarial attacks, and even without an adversary appear unstable under small changes to the input or model.
As a motivating example, consider the work of Ghorbani et al. [28], which shows that minor perturbations to an image can lead to substantially different feature attributions while preserving the original prediction. Similar to adversarial examples designed to alter model predictions [29, 43, 61], this phenomenon violates the intuition that explanations should be invariant to imperceptible changes. A natural question, and the focus of this work, is therefore: When can we guarantee that feature
*Equal contribution. †Work done while at the University of Washington.
attributes are robust to small changes in the input or small changes in the model? In answering this question, we find that previous theoretical work on this topic has primarily focused on gradient-based methods. Hence, in this work we provide a unified analysis to characterize the robustness of removal-based feature attributions, under both input and model perturbations, and considering a range of existing methods in combination with different approaches to implement feature removal.
Related work. Recent work has demonstrated that gradient-based attributions for neural networks are susceptible to small input perturbations that can lead to markedly different attributions [21, 28, 41]. Theoretical analyses have established that this issue relates to model smoothness, and approaches like stochastic smoothing of gradients, weight decay regularization, smoothing activation functions, and Hessian minimization have been shown to generate more robust gradient-based attributions [4, 21, 22, 65]. As for the robustness of removal-based attributions under input perturbations, Alvarez-Melis and Jaakkola [5] empirically assess the robustness of LIME, SHAP, and Occlusion with the notion of Lipschitz continuity in local neighborhoods. Khan et al. [40] provide theoretical guarantees on the robustness of SHAP, RISE, and leave-one-out attributions using Lipschitz continuity, in the specific setting where held-out features are replaced with zeros. Agarwal et al. [3] study the robustness of discrete attributions for graph neural networks and establish Lipschitz upper bounds for the robustness of GraphLIME and GraphMASK. Our analysis of input perturbations is most similar to Khan et al. [40], but we generalize this work by considering alternative techniques for both feature removal and importance summarization (e.g., LIME's weighted least squares approach [53]).
Other works have considered model perturbations and manipulated neural networks to produce targeted, arbitrary gradient-based attributions [7, 34]. Anders et al. [7] explain this phenomenon through the insight that neural network gradients are underdetermined with many degrees of freedom to exploit, and they demonstrate better robustness when gradient-based attributions are projected onto the tangent space of the data manifold. As for removal-based attributions, it has been demonstrated that models can be modified to hide their reliance on sensitive features in LIME and SHAP [20, 58]; however, Frye et al. [27] show that such hidden features can be discovered if features are removed using their conditional distribution, suggesting a key role for the feature removal approach in determining sensitivity to such attacks. Our work formalizes these results through the lens of robustness under model perturbations, and provides new theoretical guarantees.
Contribution. The main contribution of this work is to develop a comprehensive characterization of the robustness properties of removal-based feature attributions, focusing on two notions of robustness that have received attention in the literature: stability under input changes and stability under model changes. Our specific contributions are the following: (1) We provide theoretical guarantees for the robustness of model predictions with the removal of arbitrary feature subsets, under both input and model perturbations. (2) We analyze the robustness of techniques for summarizing each feature's influence (e.g., Shapley values) and derive best- and worst-case robustness within several classes of such techniques. (3) We combine the analyses above to prove unified attribution robustness properties to both input and model perturbations, where changes in removal-based attributions are controlled by the scale of perturbations in either input space or function space. (4) We validate our theoretical results and demonstrate practical implications using synthetic and real-world datasets.
Here, we introduce the notation used in the paper and review removal-based feature attributions.
Let  $f: \mathbb{R}^d \mapsto \mathbb{R}$  be a model whose predictions we seek to explain. We consider that the input variable  $\pmb{x}$  consists of  $d$  separate features, or  $\pmb{x} = (x_1, \dots, x_d)$ . Given an index set  $S \subseteq [d] \equiv \{1, \dots, d\}$ , we define the corresponding feature subset as  $\pmb{x}_S \equiv \{\pmb{x}_i : i \in S\}$ . The power set of  $[d]$  is denoted by  $\mathcal{P}(d)$ , and we let  $\bar{S} \equiv [d] \setminus S$  denote the set complement. We consider the model's output space to be one-dimensional, which is not restrictive because attributions are typically calculated for scalar predictions (e.g., the probability for a given class). We use the bold symbols  $\pmb{x}$ ,  $\pmb{x}_S$  to denote random variables, the symbols  $x, x_S$  to denote specific values, and  $p(\pmb{x})$  to represent the data distribution with support on  $\mathcal{X} \subseteq \mathbb{R}^d$ . We assume that all explanations are generated for inputs such that  $x \in \mathcal{X}$ .
2.2 Removal-based feature attributions
Our work focuses on a class of methods known as removal-based explanations [17]. Intuitively, these are algorithms that remove subsets of inputs and summarize how each feature affects the model. This framework describes a large number of methods that are distinguished by two main implementation choices: (1) how feature information is removed from the model, and (2) how the algorithm summarizes each feature's influence. This perspective shows a close connection between methods like leave-one-out [67], LIME [53] and Shapley values [45], and enables a unified analysis of their robustness properties. Below, we describe the two main implementation choices in detail.
Feature removal. Most machine learning models require all feature values to generate predictions, so we must specify a convention for depriving the model of feature information. We denote the prediction given partial inputs by  $f(x_{S})$ . Many implementations have been discussed in the literature [17], but we focus here on three common choices. Given a set of observed values  $x_{S}$ , these techniques can all be viewed as averaging the prediction over a distribution  $q(\boldsymbol{x}_{\bar{S}})$  for the held-out feature values:
$$
f (x _ {S}) := \mathbb {E} _ {q (\boldsymbol {x} _ {\bar {S}})} [ f (x _ {S}, \boldsymbol {x} _ {\bar {S}}) ] = \int f (x _ {S}, x _ {\bar {S}}) q (x _ {\bar {S}}) d x _ {\bar {S}}. \tag {1}
$$
Specifically, the three choices we consider are:
- (Baseline values) Given a baseline input  $b \in \mathbb{R}^d$ , we can set the held-out features to their corresponding values  $b_{\bar{S}}$  [59]. This is equivalent to letting  $q(\pmb{x}_{\bar{S}})$  be a Dirac delta centered at  $b_{\bar{S}}$ .
- (Marginal distribution) Rather than using a single replacement value, we can average across values sampled from the input's marginal distribution, or let  $q(\pmb{x}_{\bar{S}}) = p(\pmb{x}_{\bar{S}})$  [45].
- (Conditional distribution) Finally, we can average across replacement values while conditioning on the available features, which is equivalent to setting  $q(\pmb{x}_{\bar{S}}) = p(\pmb{x}_{\bar{S}} \mid x_{S})$  [27].
The first two options are commonly implemented in practice because they are simple to estimate, but the third choice is viewed by some work as more informative to users [1, 17, 27]. The conditional distribution approach requires more complex and error-prone estimation procedures (see [12] for a discussion), but we assume here that all three versions of  $f(x_{S})$  can be calculated exactly.
Summary technique. Given a feature removal technique that allows us to query the model with arbitrary feature sets, we must define a convention for summarizing each feature's influence. This is challenging due to the exponential number of feature sets, or because  $|\mathcal{P}(d)| = 2^d$ . Again, many techniques have been discussed in the literature [17], with the simplest option comparing the prediction with all features included and with a single feature missing [67]. We take a broad perspective here, considering a range of approaches that define attributions as a linear combination of predictions with different feature sets. These methods include leave-one-out [67], RISE [51], LIME [53], Shapley values [45], and Banzhaf values [13], among other possible options.
All of these methods yield per-feature attribution scores  $\phi_i(f,x)\in \mathbb{R}$  for  $i\in [d]$ . For example, the Shapley value calculates feature attribution scores as follows [45]:
$$
\phi_ {i} (f, x) = \frac {1}{d} \sum_ {S \subseteq [ d ] \backslash \{i \}} \binom {d - 1} {| S |} ^ {- 1} \left(f \left(x _ {S \cup \{i \}}\right) - f \left(x _ {S}\right)\right). \tag {2}
$$
The specific linear combinations for the other approaches we consider are shown in Table 3. The remainder of the paper uses the notation  $f(x_{S})$  to refer to predictions with partial information, and  $\phi (f,x) = [\phi_1(f,x),\ldots ,\phi_d(f,x)]\in \mathbb{R}^d$  to refer to feature attributions. We do not introduce separate notation to distinguish between implementation choices, opting instead to make the relevant choices clear in each result.
2.3 Problem formulation
The problem formulation in this work is straightforward: our goal is to understand the stability of feature attributions under input perturbations and model perturbations. Formally, we aim to study
2The framework also considers the choice to explain different model behaviors (e.g., the loss calculated over one or more examples) [17], but we focus on methods that explain individual predictions.
1. whether  $\| \phi (f,x) - \phi (f,x^{\prime})\|$  is controlled by  $\| x - x^{\prime}\|$  (input perturbation), and
2. whether  $\| \phi (f,x) - \phi (f^{\prime},x)\|$  is controlled by  $\| f - f^{\prime}\|$  (model perturbation).
The following sections show how this can be guaranteed using certain distance metrics, and under certain assumptions about the model and/or the data distribution.
3 Preliminary results
As an intermediate step towards understanding explanation robustness, we first provide results for sub-components of the algorithms. We begin by addressing the robustness of predictions with partial information to input and model perturbations, and we then discuss robustness properties of the summary technique. Proofs for all results are in the Appendix.
Before proceeding, we introduce two assumptions that are necessary for our analysis.
Assumption 1. We assume that the model  $f$  is globally L-Lipschitz continuous, or that we have  $|f(x) - f(x')| \leq L \cdot \| x - x' \|_2$  for all  $x, x' \in \mathbb{R}^d$ .
Assumption 2. We assume that the model  $f$  has bounded predictions, or that there exists a constant  $B$  such that  $|f(x)| \leq B$  for all  $x \in \mathbb{R}^d$ .
The first assumption holds for most deep learning architectures, because they typically compose a series of Lipschitz continuous layers [63]. The second assumption holds with  $B = 1$  for classification models. These are therefore mild assumptions, but they are discussed further in Section 6.
We also define the notion of a functional norm, which is useful for several of our results.
Definition 1. The  $L^p$  norm for a function  $g: \mathbb{R}^d \mapsto \mathbb{R}$  is defined as  $\| g \|_p \equiv (\int |g(x)|^p dx)^{1/p}$ , where the integral is taken over  $\mathbb{R}^d$ .  $\| g \|_{p, \mathcal{X}'}$  denotes the same integral taken over the domain  $\mathcal{X}' \subseteq \mathbb{R}^d$ .
Our results adopt this to define a notion of distance between functions  $\| g - g' \|_p$ , focusing on the cases where  $p = 1$  for the  $L^1$  distance and  $p = \infty$  for the Chebyshev distance.
3.1 Prediction robustness to input perturbations
Our goal here is to understand how the prediction function  $f(x_{S})$  for a feature set  $x_{S}$  behaves under small input perturbations. We first consider the case where features are removed using the baseline or marginal approach, and we find that Lipschitz continuity is inherited from the original model.
Lemma 1. When removing features using either the baseline or marginal approaches, the prediction function  $f(x_{S})$  for any feature set  $x_{S}$  is  $L$ -Lipschitz continuous:
$$
| f (x _ {S}) - f \left(x _ {S} ^ {\prime}\right) | \leq L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} \quad \forall x _ {S}, x _ {S} ^ {\prime} \in \mathbb {R} ^ {| S |}.
$$
Next, we consider the case where features are removed using the conditional distribution approach. We show that the continuity of  $f(x_{S})$  depends not only on the original model, but also on the similarity of the conditional distribution for the held-out features.
Lemma 2. When removing features using the conditional approach, the prediction function  $f(x_{S})$  for a feature set  $x_{S}$  satisfies
$$
\left| f \left(x _ {S}\right) - f \left(x _ {S} ^ {\prime}\right) \right| \leq L \cdot \left\| x _ {S} - x _ {S} ^ {\prime} \right\| _ {2} + 2 B \cdot d _ {T V} \left(p \left(\boldsymbol {x} _ {\bar {S}} \mid x _ {S}\right), p \left(\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}\right)\right),
$$
where the total variation distance is defined via the  $L^1$  functional distance as
$$
d _ {T V} \left(p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S}), p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime})\right) \equiv \frac {1}{2} \left\| p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S}) - p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}) \right\| _ {1}.
$$
Lemma 2 does not immediately imply Lipschitz continuity for  $f(x_{S})$ , because we have not bounded the total variation distance in terms of  $\| x_S - x_S'\| _2$ . To address this, we require the following Lipschitz-like continuity property in the total variation distance.
Assumption 3. We assume that there exists a constant  $M$  such that for all  $S \subseteq [d]$ , we have
$$
d _ {T V} \left(p \left(\boldsymbol {x} _ {\bar {S}} \mid x _ {S}\right), p \left(\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}\right)\right) \leq M \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} \quad \forall x _ {S}, x _ {S} ^ {\prime} \in \mathbb {R} ^ {| S |}.
$$
Intuitively, this says that the conditional distribution  $p(\pmb{x}_{\bar{S}} \mid x_{S})$  cannot change too quickly as a function of the observed features. This property does not hold for all data distributions  $p(\pmb{x})$ , and it may hold in some cases only for large  $M$ ; in such scenarios, the function  $f(x_{S})$  is not guaranteed to change slowly. However, there are cases where it holds. For example, we have  $M = 0$  if the features are independent, and the following example shows a case where it holds with dependent features.
Example 1. For a Gaussian random variable  $\pmb{x} \sim \mathcal{N}(\mu, \Sigma)$  with mean  $\mu \in \mathbb{R}^d$  and covariance  $\Sigma \in \mathbb{R}^{d \times d}$ , Assumption 3 holds with  $M = \frac{1}{2}\sqrt{\lambda_{\max}(\Sigma^{-1}) - \lambda_{\min}(\Sigma^{-1})}$ . If  $\pmb{x}$  is assumed to be standardized, this captures the case where independent features yield  $M = 0$ . Intuitively, it also means that if one dimension is roughly a linear combination of the others, or if  $\lambda_{\max}(\Sigma^{-1})$  is large, the conditional distribution can change quickly as a function of the conditioning variables.
Now, assuming that this property holds for  $p(x)$ , we show that we can improve upon Lemma 2.
Lemma 3. Under Assumption 3, the prediction function  $f(x_{S})$  defined using the conditional approach is Lipschitz continuous with constant  $L + 2BM$  for any feature set  $x_{S}$ .
Between Lemmas 1 and 3, we have established that the function  $f(x_{S})$  remains Lipschitz continuous for any feature set  $x_{S}$ , although in some cases with a larger constant that depends on the data distribution. These results are summarized in Table 1. In Appendix B we show that our analysis can be extended to account for sampling in eq. (1) when the expectation is not calculated exactly.
Table 1: Lipschitz constants induced by each feature removal technique.
Baseline	Marginal	Conditional
L	L	L + 2BM
Table 2: Relevant functional distances for each feature removal technique.
Baseline	Marginal	Conditional
||f-f'||∞	||f-f'||∞	||f-f'||∞,x
3.2 Prediction robustness to model perturbations
Our next goal is to understand how the function  $f(x_{S})$  for a feature set  $x_{S}$  behaves under small changes to the model. The intuition is that if two models make very similar predictions with all features, they should continue to make similar predictions with a subset of features. We first derive a general result involving the proximity between two models  $f$  and  $f'$  within a subdomain.
Lemma 4. For two models  $f, f': \mathbb{R}^d \mapsto \mathbb{R}$  and a subdomain  $\mathcal{X}' \subseteq \mathbb{R}^d$ , the prediction functions  $f(x_S), f'(x_S)$  for any feature set  $x_S$  satisfy
$$
\left| f \left(x _ {S}\right) - f ^ {\prime} \left(x _ {S}\right) \right| \leq \| f - f ^ {\prime} \| _ {\infty , \mathcal {X} ^ {\prime}} \cdot Q _ {x _ {S}} \left(\mathcal {X} ^ {\prime}\right) + 2 B \cdot \left(1 - Q _ {x _ {S}} \left(\mathcal {X} ^ {\prime}\right)\right),
$$
where  $Q_{x_S}(\mathcal{X}')$  is the probability of imputed samples lying in  $\mathcal{X}'$  based on the distribution  $q(\boldsymbol{x}_S)$ :
$$
Q _ {x _ {S}} \left(\mathcal {X} ^ {\prime}\right) \equiv \mathbb {E} _ {q \left(\boldsymbol {x} _ {\bar {S}}\right)} \left[ \mathbb {I} \left\{\left(x _ {S}, \boldsymbol {x} _ {\bar {S}}\right) \in \mathcal {X} ^ {\prime} \right\} \right].
$$
The difference in predictions therefore depends on the distance between the models only within the subdomain  $\mathcal{X}'$ , as well as the likelihood of imputed sampled lying in this subdomain. This result illustrates a point shown by Slack et al. [58]: that two models which are equivalent on a small subdomain, or even on the entire data manifold  $\mathcal{X}$ , can lead to different attributions if we use a removal technique that yields a low value for  $Q_{x_S}(\mathcal{X}')$ . In fact, when  $Q_{x_S}(\mathcal{X}') \to 0$ , Lemma 4 reduces to a trivial bound  $|f(x_S) - f'(x_S)| \leq 2B$  that follows directly from Assumption 2.
The general result in Lemma 4 allows us to show two simpler ones. In both cases, we choose the subdomain  $\mathcal{X}'$  and removal technique  $q(\boldsymbol{x}_{\bar{S}})$  so that  $Q_{x_S}(\mathcal{X}') = 1$ .
Lemma 5. When removing features using the conditional approach, the prediction functions  $f(x_{S}), f'(x_{S})$  for two models  $f, f'$  and any feature set  $x_{S}$  satisfy
$$
\left| f \left(x _ {S}\right) - f ^ {\prime} \left(x _ {S}\right) \right| \leq \| f - f ^ {\prime} \| _ {\infty , \mathcal {X}}.
$$
The next result is similar but involves a potentially larger upper bound  $\| f - f' \|_{\infty} \geq \| f - f' \|_{\infty, \mathcal{X}}$ .
Lemma 6. When removing features using the baseline or marginal approach, the prediction functions  $f(x_{S}), f'(x_{S})$  for two models  $f, f'$  and any feature set  $x_{S}$  satisfy
$$
\left| f \left(x _ {S}\right) - f ^ {\prime} \left(x _ {S}\right) \right| \leq \left\| f - f ^ {\prime} \right\| _ {\infty}.
$$
Table 3: Summary technique linear operators used by various removal-based explanations.
Summary	Method	AiS (i ∈ S)	AiS (i∉ S)	||A||1,∞	||A||2
Leave-one-out	Occlusion [67]	II{|S|=d}	-III{|S|=d-1}	2√d	√d+1
Shapley value	SHAP [45]	(d-1)d!	-(S!(d-S|-1)!d!	2√d	√2/d
Banzhaf value	Banzhaf [13]	1/2d-1	-1/2d-1	2√d	1/2d/2-1
Mean when included	RISE [51]	1/2d-1	0	√d	√(d+1)/2d
Weighted least squares	LIME [53]	Depends on implementation choices (see Appendix D)
The relevant functional distances for Lemma 5 and Lemma 6 are summarized in Table 2.
Remark 1. Lemma 6 implies that if two models  $f, f'$  are functionally equivalent, or  $f(x) = f(x')$  for all  $x \in \mathbb{R}^d$ , they remain equivalent with any feature subset  $x_S$ . In Section 4, we will see that this implies equal attributions for the two models—a natural property that, perhaps surprisingly, is not satisfied by all feature attribution methods [47, 55], and that has been described in the literature as implementation invariance [60]. This property holds automatically for all removal-based methods.
Remark 2. In contrast, Lemma 5 implies the same for models that are equivalent only on the data manifold  $\mathcal{X} \subseteq \mathbb{R}^d$ . This is a less stringent requirement to guarantee equal attributions, and it is perhaps reasonable that models with equal predictions for all realistic inputs receive equal attributions. To emphasize this, we propose distinguishing between a notion of weak implementation invariance and strong implementation invariance, depending on whether equal attributions are guaranteed when we have  $f(x) = f'(x)$  everywhere ( $x \in \mathbb{R}^d$ ) or almost everywhere ( $x \in \mathcal{X}$ ).
3.3 Summary technique robustness
We previously focused on the effects of the feature removal choice, so our goal is now to understand the role of the summary technique. Given a removal approach that lets us query the model with any feature set, the summary generates attribution scores  $\phi(f, x) \in \mathbb{R}^d$ , often using a linear combination of the outputs  $f(x_S)$  for each  $S \subseteq [d]$ . We formalize this in the following proposition.