SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 1. The attributions for each method can be calculated by applying a linear operator  $A \in \mathbb{R}^{d \times 2^d}$  to a vector  $v \in \mathbb{R}^{2^d}$  representing the predictions with each feature set, or
$$
\phi (f, x) = A v,
$$
where the linear operator  $A$  for each method is listed in Table 3, and  $v$  is defined as  $v_{S} = f(x_{S})$  for each  $S \subseteq [d]$  based on the chosen feature removal technique.
In this notation, the entries of  $v$  are indexed as  $v_{S}$  for  $S \subseteq [d]$ , and the entries of  $A$  are indexed as  $A_{iS}$  for  $i \in [d]$  and  $S \subseteq [d]$ . The operation  $Av$  can be understood as summing across all subsets, or  $(Av)_i = \sum_{S \subseteq [d]} A_{iS} \cdot v_S$ . Representing the attributions this way is convenient for our next results.
The entries for the linear operator in Table 3 are straightforward for the first four methods, but LIME depends on several implementation choices: these include the choice of weighting kernel, regularization, and intercept term. The first three methods are in fact special cases of LIME [17]. The class of weighting kernel used in practice is difficult to characterize analytically, but its default parameters for tabular, image and text data approach limiting cases where LIME reduces to other methods (Appendix H). For simplicity, the remainder of this section focuses on the other methods.
Perturbing either the input or model induces a change in  $v \in \mathbb{R}^{2^d}$ , and we must consider how the attributions differ between the original vector  $v$  and a perturbed version  $v'$ . Proposition 1 suggests that we can bound the attribution difference via the change in model outputs  $\| v - v' \|$  and properties of the matrix  $A$ . We can even do this under different distance metrics, as we show in the next result.
Lemma 7. The difference in attributions given the same summary technique  $A$  and different model outputs  $v, v'$  can be bounded as
$$
\| A v - A v ^ {\prime} \| _ {2} \leq \| A \| _ {2} \cdot \| v - v ^ {\prime} \| _ {2} \quad \text {o r} \quad \| A v - A v ^ {\prime} \| _ {2} \leq \| A \| _ {1, \infty} \cdot \| v - v ^ {\prime} \| _ {\infty},
$$
where  $\| A \|_2$  is the spectral norm, and the operator norm  $\| A \|_{1,\infty}$  is the square root of the sum of squared row 1-norms, with values for each  $A$  given in Table 3.
For both inequalities, smaller norms  $\| A\|$  represent stronger robustness to perturbations. Neither bound is necessarily tighter, and the choice of which to use depends on how  $v - v^{\prime}$  is bounded. The first bound using the Euclidean distance  $\| v - v^{\prime}\|_{2}$  is perhaps more intuitive, but the second bound is more useful here because the results in Sections 3.1 and 3.2 effectively relate to  $\| v - v^{\prime}\|_{\infty}$ .
This view of the summary technique's robustness raises a natural question, which is whether the approaches used in practice have relatively good or bad robustness (Table 3). The answer is not immediately clear, because within the class of linear operators we can control the robustness arbitrarily: we can achieve  $\| A \|_2 = \| A \|_{1,\infty} = 0$  by setting  $A = 0$  (strong robustness), and we can likewise get  $\| A \|_2$ ,  $\| A \|_{1,\infty} \to \infty$  by setting entries of  $A$  to a large value (weak robustness). These results are not useful, however, because neither limiting case results in meaningful attributions.
Rather than considering robustness within the class of all linear operators  $A \in \mathbb{R}^{d \times 2^d}$ , we therefore restrict our attention to attributions that are in some sense meaningful. There are multiple ways to define this, but we consider solutions that satisfy one or more of the following properties, which are motivated by axioms from the literature on game-theoretic credit allocation [46, 54].
Definition 2. We define the following properties for linear operators  $A \in \mathbb{R}^{d \times 2^d}$  that are used for removal-based explanations:
- (Boundedness) For all  $v \in \mathbb{R}^{2^d}$ , the attributions are bounded by each feature's smallest and largest contributions, or  $\min_{S \not\ni i} (v_{S \cup \{i\}} - v_S) \leq (Av)_i \leq \max_{S \not\ni i} (v_{S \cup \{i\}} - v_S)$  for all  $i \in [d]$ .
- (Symmetry) For all  $v \in \mathbb{R}^{2^d}$ , two features that make equal marginal contributions to all feature sets must have equal attributions, or  $(Av)_i = (Av)_j$  if  $v_{S \cup \{i\}} = v_{S \cup \{j\}}$  for all  $S \subseteq [d]$ .
- (Efficiency) For all  $v \in \mathbb{R}^{2^d}$ , the attributions sum to the difference in predictions for the complete and empty sets, or  $\mathbf{1}^\top Av = v_{[d]} - v_{\{\}}$ .
Among these properties, we prioritize boundedness because this leads to a class of solutions that are well-known in game theory, and which are called probabilistic values [46]. We now show that constraining the summary technique to satisfy different combinations of these properties yields clear bounds for the robustness. We first address the operator norm  $\| A\|_{1,\infty}$ , which is the simpler case.
Lemma 8. When the linear operator  $A$  satisfies the boundedness property, we have  $\| A \|_{1,\infty} = 2\sqrt{d}$ .
The above result applies to several summary techniques shown in Table 3, including leave-one-out [67], Shapley values [45] and Banzhaf values [13]. We now address the case of the spectral norm  $\| A\| _2$ , beginning with the case where boundedness and symmetry are satisfied.
Lemma 9. When the linear operator  $A$  satisfies the boundedness and symmetry properties, the spectral norm is bounded as follows,
$$
\frac {1}{2 ^ {d / 2 - 1}} \leq \| A \| _ {2} \leq \sqrt {d + 1},
$$
with  $1 / 2^{d / 2 - 1}$  achieved by the Banzhaf value and  $\sqrt{d + 1}$  achieved by leave-one-out.
This shows that the Banzhaf value is the most robust summary within a class of linear operators, which are known as semivalues [46]. Its robustness is even better than the Shapley value, a result that was recently discussed by Wang and Jia [64]. However, the Banzhaf value is criticized for failing to satisfy the efficiency property [25], so we now address the case where this property holds.
Lemma 10. When the linear operator  $A$  satisfies the boundedness and efficiency properties, the spectral norm is bounded as follows,
$$
\sqrt {\frac {2}{d}} \leq \| A \| _ {2} \leq \sqrt {2 + 2 \cdot \cos \left(\frac {\pi}{d + 1}\right)},
$$
with  $\sqrt{2 / d}$  achieved by the Shapley value.
We therefore observe that the Shapley value is the most robust choice within a different class of linear operators, which are known as random-order values [46]. In comparison, the worst-case robustness
is achieved by a method that has not been discussed in the literature, but which is a special case of a previously proposed method [26]. The upper bound in Lemma 10 is approximately equal to 2, whereas the upper bound in Lemma 9 grows with the input dimensionality, suggesting that the efficiency property imposes a minimum level of robustness but limits robustness in the best case.
Finally, we can trivially say that  $\| A\| _2 = \sqrt{2 / d}$  when boundedness, symmetry and efficiency all hold, because the Shapley value is the only linear operator to satisfy all three properties [46, 54].
Lemma 11. When the linear operator  $A$  satisfies the boundedness, symmetry and efficiency properties, the spectral norm is  $\| A \|_2 = \sqrt{2 / d}$ .
4 Feature attribution robustness: main results
We now shift our attention to the robustness properties of entire feature attribution algorithms. These results build on those presented in Section 3, combining them to provide general results that apply to existing methods, and to new methods that combine their implementation choices arbitrarily.
Our first main result relates to the robustness to input perturbations.
Theorem 1. The robustness of removal-based explanations to input perturbations is given by the following meta-formula,
$$
\left\| \phi (f, x) - \phi \left(f, x ^ {\prime}\right) \right\| _ {2} \leq g (\text {r e m o v a l}) \cdot h (\text {s u m m a r y}) \cdot \| x - x ^ {\prime} \| _ {2},
$$
where the factors for each method are defined as follows:
$$
g (\text {r e m o v a l}) = \left\{ \begin{array}{l l} L & \text {i f r e m o v a l = b a s e l i n e o r m a r g i n a l} \\ L + 2 B M & \text {i f r e m o v a l = c o n d i t i o n a l}, \end{array} \right.
$$
$$
h (\text {s u m m a r y}) = \left\{ \begin{array}{l l} 2 \sqrt {d} & i f \text {s u m m a r y} = \text {S h a p l e y , B a n z h a f , o r l e a v e - o n e - o u t} \\ \sqrt {d} & i f \text {s u m m a r y} = \text {m e a n w h e n i n c l u d e d .} \end{array} \right.
$$
We therefore observe that the explanation functions themselves are Lipschitz continuous, but with a constant that combines properties of the original model with implementation choices of the attribution method. The model's inherent robustness interacts with the removal choice to yield the  $g$  (removal) factor, and the summary technique is accounted for by  $h$  (summary).
Our second result takes a similar form, but relates to the robustness to model perturbations.
Theorem 2. The robustness of removal-based explanations to model perturbations is given by the following meta-formula,
$$
\left\| \phi (f, x) - \phi \left(f ^ {\prime}, x\right) \right\| _ {2} \leq h (\text {s u m m a r y}) \cdot \left\| f - f ^ {\prime} \right\|,
$$
where the functional distance and factor associated with the summary technique are specified as follows:
$$
\| f - f ^ {\prime} \| = \left\{ \begin{array}{l l} \| f - f ^ {\prime} \| _ {\infty} & i f \text {r e m o v a l} = \text {b a s e l i n e o r m a r g i n a l} \\ \| f - f ^ {\prime} \| _ {\infty , \mathcal {X}} & i f \text {r e m o v a l} = \text {c o n d i t i o n a l}, \end{array} \right.
$$
$$
h (\text {s u m m a r y}) = \left\{ \begin{array}{l l} 2 \sqrt {d} & \text {i f s u m m a r y = S h a p l e y , B a n z h a f , o r l e a v e - o n e - o u t} \\ \sqrt {d} & \text {i f s u m m a r y = m e a n w h e n i n c l u d e d .} \end{array} \right.
$$
Similarly, we see here that the attribution difference is determined by the strength of the model perturbation, as measured by a functional distance metric that depends on the feature removal technique. This represents a form of Lipschitz continuity with respect to the model  $f$ . These results address the case with either an input perturbation or model perturbation, but we can also account for simultaneous input and model perturbations, as shown in Corollary 1 in Appendix E. The remainder of the paper empirically verifies these results and discusses their practical implications.
5 Summary of experiments
Due to space constraints, we defer our experiments to Appendix A but briefly describe the results here. First, we empirically validate our results using synthetic data and a logistic regression classifier, where
$\left|\left|x - x^{\prime}\right|\right|_{2} = 0.5$
(b)
Figure 1: Shapley attribution difference for networks trained with increasing weight decay, under input perturbations with varying perturbation norms. The results include (a) the wine quality dataset with FCNs and baseline, marginal, and conditional feature removal; and (b) MNIST with CNNs and baseline feature removal with either training set means or zeros. Error bars show the mean and  $95\%$  confidence intervals across explicit-and-perturbation pairs.
$\| x - x^{\prime}\|_{2} = 0.5$
$\left\| \left|x - x^{\prime}\right|\right\|_{2} = 1.0$
$\left|\left|x - x^{\prime}\right|\right|_{2} = 2.0$
Figure 2: Sanity checks for attributions using cascading randomization for the FCN trained on the wine quality dataset. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the mean and  $95\%$  confidence intervals across 10 random seeds.
the model's Lipschitz constant and the upper bound for the total variation constant  $M$  can be readily computed. This allows us to verify the robustness bound under input perturbations (Theorem 1). Using the same setup, we construct logistic regression classifiers where the functional distance on the data manifold is small, and we show that feature removal with the conditional distribution is indeed more robust to model perturbations than baseline or marginal removal, as implied by Theorem 2.
Next, we demonstrate two practical implications of our findings with the UCI wine quality [14, 24], MNIST [44], CIFAR-10 [42] and Imagenette datasets [19, 36]. (1) Removal-based attributions are more robust under input perturbations for networks trained with weight decay, which encourages a smaller Lipschitz constant. Formally, this is because a network's Lipschitz constant is upper bounded by the product of its layers' spectral norms [61], which are respectively upper bounded by each layer's Frobenius norm. These results are shown for Shapley value attributions in Figure 1 for the wine quality and MNIST datasets, where we see a consistent effect of weight decay across feature removal approaches. We speculate that a similar effect may be observed with other techniques that encourage
Lipschitz regularity [8, 30, 63]. (2) Viewing the sanity check with parameter randomization proposed by Adebayo et al. [2] as a form of model perturbation, we show that removal-based attributions empirically pass the sanity check, in contrast to certain gradient-based methods [55, 60]. Figure 2 shows results for the wine quality dataset, where the similarity for removal-based methods decays more rapidly than for IntGrad and Grad  $\times$  Input. Results for the remaining datasets are in Appendix A.
Our analysis focused on the robustness properties of removal-based feature attributions to perturbations in both the model  $f$  and input  $x$ . For input perturbations, Theorem 1 shows that the attribution change is proportional to the input perturbation strength. The Lipschitz constant is determined by the attribution's removal and summary technique, as well as the model's inherent robustness. We find that both implementation choices play an important role, but our analysis in Section 3 reveals that the associated factors in our main result can only be so small for meaningful attributions, implying that attribution robustness in some sense reduces to model robustness. Fortunately, there are many existing works on improving a neural network's Lipschitz regularity [8, 30, 63].
For model perturbations, Theorem 2 shows that the attribution difference is proportional to functional distance between the perturbed and original models, as measured by the Chebyshev distance (or infinity norm). Depending on the removal approach, the distance may depend on only a subdomain of the input space. One practical implication is that attributions should remain similar under minimal model changes, e.g., accurate model distillation or quantization. Another implication is that bringing the model closer to its correct form (e.g., the Bayes classifier) results in attributions closer to those of the correct model; several recent works have demonstrated this by exploring ensembled attributions [23, 31, 37, 48], which are in some cases mathematically equivalent to attributions of an ensembled model.
Our robustness results provide a new lens to compare different removal-based attribution methods. In terms of the summary technique, our main proof technique relies on the associated operator norm, which does not distinguish between several approaches (Shapley, leave-one-out, Banzhaf); however, our results for the spectral norm suggest that the Banzhaf value is in some sense most robust. This result echoes recent work [64] and has implications for other uses of game-theoretic credit allocation, but we also find that the Shapley value is most robust within a specific class of summary techniques. Our findings regarding the removal approach are more complex: the conditional distribution approach has been praised by some works for being more informative than baseline or marginal [17, 27], but this comes at a cost of worse robustness to input perturbations. On the other hand, it leads to improved robustness to model perturbations, which has implications for fooling attributions with imperceptible off-manifold model changes [58].
Next, we make connections between current understanding of gradient-based methods and our findings for removal-based methods. Overall, the notion of Lipschitz continuity is important for robustness guarantees under input perturbations. Our results for removal-based attributions rely on Lipschitz continuity of the model itself, whereas gradient-based attributions rely on Lipschitz continuity of the model's gradient (i.e., Lipschitz smoothness) [22]. Practically, both are computationally intensive to determine for realistic networks [30, 63]. Empirically, weight decay is an effective approach for generating robust attributions, for both removal-based methods (our results) and gradient-based methods (see Dombrowski et al. [22]), as the Frobenius norms of a neural network's weights upper bound both the network's Lipschitz constant and Hessian norm. Under model perturbations, removal-based attributions are more robust when features are removed such that  $\boldsymbol{x}_S$  stay on the data manifold, whereas gradient-based attributions are more robust when gradients are projected on the tangent space of the data manifold [21].
Finally, we consider limitations of our analysis. Future work may consider bounding attribution differences via different distance measures (e.g.,  $\ell_p$  norms for  $p \neq 2$ ). The main limitation to our results is that they are conservative. By relying on global properties of the model, i.e., the Lipschitz constant and global prediction bounds, we arrive at worst-case bounds that are in many cases overly conservative, as reflected in our experiments. An alternative approach can focus on localized versions of our bounds. Analogous to recent work on certified robustness [38, 39, 52, 57, 62], tighter bounds may be achievable with specific perturbation strengths, unlike our approach that provides bounds simultaneously for any perturbation strength via global continuity properties.
We thank members of the Lee Lab and Jonathan Hayase for helpful discussions. This work was funded by the National Science Foundation [DBI-1552309 and DBI-1759487]; and the National Institutes of Health [R35 GM 128638 and R01 NIA AG 061132].
[1] Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to Shapley values. Artificial Intelligence, 298: 103502, 2021.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31, 2018.
[3] Chirag Agarwal, Marinka Zitnik, and Himabindu Lakkaraju. Probing gnn explainers: A rigorous theoretical and empirical analysis of gnn explanation methods. In International Conference on Artificial Intelligence and Statistics, pages 8969-8996. PMLR, 2022.
[4] Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning, pages 110-119. PMLR, 2021.
[5] David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049, 2018.
[6] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104, 2017.
[7] Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel. Fairwashing explanations with off-manifold detergent. In International Conference on Machine Learning, pages 314–323. PMLR, 2020.
[8] Alexandre Araujo, Aaron Havens, Blaise Delattre, Alexandre Allauzen, and Bin Hu. A unified algebraic perspective on Lipschitz neural networks. arXiv preprint arXiv:2303.03169, 2023.
[9] John F Banzhaf III. Weighted voting doesn't work: A mathematical analysis. *Rutgers Law Review*, 19:317, 1964.
[10] Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 648-657, 2020.
[11] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[12] Hugh Chen, Ian C Covert, Scott M Lundberg, and Su-In Lee. Algorithms to estimate Shapley value feature attributions. arXiv preprint arXiv:2207.07605, 2022.
[13] Jianbo Chen and Michael Jordan. LS-Tree: Model interpretation when the data are linguistic. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3454–3461, 2020.
[14] Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4):547-553, 2009.
[15] Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33: 17212-17223, 2020.
[53] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[54] Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28): 307-317, 1953.
[55] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153. PMLR, 2017.
[56] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[57] Gagandeep Singh, Timon Gehr, Markus Puschel, and Martin Vechev. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3(POPL): 1-30, 2019.
[58] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: Adversarial attacks on post-hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180-186, 2020.
[59] Mukund Sundararajan and Amir Najmi. The many Shapley values for model explanation. In International Conference on Machine Learning, pages 9269–9278. PMLR, 2020.
[60] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
[61] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[62] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. arXiv preprint arXiv:1711.07356, 2017.
[63] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
[64] Tianhao Wang and Ruoxi Jia. Data Banzhaf: A data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205.15466, 2022.
[65] Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta. Smoothed geometry for robust attribution. Advances in Neural Information Processing Systems, 33:13623-13634, 2020.
[66] Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705.10941, 2017.
[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014.