SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This section conducts a range of experiments to provide empirical validation for our theoretical results, and to examine their practical implications for common machine learning models. Code is available at https://github.com/suinleelab/removal-robustness.
A.1 Synthetic data with input perturbation
First, we empirically validate our results from Section 4 using synthetic data and a logistic regression classifier, where the model's Lipschitz constant  $L$  and the data distribution's total variation constant  $M$  can be readily computed for the bound in Theorem 1.
Setup. For each input  $x \in \mathbb{R}^4$  in the synthetic dataset, features are generated from a multivariate Gaussian distribution with zero mean,  $\pmb{x}_3, \pmb{x}_4$  independent from each other and from  $\pmb{x}_1, \pmb{x}_2$ , and with a correlation  $\rho$  between  $\pmb{x}_1, \pmb{x}_2$ . That is, the covariance matrix  $\Sigma \in \mathbb{R}^{4 \times 4}$  has the following form:
$$
\Sigma = \left( \begin{array}{c c c c} 1 & \rho & 0 & 0 \\ \rho & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array} \right).
$$
The binary label  $\pmb{y} \in \{0,1\}$  for each input is sampled from a Bernoulli distribution with probability  $\sigma(\beta^{\top} \pmb{x})$ , where  $\sigma(\cdot)$  is the sigmoid function and  $\beta^{\top} = [5,0,3,1]$ . Unless stated otherwise, we consider  $\rho = 1$ , zeros are used for baseline feature removal, and the exact Gaussian distributions are used for marginal and conditional feature removal with 20,000 samples.
A logistic regression classifier was trained with 500 samples, and removal-based feature attributions were computed for 50 test samples (i.e., explicands) with and without input perturbation. In our experiment, perturbations are randomly sampled from standard Gaussian distributions with  $\ell_2$  norms normalized to values ranging between 0 to 2 with a total of 50 different values. For each perturbation norm, 50 perturbations are generated and applied to each explicand, resulting in 2,500 total explicand-perturbation pairs. As illustrated in Figure 3, a line is drawn to map the maximum  $\ell_2$  difference between original and perturbed attributions at each perturbation norm. The slope of such a line estimates the dependency of the upper bound of attribution differences with respect to the perturbation norm. Theorem 1 guarantees a specific slope, and this experiment aims to verify this property.
Furthermore, note that the sigmoid function  $\sigma(\cdot)$  is  $1/4$ -Lipschitz because its largest derivative is  $1/4$ . A logistic regression classifier is a composition of a linear function and sigmoid function  $\sigma(\cdot)$ , so the model is  $L$ -Lipschitz with  $L = \|\hat{\beta}\|_2/4$ , where  $\hat{\beta}$  is the trained model coefficients. Finally, because the input features follow a Gaussian distribution, the total variation constant  $M$  derived in Example 1 applies here. Hence, we have specific values for all terms in Theorem 1 bound, which allows a direct comparison between theoretical and empirical results.
Figure 3: Shapley attribution differences under input perturbation with various perturbation norms in our synthetic data experiment. Lines bounding the maximum attribution difference at each perturbation norm are shown as empirical upper bounds. Theoretical upper bounds are included for baseline and marginal feature removal and omitted for conditional feature removal because it is infinite for  $\rho = 1$ .
Results. As shown in Figure 3, for all three feature removal approaches, the theoretical upper bounds for the Shapley value are indeed greater than the corresponding empirical upper bounds. We also observe that the empirical upper bound with conditional feature removal exceeds those with baseline and marginal feature removal, as expected due to our results involving the conditional distribution's total variation distance (Lemma 2).
Moreover, we vary the Lipschitz constant  $L$  of the trained logistic regression classifier by a factor of  $\alpha = \{0.05, 0.1, 0.5, 1, 2\}$ , which is done by multiplying the trained coefficients  $\hat{\beta}$  by  $\alpha$ . Figure 4 shows that the perturbation norm-dependent slope of the empirical bound increases linearly with  $\alpha$ , aligning with our theoretical results. Also, Example 1 suggests that increasing  $\rho$  would lead to a larger upper bound for conditional feature removal, which is observed in the empirical bound with  $\rho = \{0, 0.25, 0.5, 0.75, 1\}$  (see Figure 4 right).
Overall, these empirical results validate our theoretical bound from Theorem 1. We note that gaps between our theoretical and empirical bounds exist, likely because we use the global Lipschitz constant to provide guarantee under all input perturbation strengths, instead of local constants for specific input regions and perturbation strengths, and also due to looseness in the total variation constant  $M$ . For example, our bound for multivariate Gaussian distributions in Example 1 suggests that  $M = \infty$  when  $\rho = 1$ , which is far more conservative than our empirical results (see Figure 3 right); this is due to looseness in Pinsker's inequality (Appendix B). Similar results are observed for Occlusion, Banzhaf, RISE, and LIME attributions, and these are included in Appendix I.
Figure 4: Slopes of empirical and theoretical upper bounds with respect to the input perturbation norm, for Shapley value attribution difference in the synthetic experiment. The parameter  $\alpha$  varies the logistic regression Lipschitz constant, and  $\rho$  varies the correlation between the two synthetic features  $x_{1}, x_{2}$ .
A.2 Synthetic data with model perturbation
Here, we construct a logistic regression classifier as well as a perturbed version such that their functional distance on the data manifold is small. We use this construction to empirically verify that removing features with the conditional distribution leads to attributions that are more robust against model perturbation, as suggested by Theorem 2.
Setup. The synthetic data generation follows that of Appendix A.1. A logistic regression classifier  $f$  is constructed with coefficients [5, 0, 3, 1], and its perturbed counterpart  $f'$  has coefficients [0, 5, 3, 1]. Because the synthetic features  $x_{1}, x_{2}$  are identical with the default value  $\rho = 1$ , we have the functional distance  $\| f - f' \|_{\infty, \mathcal{X}} = 0$ , so our theoretical bound suggests that  $\| \phi(f, x) - \phi(f', x) \|_2 = 0$  for conditional feature removal. When we consider smaller values for  $\rho$ , we still expect better robustness for conditional rather than marginal removal due to Lemma 4, because significant probability mass lies in a subdomain where  $f$  and  $f'$  are functionally close (near the hyperplane where  $x_{1} = x_{2}$ ). Feature attributions are computed for 100 explicands with the same feature removal settings as in Appendix A.1.
Results. As suggested by our theoretical bound for model perturbation when  $\rho = 1$ , the Shapley attribution difference between  $f, f'$  is near zero when features are removed using their conditional distribution, whereas the attribution difference can be large with baseline and marginal feature removal (Figure 5). When we decrease  $\rho$  in the synthetic data generation, the models  $f, f'$  become
Figure 5: Shapley value attribution differences between the logistic regression classifiers  $f, f'$  as constructed in Appendix A.2 with baseline, marginal, and conditional feature removal, and varying correlation  $\rho$ .
more likely to generate different outputs, and we observe that the robustness gap between conditional vs. baseline and marginal feature removal reduces. These empirical results corroborate Lemma 4. Similar results are observed for Occlusion, Banzhaf, RISE, and LIME attributions and are included in Appendix I. It is worth noting that the arbitrarily large bound for baseline and marginal feature removal underpins the adversarial attacks on LIME and SHAP demonstrated by Slack et al. [58]. Our theory and synthetic experiments for model perturbation are similar to those in Frye et al. [27], but we provide a unified analysis for removal-based attribution methods instead of focusing on the Shapley value, and we consider cases where models disagree even on the data manifold (Lemma 4).
A.3 Implication I: robust attributions for regularized neural networks
Our theoretical upper bound for input perturbation decreases with the model's Lipschitz constant (Theorem 1), suggesting that training a model with better Lipschitz regularity can help obtain more robust removal-based feature attributions. Weight decay is one way to achieve this, as it encourages lower Frobenius norm for the individual layers in a neural network, and the product of these Frobenius norms upper bounds the network's Lipschitz constant [66]. We therefore examine whether neural networks trained with weight decay produce attributions that are more robust against input perturbation.
Datasets. The UCI white wine quality dataset [14, 24] consists of 4,898 samples of white wine, each with 11 input features for predicting the wine quality (whether the rating  $>6$ ). Two subsets of 500 samples are randomly chosen as the validation and test set. For datasets with larger input dimension, we use MNIST [44], CIFAR-10 [42], and ten classes of ImageNet [19] (i.e., Imagenette<sup>3</sup>). From the official training set of MNIST and CIFAR-10, 10,000 images are randomly chosen as the validation set.
Setup. Fully connected networks (FCNs) and convolutional networks (CNNs) are trained with a range of weight decay values for the wine quality dataset and MNIST, respectively, where we use the values  $\{0, 0.001, 0.0025, 0.005, 0.01\}$ . ResNet-18 networks [33] are trained for CIFAR-10, with weight decay values  $\{0.0075, 0.01, 0.025, 0.05, 0.075\}$ . ResNet-50 networks pre-trained on ImageNet are fine-tuned for Imagenette, with weight decay values  $\{0.005, 0.01, 0.025, 0.05\}$ . The test set accuracy does not differ much with these weight decay values for the wine quality dataset, MNIST, and CIFAR-10 (Figure 20 and Figure 21), but large weight decay values can worsen the performance of ResNet-50 on Imagenette (Figure 21). Details on model architectures and other hyperparameters are in Appendix J. Removal-based attributions are computed for the 500 test samples and all features in the wine quality dataset, and for 100 test images with non-overlapping patches of size  $4 \times 4$  as features in MNIST and CIFAR-10, and of size  $16 \times 16$  as features in Imagenette (see Appendix G for a discussion of how our theory extends to the use of feature groups). For the wine quality dataset, removal-based attributions are computed exactly with all  $2^{11}$  feature subsets. For MNIST, certain attributions must be estimated rather than calculated exactly: Occlusion attributions are computed exactly, whereas Shapley, Banzhaf, LIME, and RISE are estimated with 10,000 random feature sets, and Shapley and Banzhaf use a least squares regression similar to KernelSHAP [45].
<sup>3</sup>https://github.com/fastai/imagenette
For CIFAR-10 and Imagenette, we focus on Occlusion, RISE, and Shapley values, computed with the same settings as for MNIST, except RISE and Shapley values are estimated with 20,000 random feature sets for Imagenette due to its large number of features.
Figure 6: Identical to Figure 1 and shown here again for readability. Shapley attribution difference for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include (a) the wine quality dataset with FCNs and baseline (replacing with training set minimums), marginal, and conditional feature removal; and (b) MNIST with CNNs and baseline feature removal with either training set means or zeros. Error bars show the mean and  $95\%$  confidence intervals across explicit-and-perturbation pairs.
Figure 7: Attribution difference for ResNet-18 networks trained on CIFAR-10 with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
When using baseline feature removal, features are replaced with the training set minimums for the wine quality dataset, with the training set means or zeros for MNIST and CIFAR-10, and with zeros (identical to the training set means due to normalization) for Imagenette. The wine quality training set is used for removing features with their marginal distributions, while a conditional variational autoencoder (CVAE) is trained following the procedure in Frye et al. [27] to approximate feature removal with the conditional distribution (details in Appendix J). For both marginal and conditional feature removal in the wine quality dataset, 250 samples are used. Marginal and conditional feature removal are computationally expensive and less common for image data, so we use only baseline feature removal as is often done in practice. Finally, input perturbations are generated as in Appendix A.1, with 10 random perturbations applied per explicand and with perturbation norms  $\{0.1, 0.5, 1, 2\}$  for the wine quality dataset, MNIST, and CIFAR-10. For Imagenette, 5 random perturbations with norms  $\{50, 100, 200, 400\}$  are applied per explicand to obtain similar perturbation strength per input.
Results. Figure 6 shows that Shapley value attribution differences decrease with increasing weight decay, across different feature removal techniques, for both the wine quality dataset and MNIST, and for all perturbation norms in  $\{0.1, 0.5, 1, 2\}$ . This demonstrates the practical implication that removal-based attributions can be made more robust against input perturbation by training with weight decay. Interestingly, weight decay has also been shown to improve the robustness of gradient-based attributions [22]. In Appendix K, similar results are observed for Occlusion, Banzhaf, RISE, and LIME attributions, and also when we measure explanation similarity using Pearson or Spearman rank correlation rather than  $\ell_2$  distance. For ResNet-18 networks trained on CIFAR-10 and ResNet-50 networks trained on Imagenette, similar trends are observed for Occlusion, RISE, and Shapley attributions (Figure 7 and Figure 8).
Figure 8: Attribution difference for ResNet-50 networks trained on Imagenette with increasing weight decay, under input perturbations with perturbation norms  $\{50,100,200,400\}$ . Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
A.4 Implication II: sanity checks for removal-based attributions
The robustness bound for model perturbation in Theorem 2 suggests that the attribution difference between an intact and perturbed model should be small if the model perturbation is small. Similarly, we expect that if the perturbation intensifies, the attribution differences may become larger. One way to perturb a neural network with increasing intensity is to sequentially randomize its parameters from the output to input layer, a procedure known as cascading randomization introduced by Adebayo
et al. [2] for sanity checking feature attributions. Here, we study whether removal-based attributions indeed pass the parameter randomization check, as suggested by our theoretical bound.
Figure 9: Identical to Figure 2 and shown here again for readability. Sanity checks for attributions using cascading randomization for the FCN trained on the wine quality dataset. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the mean and  $95\%$  confidence intervals across 10 random seeds.
Figure 10: Sanity checks for attributions using cascading randomization for the CNN trained on MNIST. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the mean and  $95\%$  confidence intervals across 10 random seeds.
Datasets. We use the same datasets and splits as in Appendix A.3.
Setup. The best performing networks from Appendix A.3 are used here. The settings for removal-based attributions are the same as in Appendix A.3, except 50,000 random feature sets are used for more stable Shapley estimations in Imagenette. Parameters for each network are randomized in a cascading fashion from the output to input layer as in Adebayo et al. [2], with attributions computed after randomizing each layer and compared to the original attributions using Pearson correlation and Spearman rank correlation. Randomization is performed using the layer's default initialization. Feature attributions with the original model are computed twice and compared to capture variations due to sampling in feature removal and in attribution estimation using feature sets. This experiment is run 10 times with different random seeds for the wine quality dataset, MNIST, and CIFAR-10, and
3 times for Imagenette. Gradient-based attribution methods such as Vanilla Gradients [56], Integrated Gradients [60], and Grad  $\times$  Input [55] are included as references, as they were the focus of the original work on sanity checks [2].
Results. As suggested by our robustness bound for model perturbation, the original and perturbed removal-based attributions become dissimilar as more parameters are randomized, leading to near-zero correlations when the entire network is re-initialized (Figure 9). This shows that removal-based attributions reliably pass the model randomization sanity check. In contrast, we observe that for Integrated Gradients and Grad  $\times$  Input, the correlation between intact and perturbed attributions remains around 0.5 even when the entire network is randomized. Similarly, for the CNN trained on MNIST, the original and perturbed removal-based attributions have lower correlations as more parameters are randomized, whereas the correlations decrease only slightly for Integrated Gradients and Grad  $\times$  Input (Figure 10). Similar trends are observed for the ResNet-18 trained on CIFAR-10 and ResNet-50 trained on Imagenette (Figure 11, Figure 12, and Figure 29).
Figure 11: Sanity checks for attributions using cascading randomization for the ResNet-18 trained on CIFAR-10. Features are removed by replacing them with training set means. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the means and  $95\%$  confidence intervals across 10 random seeds. Missing points correspond to undefined correlations due to constant attributions under model randomization.
Figure 12: Sanity checks for attributions using cascading randomization for the ResNet-50 trained on Imagenette. Features are removed by replacing them with zeros (identical to training set means due to normalization). Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the means and  $95\%$  confidence intervals across 3 random seeds. Missing points correspond to undefined correlations due to constant attributions under model randomization.
B Proofs: prediction robustness to input perturbations
We now re-state and prove our results from Section 3.1.
Lemma 1. When removing features using either the baseline or marginal approaches, the prediction function  $f(x_{S})$  for any feature set  $x_{S}$  is L-Lipschitz continuous:
$$
| f (x _ {S}) - f (x _ {S} ^ {\prime}) | \leq L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} \quad \forall x _ {S}, x _ {S} ^ {\prime} \in \mathbb {R} ^ {| S |}.
$$
Proof. The key insight for this result is that removing features using either the baseline or marginal approach is equivalent to using a distribution  $q(\boldsymbol{x}_S)$  that does not depend on the observed features. Because of that, we have:
$$
\begin{array}{l} | f (x _ {S}) - f \left(x _ {S} ^ {\prime}\right) | = \left| \int f \left(x _ {S}, x _ {\bar {S}}\right) q \left(x _ {\bar {S}}\right) d x _ {\bar {S}} - \int f \left(x _ {S} ^ {\prime}, x _ {\bar {S}}\right) q \left(x _ {\bar {S}}\right) d x _ {\bar {S}} \right| \\ \leq \int \left| f (x _ {S}, x _ {\bar {S}}) - f (x _ {S} ^ {\prime}, x _ {\bar {S}}) \right| \cdot q (x _ {\bar {S}}) d x _ {\bar {S}} \\ \leq L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} \cdot \int q (x _ {\bar {S}}) d x _ {\bar {S}} \\ = L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2}. \\ \end{array}
$$
$$
| f (x _ {S}) - f (x _ {S} ^ {\prime}) | \leq L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} + 2 B \cdot d _ {T V} \Big (p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S}), p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}) \Big),
$$
$$
d _ {T V} \Big (p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S}), p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}) \Big) \equiv \frac {1}{2} \Big \| p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S}) - p (\boldsymbol {x} _ {\bar {S}} \mid x _ {S} ^ {\prime}) \Big \| _ {1}.
$$
Proof. In this case, we must be careful to account for the different distributions used when marginalizing out the removed features.
$$
\begin{array}{l} | f (x _ {S}) - f (x _ {S} ^ {\prime}) | = \left| \int f (x _ {S}, x _ {\bar {S}}) p (x _ {\bar {S}} \mid x _ {S}) d x _ {\bar {S}} - \int f (x _ {S} ^ {\prime}, x _ {\bar {S}}) p (x _ {\bar {S}} \mid x _ {S} ^ {\prime}) d x _ {\bar {S}} \right| \\ \leq \int \left| f \left(x _ {S}, x _ {\bar {S}}\right) p \left(x _ {\bar {S}} \mid x _ {S}\right) - f \left(x _ {S} ^ {\prime}, x _ {\bar {S}}\right) p \left(x _ {\bar {S}} \mid x _ {S} ^ {\prime}\right) \right| d x _ {\bar {S}} \\ \leq \int | f (x _ {S}, x _ {\bar {S}}) p (x _ {\bar {S}} \mid x _ {S}) - f (x _ {S} ^ {\prime}, x _ {\bar {S}}) p (x _ {\bar {S}} \mid x _ {S}) | d x _ {\bar {S}} \\ + \int \left| f \left(x _ {S} ^ {\prime}, x _ {\bar {S}}\right) p \left(x _ {\bar {S}} \mid x _ {S}\right) - f \left(x _ {S} ^ {\prime}, x _ {\bar {S}}\right) p \left(x _ {\bar {S}} \mid x _ {S} ^ {\prime}\right) \right| d x _ {\bar {S}} \\ \leq \int \left| f (x _ {S}, x _ {\bar {S}}) - f (x _ {S} ^ {\prime}, x _ {\bar {S}}) \right| \cdot p (x _ {\bar {S}} \mid x _ {S}) d x _ {\bar {S}} \\ + \int \left| f \left(x _ {S} ^ {\prime}, x _ {\bar {S}}\right) \right| \cdot \left| p \left(x _ {\bar {S}} \mid x _ {S}\right) - p \left(x _ {\bar {S}} \mid x _ {S} ^ {\prime}\right) \right| d x _ {\bar {S}} \\ \leq L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} \cdot \int p (x _ {\bar {S}} \mid x _ {S}) d x _ {\bar {S}} + B \cdot \int | p (x _ {\bar {S}} \mid x _ {S}) - p (x _ {\bar {S}} \mid x _ {S} ^ {\prime}) | d x _ {\bar {S}} \\ = L \cdot \| x _ {S} - x _ {S} ^ {\prime} \| _ {2} + 2 B \cdot d _ {T V} \Big (p (x _ {\bar {S}} \mid x _ {S}), p (x _ {\bar {S}} \mid x _ {S} ^ {\prime}) \Big). \\ \end{array}
$$
Proof. Consider a Gaussian random variable  $\pmb{x}$  with mean  $\mu \in \mathbb{R}^d$  and positive definite covariance  $\Sigma \in \mathbb{R}^{d\times d}$ . We can partition the variable into two parts, denoted  $x_{1}$  and  $x_{2}$ , and then partition the variable and parameters as follows:
$$
\boldsymbol {x} = \left( \begin{array}{c} \boldsymbol {x} _ {1} \\ \boldsymbol {x} _ {2} \end{array} \right) \qquad \mu = \left( \begin{array}{c} \mu_ {1} \\ \mu_ {2} \end{array} \right) \qquad \Sigma = \left( \begin{array}{c c} \Sigma_ {1 1} & \Sigma_ {1 2} \\ \Sigma_ {2 1} & \Sigma_ {2 2} \end{array} \right).
$$
We can then consider the distribution for  $x_{1}$  conditioned on a specific  $x_{2}$  value. The conditional distribution is  $x_{1} \mid x_{2} \sim \mathcal{N}(\bar{\mu},\bar{\Sigma})$  with the parameters defined as
$$
\bar {\mu} = \mu_ {1} + \Sigma_ {1 2} \Sigma_ {2 2} ^ {- 1} (x _ {2} - \mu_ {2}) \quad \bar {\Sigma} = \Sigma_ {1 1} - \Sigma_ {1 2} \Sigma_ {2 2} ^ {- 1} \Sigma_ {2 1}.
$$
Our goal is to understand the total variation distance when conditioning on different values  $x_{2}, x_{2}^{\prime}$ . Pinsker's inequality gives us the following upper bound based on the KL divergence:
$$
d _ {T V} \left(p \left(\boldsymbol {x} _ {1} \mid x _ {2}\right), p \left(\boldsymbol {x} _ {1} \mid x _ {2} ^ {\prime}\right)\right) \leq \sqrt {\frac {1}{2} D _ {\mathrm {K L}} \left(p \left(\boldsymbol {x} _ {1} \mid x _ {2}\right) \mid | p \left(\boldsymbol {x} _ {1} \mid x _ {2} ^ {\prime}\right)\right)}.
$$