SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To generalize our earlier results to this setting, we first remark that querying the model with each subset induces a vector  $v \in \mathbb{R}^{2^g}$  containing the predictions for each union  $G_T$  with  $T \subseteq [g]$ . The attributions are then calculated according to  $\phi(f, x) = Av$ , where  $A \in \mathbb{R}^{g \times 2^g}$  is the linear operator associated with each summary technique (see Table 3). Our approach for bounding the attribution difference under either input or model perturbation is similar here, because we can use the same inequalities from Lemma 7, or  $||A(v - v')||_2 \leq ||A||_2 \cdot ||v - v'||_2$  and  $||A(v - v')||_2 \leq ||A||_{1,\infty} \cdot ||v - v'||_\infty$ , depending on which norm is available for the vectors of predictions  $||v - v'||$ .
Following the same proof techniques used previously, we can guarantee the following bounds in the feature grouping setting.
Input perturbation. Lemma 1 and Lemma 3 guarantee that  $f(x_{S})$  is Lipschitz continuous for any subset  $S \subseteq [d]$ . These results automatically apply to the specific subsets induced by feature grouping. If  $f(x_{S})$  is  $L^{\prime}$ -Lipschitz continuous for all  $S \subseteq [d]$ , we therefore have the following inequality,
$$
| f (x _ {G _ {T}}) - f (x _ {G _ {T}} ^ {\prime}) | \leq L ^ {\prime} \cdot | | x _ {G _ {T}} - x _ {G _ {T}} ^ {\prime} | | _ {2},
$$
for all group subsets  $T \subseteq [g]$ . As a result, we can guarantee that the induced vectors  $v, v' \in \mathbb{R}^{2^g}$  for two inputs  $x, x'$  have the following bound:
$$
\left| \left| v - v ^ {\prime} \right| \right| _ {\infty} \leq L ^ {\prime} \cdot \left| \left| x - x ^ {\prime} \right| \right| _ {2}.
$$
Finally, we see that Theorem 1 only needs to be changed by replacing  $d$  with  $g$  in the  $h(\text{summary})$  term. This bound may even be tighter when we use feature groups, because the Lipschitz constant may be smaller when we restrict our attention to a smaller number of subsets.
Model perturbation. Lemma 5 and Lemma 6 guarantee that the predictions  $f(x_{S})$  and  $f^{\prime}(x_S)$  for two models  $f, f^{\prime}$  can differ by no more than the functional difference  $||f - f^{\prime}||$ , where the specific norm depends on the feature removal approach. These results apply to all subsets  $S\subseteq [d]$ , so they automatically apply to those induced by feature grouping. We therefore have the following inequality,
$$
\left| f \left(x _ {G _ {T}}\right) - f ^ {\prime} \left(x _ {G _ {T}}\right) \right| \leq \left| \left| f - f ^ {\prime} \right| \right|,
$$
for all  $x$  and all group subsets  $T \subseteq [g]$ . For the induced vectors  $v, v' \in \mathbb{R}^{2^g}$ , we also have
$$
\left| \left| v - v ^ {\prime} \right| \right| _ {\infty} \leq \left| \left| f - f ^ {\prime} \right| \right|.
$$
Finally, we see that Theorem 2 applies as well, where the only change required is replacing  $d$  with  $g$  in the  $h(\mathrm{summary})$  term.
H LIME weighted least squares linear operator
In this section, we discuss the linear operator  $A \in \mathbb{R}^{d \times 2^d}$  associated with LIME [53]. We begin with a theoretical characterization that focuses on LIME's default implementation choices, and we then corroborate these points with empirical results.
H.1 Theory
The observation that LIME is equivalent to a linear operator  $A \in \mathbb{R}^{d \times 2^d}$  is discussed in Appendix D. We pointed out there that LIME's specific operator depends on several implementation choices, including the choice of weighting kernel, regularization and intercept term. Here, we consider that no regularization is used, that an intercept term is fit, and we focus on the default weighting kernel in LIME's popular implementation. The default is the following exponential kernel,
$$
w (S) = \exp \left(- \frac {\left(1 - \sqrt {| S | / d}\right) ^ {2}}{\sigma^ {2}}\right), \tag {H.1.1}
$$
where  $\sigma^2 > 0$  is a hyperparameter. The user can set this parameter arbitrarily, but LIME also considers default values for each data type:  $\sigma^2 = 0.25$  for images,  $\sigma^2 = 25$  for language, and  $\sigma^2 = 0.75\sqrt{d}$  for tabular data. We find that these choices approach two limiting cases,  $\sigma^2 \to \infty$  and  $\sigma^2 \to 0$ , where LIME reduces to other summary techniques; namely, we find that these limiting cases respectively recover the Banzhaf value [9] and leave-one-out [67] summary techniques. We argue this first from a theoretical perspective, and Appendix H.2 then provides further empirical evidence.
Limiting case  $\sigma^2\to \infty$ . In this case, it is easy to see from eq. (H.1.1) that we have  $w(S)\rightarrow 1$  for all  $S\subseteq [d]$ . The weighted least squares objective for LIME then reduces to an unweighted least squares problem, which according to results from Hammer and Holzman [32] recovers the Banzhaf value. We might expect that the default settings of  $\sigma^2$  approach this outcome in practice (e.g.,  $\sigma^2 = 25$ ), and we verify this in our empirical results.
Limiting case  $\sigma^2 \to 0$ . As we let  $\sigma^2$  approach 0, increasing weight is applied to  $w(S)$  when the cardinality  $|S|$  is large. Similar to a softmax activation, the differences between cardinalities become exaggerated with low temperatures; for very low temperatures, the weight applied to  $|S| = d$  is much larger than the weight applied to  $|S| = d - 1$ , which is much larger than the weight for  $|S| = d - 2$ , etc. In this regime, applying weight only to the subset  $S = [d]$  leaves the problem undetermined, so the small residual weight on  $|S| = d - 1$  plays an important role. The problem begins to resemble one where only subsets with  $|S| \geq d - 1$  are taken into account, which yields a unique solution: following Covert et al. [17], this reduces LIME to leave-one-out (Occlusion). More formally, if we denote  $A_{\mathrm{LIME}}(\sigma^2)$  as the linear operator resulting from a specific  $\sigma^2$  value, we argue that  $\lim_{\sigma^2 \to 0} A_{\mathrm{LIME}}(\sigma^2)$  is the operator associated with leave-one-out. We do not prove this result due to the difficulty of deriving  $A_{\mathrm{LIME}}(\sigma^2)$  for a specific  $\sigma^2$  value, but Appendix H.2 provides empirical evidence for this claim.
H.2 Empirical analysis
Here, we denote the LIME linear operator with the exponential kernel as  $A_{\mathrm{LIME}}$  instead of  $A_{\mathrm{LIME}}(\sigma^2)$  for brevity. Recall from Appendix D that the LIME linear operator has the form  $A_{\mathrm{LIME}} = (Z'^\top W Z')_{[1:d]}^{-1} Z'^\top W$ , where  $Z' = \{0, 1\}^{2^d \times (d + 1)}$  is a binary matrix with all ones in the first column and the other columns indicating whether each feature is in each subset, and  $W \in \mathbb{R}^{2^d \times 2^d}$  is a diagonal matrix with  $W_{S,S} = w(S)$  for all  $S \in [d]$  and zeros elsewhere.
To empirically compare the linear operator of LIME to those of the Banzhaf value and leave-one-out (Occlusion), we first derive closed-form expressions of  $Z^{\prime \top}WZ^{\prime}$  and  $Z^{\prime \top}W$  as in the following proposition.
4https://github.com/marcotcr/lime
Proposition 2. Define
$$
a _ {0} \equiv \sum_ {k = 0} ^ {d} {\binom {d} {k}} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right),
$$
$$
a _ {1} \equiv \sum_ {k = 1} ^ {d} \binom {d - 1} {k - 1} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right), a n d
$$
$$
a _ {2} \equiv \sum_ {k = 2} ^ {d} \binom {d - 2} {k - 2} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right).
$$
Then  $(Z^{\prime}^{\top}WZ^{\prime})$  is a  $(d + 1)\times (d + 1)$  matrix with the following form:
$$
(Z ^ {\prime \top} W Z ^ {\prime}) = \left( \begin{array}{c c} a _ {0} & B \\ B ^ {\top} & C \end{array} \right),
$$
where  $B = (a_{1},\dots,a_{1})\in \mathbb{R}^{1\times d}$ , and  $C$  is a  $d\times d$  matrix with  $a_1$  on the diagonal entries and  $a_2$  on the off-diagonal entries. Furthermore, for all  $S\in [d]$
$$
(Z ^ {\prime} ^ {\top} W) _ {i S} = \left\{ \begin{array}{l l} w (S), & i f i = 1 \\ \mathbb {1} \{i - 1 \in S \} w (S) & o t h e r w i s e. \end{array} \right.
$$
Proof. We first find an expression for each element in  $Z'^\top WZ'$ . Consider the first diagonal entry:
$$
\begin{array}{l} \left(Z ^ {\prime \top} W Z ^ {\prime}\right) _ {1 1} = Z _ {1} ^ {\top} W Z _ {1} ^ {\prime} = \mathbf {1} _ {2 ^ {d}} ^ {\top} W \mathbf {1} _ {2 ^ {d}} \\ = \sum_ {S \subseteq [ d ]} w (S) \\ = \sum_ {k = 0} ^ {d} {\binom {d} {k}} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right) = a _ {0}. \\ \end{array}
$$
For the rest of the diagonal entries, we have:
$$
\begin{array}{l} \left(Z ^ {\prime \top} W Z ^ {\prime}\right) _ {i i} = Z _ {i} ^ {\prime \top} W Z _ {i} ^ {\prime} = \sum_ {S: i - 1 \in S} w (S) \\ = \sum_ {k = 1} ^ {d} \binom {d - 1} {k - 1} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right) = a _ {1}, \\ \end{array}
$$
for  $i = 2,\dots,d + 1$ . For the off-diagonal entries, we have:
$$
(Z ^ {\prime \top} W Z ^ {\prime}) _ {1 i} = (Z ^ {\prime \top} W Z ^ {\prime}) _ {i 1} = \mathbf {1} _ {2 ^ {d}} ^ {\top} W Z _ {i} ^ {\prime} = \sum_ {S: i - 1 \in S} w (S) = a _ {1},
$$
for  $i = 2, \dots, d + 1$ . Finally, consider the rest of the off-diagonal entries:
$$
\begin{array}{l} (Z ^ {\prime \top} W Z ^ {\prime}) _ {i j} = Z _ {i} ^ {\prime \top} W Z _ {j} ^ {\prime} = \sum_ {S: i - 1 \in S, j - 1 \in S} w (S) \\ = \sum_ {k = 2} ^ {d} \binom {d - 2} {k - 2} \exp \left(- \left(1 - \sqrt {k / d}\right) ^ {2} / \sigma^ {2}\right) = a _ {2}, \\ \end{array}
$$
for  $i,j = 2,\ldots ,d + 1$  such that  $i\neq j$ . Taken all these together,  $Z^{\prime \top}WZ^{\prime}$  has the closed form expression as shown in the proposition.
We now proceed to find an expression for each element in  $Z^{\prime \top}W$ , which has the first row entries:
$$
(Z ^ {\prime} ^ {\top} W) _ {1 S} = \mathbf {1} _ {2 ^ {d}} ^ {\top} W _ {S} = w (S),
$$
for all  $S \subseteq [d]$ , whereas the rest of the rows have entries:
$$
(Z ^ {\prime \top} W) _ {i S} = Z _ {i} ^ {\top} W _ {S} = \mathbb {1} \{i - 1 \in S \} w (S),
$$
for  $i = 2, \dots, d + 1$ . Hence the closed form expression of  $Z^{\prime \top}W$  in the proposition follows.
Figure 14: Frobenius norm of the difference between the LIME linear operator  $A_{\mathrm{LIME}}$  and the Banzhaf linear operator  $A_{\mathrm{Banzhaf}}$ , with varying LIME exponential kernel hyperparameter  $\sigma^2$  and number of features. We include reference lines for the default LIME hyperparameters:  $\sigma^2 = 0.25$  for images,  $\sigma^2 = 0.75\sqrt{d}$  for tabular data, and  $\sigma^2 = 25$  for text data.
With closed form expressions for  $Z^{\prime \top}WZ^{\prime}$  and  $Z^{\prime \top}W$ , we numerically compute the LIME linear operator  $A_{\mathrm{LIME}} = (Z^{\prime \top}WZ^{\prime})_{[1:]}^{-1}Z^{\prime \top}W$  and compare it to the linear operators of the Banzhaf value  $(A_{\mathrm{Banzhaf}})$  and Occlusion  $(A_{\mathrm{Occlusion}})$  with both large and small  $\sigma^2$  values.
As shown in Figure 14, as  $\sigma^2$  increases, the Frobenius norm difference between the LIME and Banzhaf linear operators approaches zero, verifying our theoretical argument for the limiting case of  $\sigma^2 \to \infty$ . Furthermore, we observe that  $A_{\mathrm{LIME}}$  and  $A_{\mathrm{Banzhaf}}$  have near zero difference for all default  $\sigma^2$  values when there are  $\{10, 15, 20, 25\}$  features. As  $\sigma^2$  approaches zero, the Frobenius norm difference between the LIME and Occlusion linear operators empirically approaches zero (Figure 15), confirming our intuition for the limiting case where  $\sigma^2 \to 0$ . Based on our computational results, LIME with the default  $\sigma^2$  values indeed has similar spectral norm  $\| \cdot \|_2$  and the operator norm  $\| \cdot \|_{1,\infty}$  compared to the Banzhaf value (Figure 16).
Figure 15: Frobenius norm of the difference between the LIME linear operator  $A_{\mathrm{LIME}}$  and the Occlusion linear operator  $A_{\mathrm{Occlusion}}$ , with varying LIME parameter  $\sigma^2$  and number of features.
Figure 16: Computed norms of summary technique linear operators. Note that all lines except for RISE overlap on the left-hand-side showing the operator norm.
I Additional results for synthetic experiments
Figure 17: Occlusion, Banzhaf, RISE, and LIME attribution differences under input perturbation with various perturbation norms in our synthetic data experiment. Lines bounding the maximum attribution difference at each perturbation norm are shown as empirical upper bounds. Theoretical upper bounds are included for baseline and marginal feature removal, and omitted for conditional feature removal because it is infinite for  $\rho = 1$ .
$\alpha$
P
Figure 18: Slopes of empirical and theoretical upper bounds with respect to the input perturbation norm, for Occlusion, Banzhaf, RISE, and LIME attribution differences in the synthetic experiment. The parameter  $\alpha$  varies the logistic regression Lipschitz constant, and  $\rho$  varies the correlation between the two synthetic features  $x_{1}, x_{2}$ .
Figure 19: Occlusion, Banzhaf, RISE, and LIME attribution differences between the logistic regression classifiers  $f$ ,  $f'$  as constructed in Appendix A.2 with baseline, marginal, and conditional feature removal, and varying correlation  $\rho$ .
J Implementation details
Fully connected networks. For the wine quality dataset, we trained FCNs with varying levels of weight decay. The architecture consists of 3 hidden layers of width 128 with ReLU activations and was trained with Adam for 50 epochs with learning rate 0.001. We also trained a conditional VAE (CVAE) following the procedure in Frye et al. [27] to approximate conditional feature removal. The CVAE encoder and decoder both consist of 2 hidden layers of width 64 with ReLU activations and latent dimension 8. We trained the CVAE using a prior regularization strength of 0.5 for at most 500 epochs with Adam and learning rate 0.0001, with early stopping if the validation mean squared error did not improve for 100 consecutive epochs. All models were trained with NVIDIA GeForce RTX 2080 Ti GPUs with 11GB memory.
Convolutional networks. We trained CNNs for MNIST with varying levels of weight decay. Following the LeNet-like architecture in Adebayo et al. [2], our network has the following structure: Input  $\rightarrow$  Conv(5 × 5, 32)  $\rightarrow$  MaxPool(2 × 2)  $\rightarrow$  Conv(5 × 5, 64)  $\rightarrow$  MaxPool(2 × 2)  $\rightarrow$  Flatten  $\rightarrow$  Linear(1024, 1024)  $\rightarrow$  Linear(1024, 10). We trained the model with Adam for 200 epochs with learning rate 0.001. The same GPUs for FCN training were used to train the CNNs.
ResNets. We trained ResNet-18 networks for CIFAR-10 from scratch with Adam for 500 epochs with learning rate 0.00005. The same GPUs for FCN training were used to train the ResNet-18 networks. We used a ResNet-50 network pre-trained on ImageNet and fine-tuned it forImagenette with Adam for 20 epochs with learning rate 0.0005. The ResNet-50 networks were trained with NVIDIA Quadro RTX6000 GPUs with 24GB memory.
K Additional results for practical implications
Figure 20: Test accuracy of FCN trained on the wine quality dataset and CNN trained on MNIST with varying degree of weight decay.
Figure 21: Test accuracy of ResNet-18 trained on CIFAR-10 and ResNet-50 fine-tuned on Imagenette with varying degree of weight decay.
Figure 22: Attribution difference for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include the wine quality dataset with FCNs and baseline (replacing with training set minimums), marginal, and conditional feature removal. Error bars show the mean and  $95\%$  confidence intervals across explicit-and-perturbation pairs.
Figure 23: Pearson correlation of attributions for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include the wine quality dataset with FCNs and baseline (replacing with training set minimums), marginal, and conditional feature removal. Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
Figure 24: Spearman rank correlation of attributions for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include the wine quality dataset with FCNs and baseline (replacing with training set minimums), marginal, and conditional feature removal. Error bars show the means and  $95\%$  confidence intervals across explicit-and-perturbation pairs.
Figure 25: Attribution difference for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include MNIST with CNNs and baseline feature removal with either training set means or zeros. Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
Figure 26: Pearson correlation of attributions for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include MNIST with CNNs and baseline feature removal with either training set means or zeros. Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
Figure 27: Spearman rank correlation of attributions for networks trained with increasing weight decay, under input perturbations with perturbation norms  $\{0.1, 0.5, 1, 2\}$ . The results include MNIST with CNNs and baseline feature removal with either training set means or zeros. Error bars show the means and  $95\%$  confidence intervals across expicand-perturbation pairs.
Figure 28: Sanity checks for attributions using cascading randomization for the CNN trained on MNIST. Features are removed by replacing them with zeros. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the means and  $95\%$  confidence intervals across 10 random seeds.
Figure 29: Sanity checks for attributions using cascading randomization for the ResNet-18 trained on CIFAR-10. Features are removed by replacing them with zeros. Attribution similarity is measured by Pearson correlation and Spearman rank correlation. We show the means and  $95\%$  confidence intervals across 10 random seeds. Missing points correspond to undefined correlations due to constant attributions under model randomization.