SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Small sample weights in LIME. The distribution of the ratio of non-zero elements to the total number of super-pixels, along with the corresponding weights for LIME and GLIME-BINOMIAL, is depicted in Figure 2. Notably, most samples exhibit approximately  $d / 2$  non-zero elements. However, when  $\sigma$  takes values such as 0.25 or 0.5, a significant portion of samples attains weights that are nearly zero. For instance, when  $\sigma = 0.25$  and  $\| \mathbf{z}'\|_0 = d / 2$ ,  $\pi (\mathbf{z}')$  reduces to  $\exp (-8d)$ , which is approximately  $10^{-70}$  for  $d = 20$ . Even with  $\| \mathbf{z}'\|_0 = d - 1$ ,  $\pi (\mathbf{z}')$  equals  $e^{-16}$ , approximating  $10^{-7}$ . Since LIME samples from  $\mathrm{Uni}(\{0,1\}^d)$ , the probability that a sample  $\mathbf{z}'$  has  $\| \mathbf{z}'\|_0 = d - 1$  or  $d$  is approximately  $2\times 10^{-5}$  when  $d = 20$ . Therefore, most samples have very small weights. Consequently, the sample estimation of the expectation in Equation 1 tends to be much smaller than the true expectation with high probability and is thus inaccurate (see Appendix B.3 for more details). Given the default regularization strength  $\lambda = 1$ , this imbalance implies the domination of the regularization term in the objective function of Equation 1. As a result, LIME tends to yield explanations close to zero in such cases, diminishing their meaningfulness and leading to instability.
GLIME converges exponentially faster than LIME in the presence of regularization. Through the integration of the weighting function into the sampling process, every sample uniformly carries a weight of 1, contributing equally to Equation 3. Our analysis reveals that GLIME requires substantially fewer samples than LIME to transition beyond the regime where the regularization term dominates. Consequently, GLIME-BINOMIAL converges exponentially faster than LIME. Recall that  $\hat{\mathbf{w}}^{\mathrm{LIME}}$  and  $\hat{\mathbf{w}}^{\mathrm{GLIME}}$  represent the empirical solutions of Equation 1 and Equation 3, respectively, obtained by replacing the expectations with the sample average.  $\hat{\mathbf{w}}^{\mathrm{BINOMIAL}}$  is the empirical solution of Equation 3 with the transformed sampling distribution  $\widetilde{\mathcal{P}} (\mathbf{z}^{\prime},\| \mathbf{z}^{\prime}\|_{0} = k) = e^{k / \sigma^{2}} / (1 + e^{1 / \sigma^{2}})^{d}$ , where  $k = 0,1,\dots ,d$ . In the subsequent theorem, we present the sample complexity bound for LIME (refer to Appendix B.4 for proof).
Theorem 4.1. Suppose samples  $\{\mathbf{z}_i^{\prime}\}_{i = 1}^{n}\sim \mathrm{Uni}(\{0,1\}^{d})$  are used to compute the LIME explanation. For any  $\epsilon >0,\delta \in (0,1)$  , if  $n = \Omega (\epsilon^{-2}d^{9}2^{8d}e^{8 / \sigma^{2}}\log (4d / \delta)),\lambda \leq n$  , we have  $\mathbb{P}(\| \hat{\mathbf{w}}^{LIME} - \mathbf{w}^{LIME}\|_{2} <   \epsilon)\geq 1 - \delta .\mathbf{w}^{LIME} = \lim_{n\to \infty}\hat{\mathbf{w}}^{LIME}.$
Next, we present the sample complexity bound for GLIME (refer to Appendix B.5 for proof).
Theorem 4.2. Suppose  $\mathbf{z}'\sim \mathcal{P}$  such that the largest eigenvalue of  $\mathbf{z}'(\mathbf{z}')^\top$  is bounded by  $R$  and  $\mathbb{E}[\mathbf{z}'(\mathbf{z}')^\top ] = (\alpha_1 - \alpha_2)\mathbf{I} + \alpha_2\mathbf{11}^\top$ ,  $\| Var(\mathbf{z}'(\mathbf{z}')^\top)\| _2\leq \nu^2$ ,  $|(z'f(\mathbf{z}))_i|\leq M$  for some  $M > 0$ .  $\{\mathbf{z}_i'\}_{i = 1}^n$  are i.i.d. samples from  $\mathcal{P}$  and are used to compute GLIME explanation  $\hat{\mathbf{w}}^{\mathrm{GLIME}}$ . For any  $\epsilon >0,\delta \in (0,1)$ , if  $n = \Omega (\epsilon^{-2}M^{2}\nu^{2}d^{3}\gamma^{4}\log (4d / \delta))$  where  $\gamma$  is a function of  $\lambda ,d,\alpha_{1},\alpha_{2}$ , we have  $\mathbb{P}(\| \hat{\mathbf{w}}^{\mathrm{GLIME}} - \mathbf{w}^{\mathrm{GLIME}}\| _2 < \epsilon)\geq 1 - \delta$ .  $\mathbf{w}^{\mathrm{GLIME}} = \lim_{n\to \infty}\hat{\mathbf{w}}^{\mathrm{GLIME}}$ .
Since GLIME-BINOMIAL samples from a binomial distribution, which is sub-Gaussian with parameters  $M = \sqrt{d}$ ,  $\nu = 2$ ,  $\alpha_{1} = 1 / (1 + e^{-1 / \sigma^{2}})$ ,  $\alpha_{2} = 1 / (1 + e^{-1 / \sigma^{2}})^{2}$ , and  $\gamma (\alpha_{1},\alpha_{2},d) = de^{2 / \sigma^{2}}$  (refer to Appendix B.5 for proof), we derive the following corollary:
Corollary 4.3. Suppose  $\{\mathbf{z}_i^{\prime}\}_{i = 1}^{n}$  are i.i.d. samples from  $\widetilde{\mathcal{P}} (\mathbf{z}^{\prime},\| \mathbf{z}^{\prime}\|_{0} = k) = e^{k / \sigma^{2}} / (1 + e^{1 / \sigma^{2}})^{d},k = 1,\ldots ,d$  and are used to compute GLIME-BINOMIAL explanation. For any  $\epsilon >0,\delta \in (0,1)$ , if  $n = \Omega (\epsilon^{-2}d^{5}e^{4 / \sigma^{2}}\log (4d / \delta))$ , we have  $\mathbb{P}(\| \hat{\mathbf{w}}^{Binomial} - \mathbf{w}^{Binomial}\|_2 < \epsilon)\geq 1 - \delta$ .  $\mathbf{w}^{Binomial} = \lim_{n\to \infty}\hat{\mathbf{w}}^{Binomial}$ .
Figure 3: The distribution of sample distances to the original input. The samples produced by LIME display a considerable distance from the original input  $\mathbf{x}$ , whereas the samples generated by GLIME demonstrate a more localized distribution. LIME has a tendency to overlook sampling points that are in close proximity to  $\mathbf{x}$ .
Comparing the sample complexities outlined in Theorem 4.1 and Corollary 4.3, it becomes evident that LIME necessitates an exponential increase of  $\exp (d,\sigma^{-2})$  more samples than GLIME-BINOMIAL for convergence. Despite both LIME and GLIME-BINOMIAL samples being defined on the binary set  $\{0,1\}$ , the weight  $\pi (\mathbf{z}^{\prime})$  associated with a sample  $\mathbf{z}^{\prime}$  in LIME is notably small. Consequently, the square loss term in LIME is significantly diminished compared to that in GLIME-BINOMIAL. This situation results in the domination of the regularization term over the square loss term, leading to solutions that are close to zero. For stable solutions, it is crucial that the square loss term is comparable to the regularization term. Consequently, GLIME-BINOMIAL requires significantly fewer samples than LIME to achieve stability.
4.2 Designing locally faithful explanation methods within GLIME
Non-local and biased sampling in LIME. LIME employs uniform sampling from  $\{0,1\}^d$  and subsequently maps the samples to the original input space  $\mathcal{X}$  with the inclusion of a reference. Despite the integration of a weighting function to enhance locality, the samples  $\{\mathbf{z}_i\}_{i=1}^n$  generated by LIME often exhibit non-local characteristics, limiting their efficacy in capturing the local behaviors of the model  $f$  (as depicted in Figure 3). This observation aligns with findings in [16, 21], which demonstrate that LIME frequently approximates the global behaviors instead of the local behaviors of  $f$ . As illustrated earlier, the weighting function contributes to LIME's instability, emphasizing the need for explicit enforcement of locality in the sampling process.
Local and unbiased sampling in GLIME. In response to these challenges, GLIME introduces a sampling procedure that systematically enforces locality without reliance on a reference point. One approach involves sampling  $\mathbf{z}' \sim \mathcal{P} = \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$  and subsequently obtaining  $\mathbf{z} = \mathbf{x} + \mathbf{z}'$ . This method, referred to as GLIME-GAUSS, utilizes a weighting function  $\pi(\cdot) \equiv 1$ , with other components chosen to mirror those of LIME. The feature attributions derived from this approach successfully mitigate the aforementioned issues. Similarly, alternative distributions, such as  $\mathcal{P} = \mathrm{Laplace}(\mathbf{0}, \sigma)$  or  $\mathcal{P} = \mathrm{Uni}([- \sigma, \sigma]^d)$ , can be employed, resulting in explanation methods known as GLIME-LAPLACE and GLIME-UNIFORM, respectively.
4.3 Sampling distribution selection for user-specific objectives
Users may possess specific objectives they wish the explanation method to fulfill. For instance, if a user seeks to enhance local fidelity within a neighborhood of radius  $\epsilon$ , they can choose a distribution and corresponding parameters aligned with this objective (as depicted in Figure 5). The flexible design of the sample distribution in GLIME empowers users to opt for a distribution that aligns with their particular use cases. Furthermore, within the GLIME framework, it is feasible to integrate feature correlation into the sampling distribution, providing enhanced flexibility. In summary, GLIME affords users the capability to make more tailored choices based on their individual needs and objectives.
Dataset and models. Our experiments are conducted on the ImageNet dataset<sup>1</sup>. Specifically, we randomly choose 100 classes and select an image at random from each class. The models chosen for explanation are ResNet18 [13] and the tiny Swin-Transformer [18] (refer to Appendix A.7 for results). Our implementation is derived from the official implementation of LIME<sup>2</sup>. The default segmentation algorithm in LIME, Quickshift [32], is employed. Implementation details of our experiments are provided in Appendix A.1. For experiment results on text data, please refer to Appendix A.9. For experiment results on ALIME, please refer to Appendix A.8.
Metrics. (1) Stability: To gauge the stability of an explanation method, we calculate the average top- $K$  Jaccard Index (JI) for explanations generated by 10 different random seeds. Let  $\mathbf{w}_1, \ldots, \mathbf{w}_{10}$  denote the explanations obtained from 10 random seeds. The indices corresponding to the top- $K$  largest values in  $\mathbf{w}_i$  are denoted as  $R_{i,:K}$ . The average Jaccard Index between pairs of  $R_{i,:K}$  and  $R_{j,:K}$  is then computed, where  $\mathrm{JI}(A,B) = |A \cap B| / |A \cup B|$ .
(2) Local Fidelity: To evaluate the local fidelity of explanations, reflecting how well they capture the local behaviors of the model, we employ two approaches. For LIME, which uses a non-local sampling neighborhood, we use the  $R^2$  score returned by the LIME implementation for local fidelity assessment [33]. Within GLIME, we generate samples  $\{\mathbf{z}_i\}_{i=1}^m$  and  $\{\mathbf{z}_i'\}_{i=1}^m$  from the neighborhood  $B_{\mathbf{x}}(\epsilon)$ . The squared difference between the model's output and the explanation's output on these samples is computed. Specifically, for a sample  $\mathbf{z}$ , we calculate  $(f(\mathbf{z}) - \hat{\mathbf{w}}^\top \mathbf{z}')^2$  for the explanation  $\hat{\mathbf{w}}$ . The local fidelity of an explanation  $\hat{\mathbf{w}}$  at the input  $\mathbf{x}$  is defined as  $1 / (1 + \frac{1}{m} \sum_i (f(\mathbf{z}_i) - \hat{\mathbf{w}}^\top \mathbf{z}_i'))^2$ , following the definition in [16]. To ensure a fair comparison between different distributions in GLIME, we set the variance parameter of each distribution to match that of the Gaussian distribution. For instance, when sampling from the Laplace distribution, we use Laplace  $(\mathbf{0}, \sigma / \sqrt{2})$ , and when sampling from the uniform distribution, we use  $\mathrm{Uni}([-\sqrt{3}\sigma, \sqrt{3}\sigma]^d)$ .
5.1 Stability of LIME and GLIME
LIME's instability and the influence of regularization/weighting. In Figure 4a, it is evident that LIME without the weighting function  $(\mathrm{LIME} + \pi = 1)$  demonstrates greater stability compared to its weighted counterpart, especially when  $\sigma$  is small (e.g.,  $\sigma = 0.25, 0.5$ ). This implies that the weighting function contributes to instability in LIME. Additionally, we observe that LIME without regularization  $(\mathrm{LIME} + \lambda = 0)$  exhibits higher stability than the regularized LIME, although the improvement is not substantial. This is because, when  $\sigma$  is small, the sample weights approach zero, causing the Ridge regression problem to become low-rank, leading to unstable solutions. Conversely, when  $\sigma$  is large, significant weights are assigned to all samples, reducing the effectiveness of regularization. For instance, when  $\sigma = 5$  and  $d = 40$ , most samples carry weights around 0.45, and even samples with only one non-zero element left possess weights of approximately 0.2. In such scenarios, the regularization term does not dominate, even with limited samples. This observation is substantiated by the comparable performance of LIME,  $\mathrm{LIME} + \pi = 1$ , and  $\mathrm{LIME} + \lambda = 0$  when  $\sigma = 1$  and 5. Further results are presented in Appendix A.2.
Enhancing stability in LIME with GLIME. In Figure 4a, it is evident that LIME achieves a Jaccard Index of approximately 0.4 even with over 2000 samples when using the default  $\sigma = 0.25$ . In contrast, both GLIME-BINOMIAL and GLIME-GAUSS provide stable explanations with only 200-400 samples. Moreover, with an increase in the value of  $\sigma$ , the convergence speed of LIME also improves. However, GLIME-BINOMIAL consistently outperforms LIME, requiring fewer samples for comparable stability. The logarithmic scale of the horizontal axis in Figure 4a highlights the exponential faster convergence of GLIME compared to LIME.
Convergence of LIME and GLIME-BINOMIAL to a common limit. In Figure 8 of Appendix A.3, we explore the difference and correlation between explanations generated by LIME and GLIME-BINOMIAL. Mean Squared Error (MSE) and Mean Absolute Error (MAE) are employed as metrics to quantify the dissimilarity between the explanations, while Pearson correlation and Spearman rank correlation assess their degree of correlation. As the sample size increases, both LIME and GLIME
$^{1}$ Code is available at https://github.com/thutzr/GLIME-General-Stable-and-Local-LIME-Explanation $^{2}$ https://github.com/marcotcr/lime
Figure 4: GLIME consistently enhances stability and local fidelity compared to LIME across various values of  $\sigma$ .
Figure 5: Local fidelity of GLIME across different neighborhood radii. Explanations produced under a distribution with a standard deviation of  $\sigma$  demonstrate the ability to capture behaviors within local neighborhoods with radii exceeding  $\sigma$ .
BINOMIAL exhibit greater similarity and higher correlation. The dissimilarity in their explanations diminishes rapidly, approaching zero when  $\sigma$  is significantly large (e.g.,  $\sigma = 5$ ).
5.2 Local fidelity of LIME and GLIME
Enhancing local fidelity with GLIME. A comparison of the local fidelity between LIME and the explanation methods generated by GLIME is presented in Figure 4b. Utilizing 2048 samples for each image to compute the  $R^2$  score, GLIME consistently demonstrates superior local fidelity compared to LIME. Particularly, when  $\sigma = 0.25$  and 0.5, LIME exhibits local fidelity that is close to zero, signifying that the linear approximation model  $(\hat{\mathbf{w}}^{\mathrm{LIME}})^{\top}\mathbf{z}^{\prime}$  is nearly constant. Through the explicit integration of locality into the sampling process, GLIME significantly improves the local fidelity of the explanations.
Local fidelity analysis of GLIME under various sampling distributions. In Figure 5, we assess the local fidelity of GLIME employing diverse sampling distributions:  $\mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$ , Laplace  $(\mathbf{0},\sigma /\sqrt{2})$
(a) Human-interpretability results for accurate predictions. Participants consistently rated GLIME significantly higher than LIME, indicating that GLIME serves as a more effective tool for explaining model predictions.
(b) Human-interpretability results for incorrect predictions. Participants consistently rated GLIME significantly higher than LIME, indicating that GLIME excels in identifying the model's errors more effectively than LIME.
Figure 6: GLIME helps explaining model predictions better than LIME.
and  $\mathrm{Uni}([-\sqrt{3}\sigma, \sqrt{3}\sigma]^d)$ . The title of each sub-figure corresponds to the standard deviation of these distributions. Notably, we observe that the value of  $\sigma$  does not precisely align with the radius  $\epsilon$  of the intended local neighborhood for explanation. Instead, local fidelity tends to peak at larger  $\epsilon$  values than the corresponding  $\sigma$ . Moreover, different sampling distributions achieve optimal local fidelity for distinct  $\epsilon$  values. This underscores the significance of selecting an appropriate distribution and parameter values based on the specific radius  $\epsilon$  of the local neighborhood requiring an explanation. Unlike LIME, GLIME provides the flexibility to accommodate such choices. For additional results and analysis, please refer to Appendix A.5.
5.3 Human experiments
In addition to numerical experiments, we conducted human-interpretability experiments to evaluate whether GLIME provides more meaningful explanations to end-users. The experiments consist of two parts, with 10 participants involved in each. The details of the procedures employed in conducting the experiments is presented in the following.
1. Can GLIME improve the comprehension of the model's predictions? To assess this, we choose images for which the model's predictions are accurate. Participants are presented with the original images, accompanied by explanations generated by both LIME and GLIME. They are then asked to evaluate the degree of alignment between the explanations from these algorithms and their intuitive understanding. Using a 1-5 scale, where 1 indicates a significant mismatch and 5 signifies a strong correspondence, participants rate the level of agreement.
2. Can GLIME assist in identifying the model's errors? To explore this, we select images for which the model's predictions are incorrect. Participants receive the original images along with explanations generated by both LIME and GLIME. They are then asked to assess the degree to which these explanations aid in understanding the model's behaviors and uncovering the reasons behind the inaccurate predictions. Using a 1-5 scale, where 1 indicates no assistance and 5 signifies substantial aid, participants rate the level of support provided by the explanations.
Figure 6 presents the experimental results. When participants examined images with accurate model predictions, along with explanations from LIME and GLIME, they assigned an average score of 2.96 to LIME and 3.37 to GLIME. On average, GLIME received a score 0.41 higher than LIME. Notably, in seven out of the ten instances, GLIME achieved a higher average score than LIME.
In contrast, when participants examined images with incorrect model predictions, accompanied by explanations from LIME and GLIME, they assigned an average score of 2.33 to LIME and 3.42 to GLIME. Notably, GLIME outperformed LIME with an average score 1.09 higher across all ten images. These results strongly indicate that GLIME excels in explaining the model's behaviors.
Post-hoc local explanation methods. In contrast to inherently interpretable models, black-box models can be explained through post-hoc explanation methods, which are broadly categorized as model-agnostic or model-specific. Model-specific approaches, such as Gradient [2, 27], SmoothGrad [28], and Integrated Gradient [30], assume that the explained model is differentiable and that gradient access is available. For instance, SmoothGrad generates samples from a Gaussian distribution centered at the given input and computes their average gradient to mitigate noise. On the other hand, model-agnostic methods, including LIME [22] and Anchor [23], aim to approximate the local model behaviors using interpretable models, such as linear models or rule lists. Another widely-used model-agnostic method, SHAP [19], provides a unified framework that computes feature attributions based on the Shapley value and adheres to several axioms.
Instability of LIME. Despite being widely employed, LIME is known to be unstable, evidenced by divergent explanations under different random seeds [35, 34, 38]. Many efforts have been devoted to stabilize LIME explanations. Zafar et al. [35] introduced a deterministic algorithm that utilizes hierarchical clustering for grouping training data and k-nearest neighbors for selecting relevant data samples. However, the resulting explanations may not be a good local approximation. Addressing this concern, Shankaranarayana et al. [24] trained an auto-encoder to function as a more suitable weighting function in LIME. Shi et al. [25] incorporated feature correlation into the sampling step and considered a more restricted sampling distribution, thereby enhancing stability. Zhou et al. [40] employed a hypothesis testing framework to determine the necessary number of samples for ensuring stable explanations. However, this improvement came at the expense of a substantial increase in computation time.
Impact of references. LIME, along with various other explanation methods, relies on references (also known as baseline inputs) to generate samples. References serve as uninformative inputs meant to represent the absence of features [4, 30, 26]. Choosing an inappropriate reference can lead to misleading explanations. For instance, if a black image is selected as the reference, important black pixels may not be highlighted [15, 6]. The challenge lies in determining the appropriate reference, as different types of references may yield different explanations [14, 6, 15]. In [15], both black and white references are utilized, while [7] employs constant, noisy, and Gaussian blur references simultaneously. To address the reference specification issue, [6] proposes Expected Gradient, considering each instance in the data distribution as a reference and averaging explanations computed across all references.
In this paper, we introduce GLIME, a novel framework that extends the LIME method for local feature importance explanations. By explicitly incorporating locality into the sampling procedure and enabling more flexible distribution choices, GLIME mitigates the limitations of LIME, such as instability and low local fidelity. Experimental results on ImageNet data demonstrate that GLIME significantly enhances stability and local fidelity compared to LIME. While our experiments primarily focus on image data, the applicability of our approach readily extends to text and tabular data.
The authors would like to thank the anonymous reviewers for their constructive comments. Zeren Tan and Jian Li are supported by the National Natural Science Foundation of China Grant (62161146004). Yang Tian is supported by the Artificial and General Intelligence Research Program of Guo Qiang Research Institute at Tsinghua University (2020GQG1017).
[1] Alejandro Barredo Arrieta, Natalia Díaz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82-115, 2020.
[38] Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. "why should you trust my explanation?" understanding uncertainty in lime explanations. arXiv preprint arXiv:1904.12991, 2019.
[39] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[40] Zhengze Zhou, Giles Hooker, and Fei Wang. S-lime: Stabilized-lime for model explanation. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2429-2438, 2021.
A.1 Implementation details
Dataset selection. The experiments use images from the validation set of the ImageNet-1k dataset. To ensure consistency, a fixed random seed (2022) is employed. Specifically, 100 classes are uniformly chosen at random, and for each class, an image is randomly selected.
Models. The pretrained models used are sourced from torchvision.models, with the weights parameter set toImagenet1K_V1.