SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeren Tan
thutzr1019@gmail.com
Yang Tian
tyanyang04@gmail.com
Jian Li
lapordge@gmail.com
As black-box machine learning models grow in complexity and find applications in high-stakes scenarios, it is imperative to provide explanations for their predictions. Although Local Interpretable Model-agnostic Explanations (LIME) [22] is a widely adopted method for understanding model behaviors, it is unstable with respect to random seeds [35, 24, 3] and exhibits low local fidelity (i.e., how well the explanation approximates the model's local behaviors) [21, 16]. Our study shows that this instability problem stems from small sample weights, leading to the dominance of regularization and slow convergence. Additionally, LIME's sampling neighborhood is non-local and biased towards the reference, resulting in poor local fidelity and sensitivity to reference choice. To tackle these challenges, we introduce GLIME, an enhanced framework extending LIME and unifying several prior methods. Within the GLIME framework, we derive an equivalent formulation of LIME that achieves significantly faster convergence and improved stability. By employing a local and unbiased sampling distribution, GLIME generates explanations with higher local fidelity compared to LIME. GLIME explanations are independent of reference choice. Moreover, GLIME offers users the flexibility to choose a sampling distribution based on their specific scenarios.
Why a patient is predicted to have a brain tumor [10]? Why a credit application is rejected [11]? Why a picture is identified as an electric guitar [22]? As black-box machine learning models continue to evolve in complexity and are employed in critical applications, it is imperative to provide explanations for their predictions, making interpretability a central concern [1]. In response to this imperative, various explanation methods have been proposed [39, 26, 4, 19, 22, 28, 30], aiming to provide insights into the internal mechanisms of deep learning models.
Among the various explanation methods, Local Interpretable Model-agnostic Explanations (LIME) [22] has attracted significant attention, particularly in image classification tasks. LIME explains predictions by assigning each region within an image a weight indicating the influence of this region to the output. This methodology entails segmenting the image into super-pixels, as illustrated in the lower-left portion of Figure 1a, introducing perturbations, and subsequently approximating the local model prediction using a linear model. The approximation is achieved by solving a weighted Ridge regression problem, which estimates the impact (i.e., weight) of each super-pixel on the classifier's output.
Nevertheless, LIME has encountered significant instability due to its random sampling procedure [35, 24, 3]. In LIME, a set of samples perturbing the original image is taken. As illustrated in Figure 1a, LIME explanations generated with two different random seeds display notable disparities, despite using a large sample size (16384). The Jaccard index, measuring similarity between two explanations on a scale from 0 to 1 (with higher values indicating better similarity), is below 0.4. While many prior studies aim to enhance LIME's stability, some sacrifice computational time for stability [24, 40], and others may entail the risk of overfitting [35]. The evident drawback of unstable
(a) LIME and GLIME explanations for an ImageNet image on the tiny Swin-Transformer with two random seeds.  $\sigma = 0.25$  and a sample size of 16384 are employed. The numerical value presented below the second row represents the Jaccard Index between the two explanation maps.
(b) Distribution of LIME and GLIME-GAUSS in 2D example (left) and 1D example explanations (right). LIME samples are constrained to one side of  $\mathbf{r}$  and are more distant from  $\mathbf{x}$ . Additionally, LIME explanations exhibit variability with distinct reference points.
Figure 1: GLIME enhances stability and local fidelity compared to LIME. (a) LIME demonstrates instability with the default parameter  $\sigma$ , while GLIME consistently provides meaningful explanations. (b) LIME samples from a biased and non-local neighborhood, a limitation overcome by GLIME.
explanations lies in their potential to mislead end-users and hinder the identification of model bugs and biases, given that LIME explanations lack consistency across different random seeds.
In addition to its inherent instability, LIME has been found to have poor local fidelity [16, 21]. As depicted in Figure 1a, the  $R^2$  value for LIME on the sample image approaches zero (refer also to Figure 4b). This problem arises from the non-local and skewed sampling space of LIME, which is biased towards the reference. More precisely, the sampling space of LIME consists of the corner points of the hypercube defined by the explained instance and the selected reference. For instance, in the left section of Figure 1b, only four red points fall within LIME's sampling space, yet these points are distant from  $\mathbf{x}$ . As illustrated in Figure 3, the  $L_2$  distance between LIME samples of the input  $\mathbf{x}$  and  $\mathbf{x}$  is approximately  $0.7\| \mathbf{x}\| _2$  on ImageNet. Although LIME incorporates a weighting function to enforce locality, an explanation cannot be considered as local if the samples themselves are non-local, leading to a lack of local fidelity in the explanation. Moreover, the hypercube exhibits bias towards the reference, resulting in explanations designed to explain only a portion of the local neighborhood. This bias causes LIME to generate different explanations for different references, as illustrated in Figure 1b (refer to Appendix A.4 for more analysis and results).
To tackle these challenges, we present GLIME—a local explanation framework that generalizes LIME and five other methods: KernelSHAP [19], SmoothGrad [28], Gradient [36], DLIME [35], and ALIME [24]. Through a flexible sample distribution design, GLIME produces explanations that are more stable and faithful. Addressing LIME's instability issue, within GLIME, we derive an equivalent form of LIME, denoted as GLIME-BINOMIAL, by integrating the weighting function into the sampling distribution. GLIME-BINOMIAL ensures exponential convergence acceleration compared to LIME when the regularization term is presented. Consequently, GLIME-BINOMIAL demonstrates improved stability compared to LIME while preserving superior local fidelity (see Figure 4). Furthermore, GLIME enhances both local fidelity and stability by sampling from a local distribution independent of any specific reference point.
In summary, our contributions can be outlined as follows:
- We conduct an in-depth analysis to find the source of LIME's instability, revealing the interplay between the weighting function and the regularization term as the primary cause. Additionally, we attribute LIME's suboptimal local fidelity to its non-local and biased sampling space.
- We introduce GLIME as a more general local explanation framework, offering a flexible design for the sampling distribution. With varying sampling distributions and weights, GLIME serves as a generalization of LIME and five other preceding local explanation methods.
- By integrating weights into the sampling distribution, we present a specialized instance of GLIME with a binomial sampling distribution, denoted as GLIME-BINOMIAL. We demonstrate that GLIME-BINOMIAL, while maintaining equivalence to LIME, achieves faster convergence with significantly fewer samples. This indicates that enforcing locality in the sampling distribution is better than using a weighting function.
- With regard to local fidelity, GLIME empowers users to devise explanation methods that exhibit greater local fidelity. This is achieved by selecting a local and unbiased sampling distribution tailored to the specific scenario in which GLIME is applied.
Let  $\mathcal{X}$  and  $\mathcal{Y}$  denote the input and output spaces, respectively, where  $\mathcal{X} \subset \mathbb{R}^D$  and  $\mathcal{Y} \subset \mathbb{R}$ . We specifically consider the scenario in which  $\mathcal{X}$  represents the space of images, and  $f: \mathcal{X} \to \mathcal{Y}$  serves as a machine learning model accepting an input  $\mathbf{x} \in \mathcal{X}$ . This study focuses on the classification problem, wherein  $f$  produces the probability that the image belongs to a certain class, resulting in  $\mathcal{Y} = [0,1]$ .
Before proceeding with explanation computations, a set of features  $\{s_i\}_{i=1}^d$  is derived by applying a transformation to  $\mathbf{x}$ . For instance,  $\{s_i\}_{i=1}^d$  could represent image segments (also referred to as super-pixels in LIME) or feature maps obtained from a convolutional neural network. Alternatively,  $\{s_i\}_{i=1}^d$  may correspond to raw features, i.e.,  $\mathbf{x}$  itself. In this context,  $\|\cdot\|_0$ ,  $\|\cdot\|_1$ , and  $\|\cdot\|_2$  denote the  $\ell_0$ ,  $\ell_1$ , and  $\ell_2$  norms, respectively, with  $\odot$  representing the element-wise product. Boldface letters are employed to denote vectors and matrices, while non-boldface letters represent scalars or features.  $B_{\mathbf{x}}(\epsilon)$  denotes the ball centered at  $\mathbf{x}$  with radius  $\epsilon$ .
2.2 A brief introduction to LIME
In this section, we present the original definition and implementation of LIME [22] in the context of image classification. LIME, as a local explanation method, constructs a linear model when provided with an input  $\mathbf{x}$  that requires an explanation. The coefficients of this linear model serve as the feature importance explanation for  $\mathbf{x}$ .
Features. For an input  $\mathbf{x}$ , LIME computes a feature importance vector for the set of features. In the image classification setting, for an image  $\mathbf{x}$ , LIME initially segments  $\mathbf{x}$  into super-pixels  $s_1, \ldots, s_d$  using a segmentation algorithm such as Quickshift [32]. Each super-pixel is regarded as a feature for the input  $\mathbf{x}$ .
Sample generation. Subsequently, LIME generates samples within the local vicinity of  $\mathbf{x}$  as follows. First, random samples are generated uniformly from  $\{0,1\}^d$ . The  $j$ -th coordinate  $z_j'$  for each sample  $\mathbf{z}'$  is either 1 or 0, indicating the presence or absence of the super-pixel  $s_j$ . When  $s_j$  is absent, it is replaced by a reference value  $r_j$ . Common choices for the reference value include a black image, a blurred version of the super-pixel, or the average value of the super-pixel [29, 22, 8]. Then, these  $\mathbf{z}'$  samples are transformed into samples in the original input space  $\mathbb{R}^D$  by combining them with  $\mathbf{x} = (s_1,\dots,s_d)$  using the element-wise product as follows:  $\mathbf{z} = \mathbf{x} \odot \mathbf{z}' + \mathbf{r} \odot (1 - \mathbf{z}')$ , where  $\mathbf{r}$  is the vector of reference values for each super-pixel, and  $\odot$  represents the element-wise product. In other words,  $\mathbf{z} \in \mathcal{X}$  is an image that is the same as  $\mathbf{x}$ , except that those super-pixels  $s_j$  with  $z_j' = 0$  are replaced by reference values.
Feature attributions. For each sample  $\mathbf{z}'$  and the corresponding image  $\mathbf{z}$ , we compute the prediction  $f(\mathbf{z})$ . Finally, LIME solves the following regression problem to obtain a feature importance vector (also known as feature attributions) for the super-pixels:
$$
\mathbf {w} ^ {\text {L I M E}} = \underset {\mathbf {v}} {\arg \min } \mathbb {E} _ {\mathbf {z} ^ {\prime} \sim \operatorname {U n i} \left(\{0, 1 \} ^ {d}\right)} \left[ \pi \left(\mathbf {z} ^ {\prime}\right) \left(f (\mathbf {z}) - \mathbf {v} ^ {\top} \mathbf {z} ^ {\prime}\right) ^ {2} \right] + \lambda \| \mathbf {v} \| _ {2} ^ {2}, \tag {1}
$$
where  $\mathbf{z} = \mathbf{x}\odot \mathbf{z}' + \mathbf{r}\odot (1 - \mathbf{z}')$ ,  $\pi (\mathbf{z}^{\prime}) = \exp \{-\| \mathbf{1} - \mathbf{z}^{\prime}\|_{2}^{2} / \sigma^{2}\}$ , and  $\sigma$  is the kernel width parameter.
Remark 2.1. In practice, we draw samples  $\{\mathbf{z}_i^{\prime}\}_{i = 1}^{n}$  from the uniform distribution  $\mathrm{Uni}(\{0,1\} ^d)$  to estimate the expectation in Equation 1. In the original LIME implementation [22],  $\lambda = \alpha /n$  for a
constant  $\alpha > 0$ . This choice has been widely adopted in prior studies [40, 8, 19, 9, 5, 20, 34]. We use  $\hat{\mathbf{w}}^{LIME}$  to represent the empirical estimation of  $\mathbf{w}^{LIME}$ .
2.3 LIME is unstable and has poor local fidelity
Instability. To capture the local characteristics of the neighborhood around the input  $\mathbf{x}$ , LIME utilizes the sample weighting function  $\pi(\cdot)$  to assign low weights to samples that exclude numerous super-pixels and, consequently, are located far from  $\mathbf{x}$ . The parameter  $\sigma$  controls the level of locality, with a small  $\sigma$  assigning high weights exclusively to samples very close to  $\mathbf{x}$  and a large  $\sigma$  permitting notable weights for samples farther from  $\mathbf{x}$  as well. The default value for  $\sigma$  in LIME is 0.25 for image data. However, as depicted in Figure 1a, LIME demonstrates instability, a phenomenon also noted in prior studies [35, 24, 34]. As showed in Section 4, this instability arises from small  $\sigma$  values, leading to very small sample weights and, consequently, slow convergence.
Poor local fidelity. LIME also suffers from poor local fidelity [16, 21]. The sampling space of LIME is depicted in Figure 1b. Generally, the samples in LIME exhibit considerable distance from the instance being explained, as illustrated in Figure 3, rendering them non-local. Despite LIME's incorporation of weights to promote locality, it fails to provide accurate explanations for local behaviors when the samples themselves lack local proximity. Moreover, the sampling space of LIME is influenced by the reference, resulting in a biased sampling space and a consequent degradation of local fidelity.
3 A general local explanation framework: GLIME
3.1 The definition of GLIME
We first present the definition of GLIME and show how it computes the explanation vector  $\mathbf{w}^{\mathrm{GLIME}}$ . Analogous to LIME, GLIME functions by constructing a model within the neighborhood of the input  $\mathbf{x}$ , utilizing sampled data from this neighborhood. The coefficients obtained from this model are subsequently employed as the feature importance explanation for  $\mathbf{x}$ .
Feature space. For the provided input  $\mathbf{x} \in \mathcal{X} \subset \mathbb{R}^D$ , the feature importance explanation is computed for a set of features  $\mathbf{s} = (s_1, \dots, s_d)$  derived from applying a transformation to  $\mathbf{x}$ . These features  $\mathbf{s}$  can represent image segments (referred to as super-pixels in LIME) or feature maps obtained from a convolutional neural network. Alternatively, the features  $\mathbf{s}$  can correspond to raw features, i.e., the individual pixels of  $\mathbf{x}$ . In the context of LIME, the method specifically operates on super-pixels.
Sample generation. Given features  $\mathbf{s}$ , a sample  $\mathbf{z}'$  can be generated from the distribution  $\mathcal{P}$  defined on the feature space (e.g.,  $\mathbf{s}$  are super-pixels segmented by a segmentation algorithm such Quickshift [32] and  $\mathcal{P} = \mathrm{Uni}(\{0,1\}^d)$  in LIME). It's important to note that  $\mathbf{z}'$  may not belong to  $\mathcal{X}$  and cannot be directly input into the model  $f$ . Consequently, we reconstruct  $\mathbf{z} \in \mathbb{R}^{D}$  in the original input space for each  $\mathbf{z}'$  and obtain  $f(\mathbf{z})$  (in LIME, a reference  $\mathbf{r}$  is first chosen and then  $\mathbf{z} = \mathbf{x} \odot \mathbf{z}' + \mathbf{r} \odot (1 - \mathbf{z}')$ ). Both  $\mathbf{z}$  and  $\mathbf{z}'$  are then utilized to compute feature attributions.
Feature attributions. For each sample  $\mathbf{z}'$  and its corresponding  $\mathbf{z}$ , we compute the prediction  $f(\mathbf{z})$ . Our aim is to approximate the local behaviors of  $f$  around  $\mathbf{x}$  using a function  $g$  that operates on the feature space.  $g$  can take various forms such as a linear model, a decision tree, or any Boolean function operating on Fourier bases [37]. The loss function  $\ell(f(\mathbf{z}), g(\mathbf{z}'))$  quantifies the approximation gap for the given sample  $\mathbf{z}'$ . In the case of LIME,  $g(\mathbf{z}') = \mathbf{v}^\top \mathbf{z}'$ , and  $\ell(f(\mathbf{z}), g(\mathbf{z}')) = (f(\mathbf{z}) - g(\mathbf{z}'))^2$ . To derive feature attributions, the following optimization problem is solved:
$$
\mathbf {w} ^ {\text {G L I M E}} = \underset {\mathbf {v}} {\arg \min } \mathbb {E} _ {\mathbf {z} ^ {\prime} \sim \mathcal {P}} [ \pi (\mathbf {z} ^ {\prime}) \ell (f (\mathbf {z}), g (\mathbf {z} ^ {\prime})) ] + \lambda R (\mathbf {v}), \tag {2}
$$
where  $\pi (\cdot)$  is a weighting function and  $R(\cdot)$  serves as a regularization function, e.g.,  $\| \cdot \| _1$  or  $\| \cdot \| _2^2$  (which is used by LIME). We use  $\hat{\mathbf{w}}^{\mathrm{GLIME}}$  to represent the empirical estimation of  $\mathbf{w}^{\mathrm{GLIME}}$ .
Connection with Existing Frameworks. Our formulation exhibits similarities with previous frameworks [22, 12]. The generality of GLIME stems from two key aspects: (1) GLIME operates within a broader feature space  $\mathbb{R}^d$ , in contrast to [22], which is constrained to  $\{0,1\}^d$ , and [12], which is confined to raw features in  $\mathbb{R}^D$ . (2) GLIME can accommodate a more extensive range of distribution choices tailored to specific use cases.
3.2 An alternative formulation of GLIME without the weighting function
Indeed, we can readily transform Equation 2 into an equivalent formulation without the weighting function. While this adjustment simplifies the formulation, it also accelerates convergence by sampling from the transformed distribution (see Section 4.1 and Figure 4a). Specifically, we define the transformed sampling distribution as  $\widetilde{\mathcal{P}} (\mathbf{z}^{\prime}) = \frac{\pi(\mathbf{z}^{\prime})\mathcal{P}(\mathbf{z}^{\prime})}{\int\pi(\mathbf{t})\mathcal{P}(\mathbf{t})\mathrm{d}\mathbf{t}}$ . Utilizing  $\widetilde{\mathcal{P}}$  as the sampling distribution, Equation 2 can be equivalently expressed as
$$
\mathbf {w} ^ {\text {G L I M E}} = \underset {\mathbf {v}} {\arg \min } \mathbb {E} _ {\mathbf {z} ^ {\prime} \sim \mathcal {P}} [ \ell (f (\mathbf {z}), g \left(\mathbf {z} ^ {\prime}\right)) ] + \frac {\lambda}{Z} R (\mathbf {v}), \quad Z = \mathbb {E} _ {\mathbf {t} \sim \mathcal {P}} [ \pi (\mathbf {t}) \mathcal {P} (\mathbf {t}) ] \tag {3}
$$
It is noteworthy that the feature attributions obtained by solving Equation 3 are equivalent to those obtained by solving Equation 2 (see Appendix B.1 for a formal proof). Therefore, the use of  $\pi(\cdot)$  in the formulation is not necessary and can be omitted. Hence, unless otherwise specified, GLIME refers to the framework without the weighting function.
3.3 GLIME unifies several previous explanation methods
This section shows how GLIME unifies previous methods. For a comprehensive understanding of the background regarding these methods, kindly refer to Appendix A.6.
LIME [22] and GLIME-BINOMIAL. In the case of LIME, it initiates the explanation process by segmenting pixels  $x_{1}, \dots, x_{D}$  into super-pixels  $s_{1}, \dots, s_{d}$ . The binary vector  $\mathbf{z}' \sim \mathcal{P} = \mathrm{Uni}(\{0, 1\}^{d})$  signifies the absence or presence of corresponding super-pixels. Subsequently,  $\mathbf{z} = \mathbf{x} \odot \mathbf{z}' + \mathbf{r} \odot (\mathbf{1} - \mathbf{z}')$ . The linear model  $g(\mathbf{z}') = \mathbf{v}^{\top} \mathbf{z}'$  is defined on  $\{0, 1\}^{d}$ . For image explanations,  $\ell(f(\mathbf{z}), g(\mathbf{z}')) = (f(\mathbf{z}) - g(\mathbf{z}'))^{2}$ , and the default setting is  $\pi(\mathbf{z}') = \exp(-\|\mathbf{1} - \mathbf{z}'\|_{0}^{2}/\sigma^{2})$ ,  $R(\mathbf{v}) = \| \mathbf{v}\|_{2}^{2}$  [22]. Remarkably, LIME is equivalent to the special case GLIME-BINOMIAL without the weighting function (see Appendix B.2 for the formal proof). The sampling distribution of GLIME-BINOMIAL is defined as  $\mathcal{P}(\mathbf{z}', \| \mathbf{z}'\|_{0} = k) = e^{k/\sigma^{2}}/(1 + e^{1/\sigma^{2}})^{d}$ , where  $k = 0, 1, \ldots, d$ . This distribution is essentially a Binomial distribution. To generate a sample  $\mathbf{z}' \in \{0, 1\}^{d}$ , one can independently draw  $z_{i}' \in \{0, 1\}$  with  $\mathbb{P}(z_{i} = 1) = 1/(1 + e^{-1/\sigma^{2}})$  for  $i = 1, \ldots, d$ . The feature importance vector obtained by solving Equation 3 under GLIME-BINOMIAL is denoted as  $\mathbf{w}^{\mathrm{Binomial}}$ .
KernelSHAP [19]. In our framework, the formulation of KernelSHAP aligns with that of LIME, with the only difference being  $R(\mathbf{v}) = 0$  and  $\pi (\mathbf{z}^{\prime}) = (d - 1) / \left(\binom {d}{\| \mathbf{z}^{\prime}\|_{0}}\| \mathbf{z}^{\prime}\|_{0}(d - \| \mathbf{z}^{\prime}\|_{0})\right)$ .
SmoothGrad [28]. SmoothGrad functions on raw features, specifically pixels in the case of an image. Here,  $\mathbf{z} = \mathbf{z}' + \mathbf{x}$ , where  $\mathbf{z}' \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$ . The loss function  $\ell(f(\mathbf{z}), g(\mathbf{z}'))$  is represented by the squared loss, while  $\pi(\mathbf{z}') = 1$  and  $R(\mathbf{v}) = 0$ , as established in Appendix B.6.
Gradient [36]. The Gradient explanation is essentially the limit of SmoothGrad as  $\sigma$  approaches 0.
DLIME [35]. DLIME functions on raw features, where  $\mathcal{P}$  is defined over the training data that have the same label with the nearest neighbor of  $\mathbf{x}$ . The linear model  $g(\mathbf{z}^{\prime}) = \mathbf{v}^{\top}\mathbf{z}^{\prime}$  is employed with the square loss function  $\ell$  and the regularization term  $R(\mathbf{v}) = 0$ .
ALIME [24]. ALIME employs an auto-encoder trained on the training data, with its feature space defined as the output space of the auto-encoder. The sample generation process involves introducing Gaussian noise to  $\mathbf{x}$ . The weighting function in ALIME is denoted as  $\pi (\mathbf{z}^{\prime}) = \exp (-\| \mathcal{A}\mathcal{E}(\mathbf{x}) - \mathcal{A}\mathcal{E}(\mathbf{z}^{\prime})\|_{1})$ , where  $\mathcal{A}\mathcal{E}(\cdot)$  represents the auto-encoder. The squared loss function is chosen as the loss function and no regularization function is applied.
4 Stable and locally faithful explanations in GLIME
4.1 GLIME-BINOMIAL converges exponentially faster than LIME
To understand the instability of LIME, we demonstrate that the sample weights in LIME are very small, resulting in the domination of the regularization term. Consequently, LIME tends to produce explanations that are close to zero. Additionally, the small weights in LIME lead to a considerably slower convergence compared to GLIME-BINOMIAL, despite both methods converging to the same limit.
Figure 2: The distribution and the weight of  $\| \mathbf{z}'\|_0$ . In LIME, the distribution of  $\| \mathbf{z}'\|_0$  follows a binomial distribution, and it is independent of  $\sigma$ . In GLIME-BINOMIAL, when  $\sigma$  is small,  $\| \mathbf{z}'\|_0$  concentrates around  $d$ , while in LIME, most samples exhibit negligible weights except for the all-one vector 1. As  $\sigma$  increases, the distributions in GLIME-BINOMIAL converge to those in LIME, and all LIME samples attain non-negligible weights.