SlowGuess/NeurIPS_2023_OCR huggingface dataset:
whose creation is described in Section 4.2.2. In the algorithm we iteratively learn a constrained least squares predictor for each environment, where the final (local) linear predictor is the sum of these individual predictors. In each iteration when computing the contribution of environment  $e_i$  to the final summed predictor, the most recent contributions of the other predictors are summed and the residual is optimized subject to the constraints. The first constraint is a standard lasso type constraint which tries to keep the final predictor sparse as in LIME.
Why  $l_{\infty}$  constraint? The second constraint is more unique and is a  $l_{\infty}$  constraint on the predictor of just the current environment. This constraint as we prove in Section 4.3 is essential for obtaining robust predictors. To intuitively understand why this is the case consider we have two environments. In this case if the optimal predictors for a feature in each environment have opposite signs, then the Nash equilibrium (NE) is when each predictor takes  $+\gamma$  or  $-\gamma$  values as they try to force the sum to have the same sign as them. In other words, features that have a disagreement in even the direction of their impact are eliminated by our method. LIME type methods on the other hand would simply choose some form of average value of the predictors which may be a risky choice especially for actionability/recourse given that the directions change so abruptly. On the other hand, if the optimal predictors for a feature in the two environments have the same sign, the lower absolute valued predictor would be chosen (assuming  $\gamma$  is greater) making it a careful choice. The reasoning for this and a discussion involving more than two environments is given in Section 4.3.
The overall algorithm resembles a (simultaneous) game where each environment is a player trying to find the best predictor for its environment given all other predictors and constraints. Formally, for  $i \in \{1, \dots, k\}$  the players are  $\mathcal{N} = \{\xi_i\}$ , their strategy space is  $S_i = [-\gamma, \gamma]^d$  and their utility  $u_i(\tilde{\boldsymbol{w}}_i, \tilde{\boldsymbol{w}}_{-i}^+) = -\sum_{\tilde{\boldsymbol{x}} \in \xi_i(\boldsymbol{x})} (f(\tilde{\boldsymbol{x}}) - \tilde{\boldsymbol{w}}_{-i}^{+^T} \tilde{\boldsymbol{x}} - \tilde{\boldsymbol{w}}_i^T \tilde{\boldsymbol{x}})^2$ . The optimization problem solved by each player is convex as norms are convex.
4.2.2 Creating Local Environments
In standard IRM, environments are assumed to be given. In our case of local explainability we have to decide how to produce them. We offer a few options for the environment creation functions  $\xi_{i}$ $\forall i\{1,\dots ,k\}$  in Algorithm 1.
Random Perturbation: This simple approach is similar to what LIME employs. We could perturb the input example by adding zero mean gaussian noise to create the base environment (used by LIME) and then perform bootstrap sampling to create the  $k$  different environments. This will efficiently create neighbors in each environment, although they may be unrealistic in the sense that they could correspond to low probability points w.r.t. the underlying distribution.
Realistic Generation/Selection: One could also create neighbors using data generators such as done in MeLIME [8] or select neighboring examples from the training set as done in MAPLE [39] to create
the base environment following which bootstrap sampling could be done to form the  $k$  different environments. This could provide more realistic neighbors than the previous one, but may be much more computationally expensive. Other than bootstrapping one could also oversample and try to find the optimal hard/soft partition through various clustering type objectives [2, 10].
4.3 Theoretical Results
In this section, we analyze the output of Algorithm 1 with two environments. The extension to multiple environments is discussed following this result, where the general intuition is still maintained but some special cases arise depending on whether there are an even or odd number of environments. To prove our main result we make two assumptions.
Assumption 1 The features of the samples in the local environments are independent.
This assumption is satisfied by the most standard way of creating neighborhoods/environments, where gaussian noise is used to create them as described in Section 4.2.2.
Assumption 2  $t \geq \gamma d$ , where  $d$  is the dimensionality of the feature vector.
Here  $t$  is the parameter in the  $\ell_1$  penalty and  $\gamma$  in the  $\ell_{\infty}$  as noted in Algorithm 1. Making this assumption ensures that we closely analyze the role of the  $\ell_{\infty}$  penalty, which is one of our main novelties.
Definition 2 Let the explanation that each environment  $\xi_{i}$  arrives at for an example  $x$  based on unconstrained least squares minimization be  $\boldsymbol{w}_{i}^{*}$  where,
$$
\boldsymbol {w} _ {i} ^ {*} \in \operatorname * {a r g m i n} _ {\tilde {w} \in \mathbb {R} ^ {d}} \mathbb {E} _ {\tilde {\boldsymbol {x}} \in \xi_ {i} (x)} \left[ \left(f (\tilde {\boldsymbol {x}}) - \tilde {w} ^ {\top} \tilde {\boldsymbol {x}}\right) ^ {2} \right] \tag {7}
$$
The expectation is taken w.r.t the environment generation distribution.
Theorem 1. The output of Algorithm 1 under Assumptions 1, 2 and equation 7 is given by:
$$
\boldsymbol {w} = \left(\boldsymbol {w} _ {1} ^ {*} \odot \mathbf {1} _ {\left| \boldsymbol {w} _ {2} ^ {*} \right| \geq \left| \boldsymbol {w} _ {1} ^ {*} \right|} + \boldsymbol {w} _ {2} ^ {*} \odot \mathbf {1} _ {\left| \boldsymbol {w} _ {1} ^ {*} \right| > \left| \boldsymbol {w} _ {2} ^ {*} \right|}\right) \mathbf {1} _ {\boldsymbol {w} _ {1} ^ {*} \odot \boldsymbol {w} _ {2} ^ {*} \geq 0} \tag {8}
$$
where  $\odot$  is element wise product and  $\mathbf{1}$  is the indicator function.
Proof Sketch. The above expression describes the NE of the game played between the two local environments each trying to move  $\pmb{w}$  towards their least squares optimal solution. Given assumptions 1 and 2, we witness the following behavior of our method. Let the  $i^{th}$  feature of the predictors  $\tilde{w}_{1}$  and  $\tilde{w}_{2}$  from Algorithm 1 be  $\tilde{w}_{1i}$  and  $\tilde{w}_{2i}$  respectively. Let the corresponding least squares optimal predictors for the  $i^{th}$  feature have the following relation:  $w_{1i}^{*} > w_{2i}^{*}$  and  $|w_{1i}^{*}| > |w_{2i}^{*}|$ . Then the two environments will push the ensemble predictor,  $\tilde{w}_{1i} + \tilde{w}_{2i}$ , in opposite directions during their turns, with the first environment increasing its weight,  $\tilde{w}_{1i}$ , and the second environment decreasing its weight,  $\tilde{w}_{2i}$ . Eventually, the environment with a higher absolute value  $(\xi_{1} = 1$  since  $|w_{1i}^{*}| > |w_{2i}^{*}|$ ) reaches the boundary  $(\tilde{w}_{1i} = \gamma)$  and cannot move any further due to the  $l_{\infty}$  constraint. The other environment  $\xi_{2}$  best responds, where it either hits the other end of the boundary  $(\tilde{w}_{2i} = -\gamma)$ , in which case the weight of the ensemble for component  $i$  is zero, a case which occurs if  $w_{1i}^{*}$  and  $w_{2i}^{*}$  have opposite signs; or gets close to the other boundary while staying in the interior  $(\tilde{w}_{2i} = w_{2i}^{*} - \gamma)$ , in which case the weight of the ensemble for feature  $i$  is  $w_{2i}^{*}$ , a situation which occurs if  $w_{1i}^{*}$  and  $w_{2i}^{*}$  have the same sign.
Implications of the Theorem 1: The following are the main takeaways from Theorem 1: (1) If the signs of the explanations for unconstrained least squares for the two environments differ for some feature, then the algorithm outputs a zero for that feature attribution. (2) If the signs of the explanations for the two environments are the same, then the algorithm outputs the lesser magnitude of the two. These two properties are highly desirable from an algorithmic recourse or actionability perspective, where the first biases us to not rely on features where the black-box function changes direction rapidly (unidirectionality). The second, provides a reserved estimate so that we do not incorrectly over rely on the particular feature (stability). Based on similar logic presented in the proof sketch the behavior for more than two environments for LINEX is discussed in Suppl. C.
Figure 2: Sample results using FMNIST dataset for two classes. (a-c): Class  $Dress$ , (d-f): Class Sandal. (a, d): MeLIME explanations. (b, d): LINEX explanations. (c, f): Original images. We observe that LINEX explanations capture important artifacts and thus exhibit significantly higher correlation with the original images for the same level of sparsity, where in aggregate too the correlations are high w.r.t. images belonging to a particular class, thus showcasing higher stability (i.e. high CAC) as is seen in Table 2. More examples are shown in Suppl. G.
Table 1: Below are three example positive sentiment sentences from the Rotten Tomatoes dataset. Green and red indicate the most important word highlighted by MeLIME and LINEX respectively. As seen LINEX highlights stronger positive sentiment words. More examples in Suppl. F.
Example 1	Example 2	Example 3
one-of-a-kind near-masterpiece	moving tale of love and destruction in unexpected places, unexamined lives	spare yet audacious . . .
We test our method on five real world datasets covering all three modalities: IRIS (Tabular) [12], Medical Expenditure Panel Survey (Tabular) [1], Fashion MNIST (Image) [50], CIFAR10 (Image) [31] and Rotten Tomatoes reviews (Text) [38] with LIME-like random (rand) and MeLIME-like realistic neighborhood generation (real) or MAPLE-like realistic neighborhood selection (mpl). The summary of black-box classifier accuracies, and type of realistic perturbation used for the datasets are provided in Table 3 in the Supplement. In other cases except FMNIST and CIFAR10 which come with their own test partition we randomly split the datasets into  $80 / 20\%$  train/test partition and average results for the local explanations over this test partition. For LINEX we produce two environments where the two environments are formed by performing bootstrap sampling on the base environment which is created either by rand, real or mpl type neighborhood generation. Thus in all cases the union of the environments is the same as a single neighborhood used to produce explanations for the competitors making it a fair comparison. Behavior with more environments is in Suppl. E.
Given the neighborhood generation schemes we compare LINEX with LIME, Smoothed LIME (S-LIME), MeLIME and MAPLE, where for S-LIME we average the explanations of LIME across the LINEX environments. SHAP's results are in Suppl. H, since it is not a natural fit here. Nor are methods such as saliency maps, gradcam, integrated gradients as they are white-box methods requiring access to a differentiable model.
Metrics: We evaluate using five simple metrics: Infidelity (INFD), Generalized Infidelity (GI), Coefficient Inconsistency (CI), Class Attribution Consistency (CAC) and Unidirectionality  $(\Upsilon)$ , which are defined in section 4.1. The first two evaluate faithfulness, the next two stability and the last goodness for recourse.
We report the above metrics in Table 2. Each result in Table 2 is mean  $\pm$  stan
Figure 3: Sample results using CIFAR10 dataset for dog and bird class. As can be seen LINEX focuses more on salient features such as head and legs for the dog, and wings for the bird (rather than also the background). More examples are shown in Suppl. G.
dard error of the mean over five kernel sizes  $\tau \sqrt{d}$  generally, where  $\tau = \{0.05, 0.1, 0.25, 0.5, 0.75\}$ . Test neighborhoods do not make sense for random perturbations with FMNIST, CIFAR10 and Rotten Tomatoes because the features (viz. superpixels) used by neighboring test examples are different.
Table 2: Comparison of the different methods based on infidelity (INFD), generalized infidelity (GI), coefficient inconsistency (CI), class attribution consistency (CAC) and unidirectionality  $(\Upsilon)$ .  $\uparrow$  indicates higher value for the metric is better, and  $\downarrow$  indicates lower is better. Statistically significant results based on paired t-test are bolded. LINEX is better than baselines in 21 out of 40 cases, and worse only in 5 cases. Plots showing behavior with varying neighborhood size, number of environments and kernel width are in Suppl. E.
Dataset	Method	INFD ↓	GI ↓	CI ↓	Y ↑	CAC ↑
IRIS	LIME	0.015 ± 0.011	0.132 ± 0.042	0.319 ± 0.132	0.646 ± 0.040	0.667 ± 0.167
S-LIME	0.015 ± 0.010	0.077 ± 0.011	0.143 ± 0.045	0.704 ± 0.037	0.878 ± 0.034
LINEX/random	0.013 ± 0.009	0.052 ± 0.008	0.044 ± 0.013	0.802 ± 0.043	0.921 ± 0.042
MeLIME	0.008 ± 0.003	0.049 ± 0.018	0.219 ± 0.108	0.629 ± 0.013	0.464 ± 0.100
LINEX/real	0.009 ± 0.003	0.029 ± 0.003	0.024 ± 0.002	0.744 ± 0.044	0.942 ± 0.023
MAPLE	0.009 ± 0.001	0.038 ± 0.004	0.261 ± 0.033	0.458 ± 0.032	0.586 ± 0.035
LINEX/mp1	0.013 ± 0.000	0.020 ± 0.000	0.026 ± 0.002	0.694 ± 0.008	0.929 ± 0.004
MEPS	LIME	0.158 ± 0.066	0.214 ± 0.041	0.005 ± 0.001	0.981 ± 0.006	NA
S-LIME	0.158 ± 0.066	0.214 ± 0.042	0.005 ± 0.001	0.974 ± 0.008
LINEX/random	0.130 ± 0.052	0.164 ± 0.021	0.003 ± 0.001	0.979 ± 0.006
MAPLE	0.063 ± 0.000	0.067 ± 0.000	0.007 ± 0.000	0.957 ± 0.000	NA
LINEX/mp1	0.098 ± 0.001	0.094 ± 0.001	0.007 ± 0.000	0.950 ± 0.000
FMNIST	LIME	0.162 ± 0.003	NA	NA	NA	NA
S-LIME	0.142 ± 0.003
LINEX/random	0.149 ± 0.002
MeLIME	0.001 ± 0.000	0.277 ± 0.000	0.007 ± 0.000	0.769 ± 0.000	0.327 ± 0.000
LINEX/real	0.100 ± 0.002	0.304 ± 0.001	0.002 ± 0.000	0.780 ± 0.000	0.649 ± 0.001
CIFAR10	LIME	0.191 ± 0.005	NA	NA	NA	NA
S-LIME	0.185 ± 0.002
LINEX/random	0.186 ± 0.002
MeLIME	0.100 ± 0.003	0.412 ± 0.007	0.014 ± 0.000	0.546 ± 0.003	NA
LINEX/real	0.090 ± 0.005	0.279 ± 0.001	0.006 ± 0.000	0.679 ± 0.004
Rotten Tomatoes	LIME	0.079 ± 0.036	NA	NA	NA	NA
S-LIME	0.075 ± 0.035
LINEX/random	0.069 ± 0.032
MeLIME	0.029 ± 0.001	0.391 ± 0.000	0.000 ± 0.000	0.999 ± 0.000	0.909 ± 0.000
LINEX/real	0.053 ± 0.000	0.361 ± 0.000	0.000 ± 0.000	1.000 ± 0.000	0.953 ± 0.001
Also, we do not use realistic perturbations with MEPS since KDE and VAE generators do not work well with categorical data. In addition, since MEPS data uses regression black-box, CAC cannot be computed. Also for CIFAR10 images in a class are not aligned so CAC is inapplicable. All these justify the missing entries in Table 2. The results were generated on Linux machines with 56 cores and 242 GB RAM. More details regarding the exact perturbation schemes for LIME/MeLIME/MAPLE, the perturbation neighborhood sizes and the time taken by the methods are in Suppl. A and Suppl. D.
Observations: Quantitatively, we see that in terms of CAC, LINEX is better than baselines in all cases which indicates that on average the LINEX explanations highlight the important features characterizing the entire class, making them more stable. This is also verified by looking at  $\Upsilon$  and CI metrics where LINEX is similar or better than others. For GI and INFD, the results are more evenly spread which implies that LINEX's key advantage is obtaining stable and unidirectional explanations that are faithful to a similar degree. Ablation studies showing superiority of LINEX over MeLIME on the FMNIST dataset where we have significantly higher INFD than MeLIME are given in Suppl. J.
An interesting observation is that when it comes to the stability metrics (CI and CAC) and unidirectionality LINEX with even random perturbation model is better than MeLIME in some cases. This is very promising as it means LINEX could be potentially be trusted without the need to generate realistic perturbations which may be computationally expensive or not even possible.
Qualitatively, we see in Figures 2 and 3, that LINEX explanations are more coherent and highlight more salient features compared to MeLIME. Even on the text data we see more reasonable attributions in Table 1, where "masterpiece", "moving" and "audacious" are highlighted as the most important words indicative of positive sentiment in the three examples. We also performed qualitative error
analysis on FMNIST where our INFD is much worse than MeLIME and is described in Suppl. I. We see that even where LINEX has high infidelity it invariably still focuses on salient features ignoring superfluous features which may result in lower fidelity but may not be critical for correct identification. The goodness of these features identified by LINEX can be further verified by looking at other metrics such as GI, CAC, CI and  $\Upsilon$  in Table 2 where it is either comparable or better than MeLIME.
In this paper we have provided a method based on a game theoretic formulation and inspired by the invariant risk minimization principle to provide faithful, stable and unidirectional explanations. We have defined the latter property and argued that it is somewhat of a necessity (may not be sufficient) for recourse. We have theoretically shown that our method has a strong tendency to be stable and unidirectional as we will mostly eliminate features where the black-box models gradient changes abruptly and in other cases choose a conservative value. Empirically, we have verified this where we outperform competitors in majority of the cases on these metrics. Interestingly, in some cases our method provides more stable and unidirectional explanations with just a random perturbation model relative to more expensive methods that use realistic neighbors.
We now discuss a real world use case we tested our method on. We worked with a large financial institution to explain the fraud detection model they had built. The Association of Certified Fraud Examiners (ACFE) claims that roughly  $5\%$  of a companies revenue is lost to fraud every year. Thus, catching fraud or even non-compliance is extremely important for any organization. Their model (fraud = 1 else 0) had  $\approx 91\%$  accuracy. The inputs to the model were (transactional) invoices and details corresponding to those invoices such as vendor name, invoice amount, purchase order (PO) or not, vendor address, commodity code, country perception indices (CPI), etc. Since, one of the focuses is to reduce false positives accurate explanations are important. We applied LINEX to this setting to explain why certain invoices were classified as fraudulent. The experts found that in majority of the cases (913 out of 1000) the attributions of LINEX especially in terms of sign made sense. For instance, low CPI implies high risk and so LINEX gave a negative coefficient for this feature for most examples, while LIME gave a positive coefficient for many instances. Going forward their plan is to incorporate such capabilities into their workflow to further improve fraud detection precision.
In the future, it would be worth experimenting with more varied strategies to form environments and if possible find the optimal ones [10], which may lead to picking even more relevant features that are "causal" to the local decision.
7 Summary of the Supplement
Information about black-box classifier accuracies and realistic perturbation methods used for the datasets are provided in Table 3. Suppl. A has run time comparisons. Suppl. B has proof of Theorem 1. Suppl. C discusses theoretical behavior of LINEX for more than two environments. Suppl. D has dataset details and hyperparameter specifications. Suppl. E has experiments with different hyperparameter combinations (including more than 2 environments). Suppl. F has additional examples of text data attributions. Suppl. G has example feature attributions with image data. Suppl. H has SHAP results. Suppl. I, J and K has error analysis and ablation studies. Suppl. L has additional synthetic experiments. Suppl. M discusses sensitivity to  $\gamma$ . Suppl. N demonstrates convergence of LINEX. Suppl. O discusses limitations of LINEX. Figure 30 depicts SLIME variants using median and median of means which turn out to be worse than using the (typical) mean.
[1] Agency for Healthcare Research and Quality. Medical Expenditure Panel Survey (MEPS). https://meps.ahrq.gov/mepsweb/, 2019. Last accessed 2019-08.
[2] C. Aggarwal and C. Reddy. Data Clustering: Algorithms and Applications. CRC Press, 2013.
[3] K. Ahuja, K. Shanmugam, and A. Dhurandhar. Linear regression games: Convergence guarantees to approximate out-of-distribution solutions. In AISTATS, 2021.
[45] A. A. Shrotri, N. Narodytska, A. Ignatiev, J. Marques-Silva, K. S. Meel, and M. Vardi. Constraint-driven explanations of black-box  $\{\mathrm{ml}\}$  models, 2021.
[46] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv:1312.6034, 2013.
[47] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. ICML, 2017.
[48] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, S. Sakenis, J. Huang, Y. Singer, and S. Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. NeurIPS, 2020.
[49] G. Visani, E. Bagli, F. Chesani, A. Poluzzi, and D. Capuzzo. Statistical stability indices for lime: obtaining reliable explanations for machine learning models. arXiv:2001.11757, 2020.
[50] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[51] P. N. Yannella and O. Kagan. Analysis: Article 29 working party guidelines on automated decision making under gdpr. 2018. https://www.cyberadviserblog.com/2018/01/analysis-article-29-working-party-guidelines-on-automated-decision-making-under-gdpr/.
[52] C.-K. Yeh, C.-Y. Hsieh, A. Suggala, D. I. Inouye, and P. K. Ravikumar. On the (in) fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, pages 10965-10976, 2019.
[53] H. Zhang, H. Chen, Z. Song, D. Boning, I. S. Dhillon, and C.-J. Hsieh. The limitations of adversarial training and the blind-spot attack. In International Conference on Learning Representations (ICLR), 2019.
[54] Y. Zhang, K. Song, Y. Sun, S. Tan, and M. Udell. Why should you trust my explanation? ICML-AI for Social Good, 2019.
[55] X. Zhao, W. Huang, X. Huang, V. Robu, and D. Flynn. Baylime: Bayesian local interpretable model-agnostic explanations. UAI, 2021.
Table 3: Datasets, models and neighborhoods used in experiments.  $\mathrm{RF}\rightarrow$  Random Forest,  $\mathrm{NN}\rightarrow$  Neural Network, ResNet  $\rightarrow$  Residual Network and NB  $\rightarrow$  Naive Bayes.
Dataset	Modality	Black-box model acc/R2,	Realistic neighborhood creation methods
IRIS	tabular	RF classifier, 93%	KDEGen [8], RF [39]
MEPS	tabular	RF regressor, 0.325	[39]
FMNIST	image	NN classifier, 87%	VAEGen [8]
CIFAR10	image	ResNet18, 95%	VAEGen [8]
Rotten Tomatoes	text	NB classifier, 75%	Word2VecGen [8]
A Efficiency of LINEX
It is important to note that the query complexity (i.e. number of times we query the black box to obtain an explanation) of LINEX is the same as that of LIME since the union of the environments is the same as a LIME perturbation neighborhood. This is important in todays cloud-driven world where models may exist on different cloud platforms and posthoc explanations are an independent service where each call to the model has an associated cost. In terms of running time for two environments, convergence was fast and running time was approximately 2.5 times that of LIME (LINEX took 2.5 seconds on IRIS for 30 examples as opposed to 1 second by LIME, LINEX took 47 seconds on MEPS for 500 examples as opposed to 18 seconds by LIME), which is very similar to Smoothed LIME (S-LIME) (took 2.3 seconds on IRIS and 40 seconds on MEPS) that we still outperform in majority of the cases.
Realistic neighborhood generation can be time consuming especially for MeLIME since generators have to be trained which may take up to an hour using a single GPU for datasets such as FMNIST. After the generator is trained and neighborhood sampled MeLIME takes the same amount of time as LIME since the model fitting procedure is the same. MAPLE took 1.5 seconds for the IRIS dataset for 30 examples and 27 seconds for 500 MEPS examples.